TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100713 (2010-07-13, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-07-13

General News

  • CA Cloud Squad Actions [20100713]
    • resolved dccp access to CA-VICTORIA-WESTGRID-T2 within schedconfig settings [20100708]
    • updated ToA to include corrected seinfo section for CA-VICTORIA-WESTGRID-T2 [20100708]
    • schedconfig updated (once again) to appropriate values of maxinputsize for TRIUMF and CA-VICTORIA-WESTGRID-T2 analysis queues [20100708]
    • submitted Panda Savannah ticket regarding non-persistent changes in the schedconfig values for site queues (20100713)
    • VICTORIA-LCG2 analysis queue marked online by Cloud Squad (20100708)
      • please note zero automatic subscriptions to site MCDISK (deleted) or DATADISK
    • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
      • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708)
    • Open Issues and Ongoing Discussions:
      • ROOTSYS truncated at 127 characters issue [20100608]
        • http://savannah.cern.ch/bugs/?68546 (open)
        • currently affecting Victoria (Tier3) and CA-SCINET-T2
        • varying character lengths below and above 127 characters show variable success and failure
        • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
          • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • Squid Deployment [20100713]
    • New Squid rpm release announced [20100426] - See link
      • ACTION ITEM FOR SITES:
        • Sites should please prepare their local Squid cache installations for updating as soon as possible, and confirm when the updated version is in place (Australia-ATLAS, CA-SCINET-T2, SFU-LCG2 confirmed done)
        • when configuring the squid please add the following networks as to be allowed to connect to your site squid:
          • your internal cluster network(s)
            • NOTE: This is the network from WNs connections will be made to the outside world (ie. gateway network)
          • 192.12.15.0/255.255.255.0 for the BNL testing network
          • 206.12.25.0/255.255.255.128 for CA-ALBERTA-WESTGRID-T2
          • 142.150.188.0/255.255.255.0 for CA-SCINET-T2
          • 142.104.3.226 for CA-VICTORIA-WESTGRID-T2
          • 206.12.24.0/255.255.255.0 for SFU-LCG2
          • 142.104.21.0/255.255.255.128 for VICTORIA-LCG2
          • 206.12.1.0/255.255.255.0 for TRIUMF-LCG2
          • 192.231.127.0/255.255.255.0 for Australia-ATLAS

    • SAM test: (link): currently status as of this morning (WARN is ok) [20100713]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required; backup to SFU squid passes)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2; backup to SFU squid)
    • CA-VICTORIA-WESTGRID-T2 OK (both for local at VICTORIA-LCG2 and backup to SFU squid)
    • SFU-LCG2 WARN (both CEs; local squid is OK; backup squid at VICTORIA-LCG2 fails)
    • VICTORIA-LCG2 WARN (OK for local squid but fails for connection to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid OK; backup access to VICTORIA-LCG2 fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)

  • Site Action Items:
    • VICTORIA-LCG2
      • please confirm that the external gateway network address from where your site WNs appear to originate when accessing the outside networks is as listed above.
      • please confirm the local squid ACLs and any firewall are configured to accept connections from the above addresses since SFU and Australia tests show errors.
    • SFU-LCG2
      • please confirm the local squid ACLs and any firewall are configured to accept connections from the above addresses since VICTORIA-LCG2 tests show errors.
        • only once VICTORIA-LCG2 site gateway node address is confirmed
    • CA-ALBERTA-WESTGRID-T2
      • reinstall local squid server

SAM Results

  • ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: investigate cause of lower measure of availability and reliability for May 2010

APEL Accounting

  • draft June 2010 accounting (Tier-2) information available and to be published [2010-07-16]
    • CA-ALBERTA-WESTGRID-T2 information should appear this week following correction to the site apel parsing and publishing for previous months (starting March 2010)

wlcg-accounting-aus-June2010.png
wlcg-accounting-ca-June2011.png

APEL Publishing Status

apel-status-ca-20100713.png
apel-status-aus-20100713.png

APEL Accounting Results

apel-statistics-ca-20100713.png
apel-statistics-aus-20100713.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517 and subsequently suspended by CA_ROC until site re-certification occurs

  • CA-ALBERTA-WESTGRID-T2 site status [20100713]
    • DDM: discrepancy between SRM and DQ2 usage (see above savannah)
    • corrected site accounting parsing and publishing into local database including APEL accounting dashboard
    • scheduled downtime for 2010-07-16 to 2010-07-19 registered in GOCDB for scheduled power interruption to main campus server room which hosts the Tier-2 resources

Outstanding Issues

  • CA-ALBERTA-WESTGRID-T2 site status [20100511]
    • additional open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]
    • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • VICTORIA-LCG2 SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • large number of LFC errors continue this past week (see Outstanding Issues)

Outstanding Issues

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
    prod-dashboard-ca-20100713.png

  • Within the CA Cloud (Panglia View):
    panglia-anal-week-ca-20100713.pngpanglia-prod-week-ca-20100713.png

  • How we fared to other ATLAS sites during the past week:
    prod-dashboard-all-20100713.png

  • Alternate Views (See link for last 3 days only):
    panda-dashboard-summary-all-20100713.png
  • CA Cloud (last 3 days):
    panda-dashboard-summary-ca-20100713.png

Failed Jobs

  • From Panda Monitor (last week):
    panda-dashboard-errors-header-20100713.png panda-dashboard-errors-aus-20100713.png panda-dashboard-errors-ca-20100713.png panda-dashboard-errors-sfu-20100713.png panda-dashboard-errors-triumf-20100713.png panda-dashboard-errors-victoria-20100713.png

  • From Dashboard (last week): 

name,rank,value
TRF_UNKNOWN ,1,3662
EXEPANDA_DQ2_STAGEIN,2,37
TRF_SEGFAULT ,3,36
EXEPANDA_DQ2_FILEADD,4,29
TRFERROR,5,26
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,6,16
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,1
EXEPANDA_GET_SIZEMISMATCH,8,1

    • VICTORIA-LCG2 
name,rank,value
TRF_UNKNOWN ,1,31
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,5
TRF_SEGFAULT ,3,3
EXEPANDA_JOBKILL_NOLOCALSPACE,4,2
EXEPANDA_JOBKILL_BYPILOT,5,2
TRFERROR,6,2
EXEPANDA_PILOTEXCEPTION,7,1

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRF_UNKNOWN ,1,33
EXEPANDA_GET_SIZEMISMATCH,2,6

    • Australia-ATLAS 
name,rank,value
TRFERROR,1,7
TRF_UNKNOWN ,2,7

You can find the meanings of the transform error codes on this link.

  • Yuri's notes:
    • week (June 6 - July 12)
      • active tasks from PandaDB: 309
        • distribution by cloud: CA:9 CERN:1 DE:41 ES:17 FR:31 IT:8 ND:12 NL:75 TW:11 UK:19 US:85 

6) Sun Jul.11
 + T2s+T3s/huge transferring backlog emergency.Elog 14479,14480:CA,DE, ES,FR,IT,TW,UK,US set brokeroff.TLO,THI increased to 5 days.Fixed.

-- BryanCaron - 12 Jul 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100713.png r1 manage 866.9 K 2010-07-13 - 14:48 BryanCaron  
PNGpng apel-statistics-ca-20100713.png r1 manage 1288.7 K 2010-07-13 - 14:47 BryanCaron  
PNGpng apel-status-aus-20100713.png r1 manage 15.2 K 2010-07-13 - 14:48 BryanCaron  
PNGpng apel-status-ca-20100713.png r1 manage 24.6 K 2010-07-13 - 14:48 BryanCaron  
PNGpng panda-dashboard-errors-aus-20100713.png r1 manage 416.0 K 2010-07-13 - 15:58 BryanCaron  
PNGpng panda-dashboard-errors-ca-20100713.png r1 manage 1570.3 K 2010-07-13 - 15:59 BryanCaron  
PNGpng panda-dashboard-errors-header-20100713.png r1 manage 94.1 K 2010-07-13 - 18:14 BryanCaron  
PNGpng panda-dashboard-errors-sfu-20100713.png r1 manage 342.8 K 2010-07-13 - 15:59 BryanCaron  
PNGpng panda-dashboard-errors-triumf-20100713.png r1 manage 780.7 K 2010-07-13 - 15:59 BryanCaron  
PNGpng panda-dashboard-errors-victoria-20100713.png r1 manage 703.1 K 2010-07-13 - 16:00 BryanCaron  
PNGpng panda-dashboard-summary-all-20100713.png r1 manage 1898.0 K 2010-07-13 - 14:52 BryanCaron  
PNGpng panda-dashboard-summary-ca-20100713.png r1 manage 726.3 K 2010-07-13 - 15:03 BryanCaron  
PNGpng panglia-anal-week-ca-20100713.png r1 manage 28.4 K 2010-07-13 - 15:09 BryanCaron  
PNGpng panglia-prod-week-ca-20100713.png r1 manage 26.8 K 2010-07-13 - 15:09 BryanCaron  
PNGpng prod-dashboard-all-20100713.png r1 manage 923.9 K 2010-07-13 - 18:32 BryanCaron  
PNGpng prod-dashboard-ca-20100713.png r1 manage 505.1 K 2010-07-13 - 15:24 BryanCaron  
PNGpng prod-dashboard-summary-none-scinet1-20100713.png r1 manage 67.3 K 2010-07-13 - 18:35 BryanCaron  
PNGpng prod-dashboard-summary-none-scinet2-20100713.png r1 manage 61.0 K 2010-07-13 - 18:35 BryanCaron  
PNGpng prod-dashboard-summary-none-vicwg-20100713.png r1 manage 141.1 K 2010-07-13 - 18:35 BryanCaron  
PNGpng wlcg-accounting-aus-June2010.png r1 manage 368.1 K 2010-07-12 - 16:04 BryanCaron  
PNGpng wlcg-accounting-ca-June2011.png r1 manage 316.2 K 2010-07-12 - 16:04 BryanCaron  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2010-07-13 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback