TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100518 (2010-05-25, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-05-18

General News

  • CA Cloud Squad Actions [20100518]
    • SFU-LC2 scheduled downtime added to ADC Site-Exclusion Savannah [20100510]
      • site queues marked offline in Panda [20100513] shortly after downtime began for queue draining; returned to test state after downtime end [20100517]
    • user build jobs failing at CA-SCINET-T2 [20100511]
      • follow-up with site in regards to missing rpm installation
    • removal of CA-ALBERTA-WESTGRID-T2 in manual GangaRobot blacklist [20100511]
      • new subscription of "mc0*.merge.AOD.e*_r*" dataset required to MCDISK
      • dataset subscription attempts completed but under investigation
    • TRIUMF and FTS file deletion status [20100518]
      • discussed manual steps to subscribe input datasets to SFU-LCG2 but was postponed due to the advanced start-date of the site downtime
      • Functional Tests Transfers from TRIUMF to CERN, ASGC, CNAF halted (via site services machine blacklisting of TRIUMF)
    • submitted GGUS in regards to SFU-LCG2 site storage information missing from CERN top-level bdii (lcg-bdii.cern.ch) [20100513]
    • Open Issues and Ongoing Discussions:
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • http://savannah.cern.ch/bugs/?65355 TRIUMF-LCG2 Transfer failures; source file does not exist [20100406] (see TRIUMF site report below and ggus 56849)
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • ATLAS US VOMS server information [20100511]
    • ACTION: Sites please confirm if you have added support for this VOMS server (done for SFU-LCG2)*
    • vo.racf.bnl.gov is another ATLAS VOMS server at BNL supported by default in gLite for SL4 but not for SL5
    • sites should check for the appropriate .lsc file in the SL5 WN installation
    • file content (vo.racf.bnl.gov.lsc): 
/DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov
/DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1
    • for SL4 CE using yaim the following should be added to site-info.def prior to running yaim configuration step: 
atlas vo.racf.bnl.gov 15003 /DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov 24' into VO_ATLAS_VOMSES
DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1' into VO_ATLAS_VOMS_CA_DN

  • Squid Deployment [20100518]
    • New Squid rpm release announced [20100426] - See link
      • ACTION ITEM FOR SITES:
        • Sites should please prepare their local Squid cache installations for updating as soon as possible, and confirm when the updated version is in place (Australia-ATLAS done)
        • when configuring the squid please add the following networks as to be allowed to connect to your site squid:
          • your internal cluster network(s)
          • 192.12.15.0/255.255.255.0 for the BNL testing network
          • 206.12.25.0/255.255.255.128 for CA-ALBERTA-WESTGRID-T2
          • 142.150.188.0/255.255.255.0 for CA-SCINET-T2 to be confirmed
          • 206.12.48.0/255.255.255.0 for CA-VICTORIA-WESTGRID-T2
          • 206.12.24.0/255.255.255.0 for SFU-LCG2
          • 142.104.21.0/255.255.255.128 for VICTORIA-LCG2
          • 206.12.1.0/255.255.255.0 for TRIUMF-LCG2
          • 192.231.127.0/255.255.255.0 for Australia-ATLAS

    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100518]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2)
    • SFU-LCG2 WARN (both CEs); local squid is OK; backup squid at Victoria fails
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (both for local and connection to SFU squid; see error )
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

  • ACTION ITEM (from 20100511): Bryan to investigate the precise nature of the failures affecting the low availability/reliability reported for CA-SCINET-T2 and CA-ALBERTA-WESTGRID-T2 and communicate back to the group
    • t2-avail-reliab-diagnostics-042010.pdf
      • CA-ALBERTA-WESTGRID-T2
        • from Apr 1 to Apr 5 the site was in a series of unscheduled downtimes which were extensions of the original to perform the WN migration from SL4 to SL5
      • CA-SCINET-T2
        • unscheduled downtime Apr 29 and 30
        • need to check what actual errors were reported for lcg-ce1 that yield intermittent availability and reliability

APEL Accounting

  • ACTION ITEM FOR SITES: Sites please confirm their historical APEL data that has been published looks correct when comparing with batch system utilization
    • See example steps circulated by D. Qing (20100406)
    • Confirmed checks performed to date have been done at: TRIUMF-LCG2, SFU-LCG2, CA-SCINET-T2 (20100504)
  • See EGEE Accounting Portal

  • NOTE:
    • specific issues with APEL first identified [20100510]
      • 1) the "Tier2" view for Canada now shows CA-EAST-T2 and CA-WEST-T2 as per how out WLCG commitments are organized. However, the CA-WEST-T2 only currently lists CA-ALBERTA-WESTGRID-T2. The SFU-LCG2 and VICTORIA-LCG2 sites are listed at the same level as East and West within CA, and so not counted as part of the CA-WEST-T2 federation.
    • to fix item 1) I will request that the SFU-LCG2 and VICTORIA-LCG2 sites be added into CA-WEST-T2.

APEL Publishing Status

  • apel-status-aus-20100518.png:
    apel-status-aus-20100518.png

  • apel-status-ca-20100518.png:
    apel-status-ca-20100518.png

APEL Accounting Results

  • apel-statistics-aus-20100518.png:
    apel-statistics-aus-20100518.png

  • apel-statistics-ca-20100518.png:
    apel-statistics-ca-20100518.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517 and subsequently suspended by CA_ROC until site re-certification occurs

  • CA-ALBERTA-WESTGRID-T2 site status [20100518]
    • see outstanding issues and OPS VO SAM srmv2 tests errors that have been occurring more frequently as of late
    • adjustments planned for various pool group definitions in dCache to hopefully correct this
    • liberated some space in existing reservations to enable successful SAM OPS tests

Outstanding Issues

  • CA-ALBERTA-WESTGRID-T2 site status [20100511]
    • APEL publishing to be corrected; already investigating to confirm that accounting info is at least reaching the local mySQL database
    • APEL statistics to be validated once publishing error is resolved
    • information system update
      • published value of GlueHostArchitectureSMPSize to be corrected during next yaim configuration
      • with next yaim configuration will also correct incorrect userid assigned to atcan006
    • additional open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]
    • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • scheduled down-time completed
    • campus power outage - 20100512 to 20100517

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status? [reported solved 20100518 SChelsky]

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand

  • build job issues and updated image deployment for missing rpms [20100512]

  • large number of LFC errors continue (see Outstanding Issues)

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

Outstanding Issues

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:
    • https://gus.fzk.de/ws/ticket_info.php?ticket=57695 Release 15.6.4.1 missing for tasks 131413 and 131405 at TRIUMF-LCG2 (20100427) (unsolved)
    • https://gus.fzk.de/ws/ticket_info.php?ticket=57021 AtlasProduction/15.6.6.5 not found in CA/TRIUMF (20100404) (assigned)
    • https://gus.fzk.de/ws/ticket_info.php?ticket=56849 missing files at TRIUMF-LCG2 (20100330) (in progress)

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • AUSTRALIA-ATLAS_MCDISK became full and blacklisted [20100515]
    • temporary addition of 8 TB allowed site to be unblacklisted [20100517]

  • some issues with PBS-Maui server over the weekend [20100518]
    • reboot of PBS node cleared up the issues

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
    dashboard-ca-20100518.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-all-20100518.png

  • Alternate Views (See link for last 3 days only):
    panda-dashboard-all-20100518.png
  • CA Cloud (last 3 days):
    panda-dashboard-ca-20100518.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard: 

name,rank,value
TRF_UNKNOWN ,1,1424
EXEPANDA_DQ2_STAGEIN,2,1146
TRFERROR,3,262
TRF_SEGFAULT ,4,229
EXEPANDA_DQ2PUT_LOCAL-OUTPUT-FILE-MISSING,5,33
EXEPANDA_JOBKILL_BYPILOT,6,30
EXEPANDA_Release_MISSING,7,14
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,8,13
EXEPANDA_JOBDISPATCHER_HEARTBEAT,9,11
EXEPANDA_GET_SIZEMISMATCH,10,5
EXEPANDA_DQ2PUT_FILECOPYERROR,11,3
TRF_SVRINIT ,12,1

    • VICTORIA-LCG2 
name,rank,value
TRF_SEGFAULT ,1,22
TRFERROR,2,18
EXEPANDA_JOBKILL_BYPILOT,3,6
TRF_UNKNOWN ,4,3

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRF_SEGFAULT ,1,159
TRFERROR,2,51
TRF_UNKNOWN ,3,4
EXEPANDA_GET_SIZEMISMATCH,4,4

    • Australia-ATLAS 
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,35
TRF_SEGFAULT ,2,8
TRFERROR,3,4

You can find the meanings of the transform error codes on this link.

  • Yuri's notes:
    • week (May 11 - May 17)
      • active tasks from PandaDB: 976
        • distribution by cloud: CA:17 CERN:6 DE:193 ES:108 FR:85 IT:73 ND:44 NL:172 TW:14 UK:33 US:231 

2) Wed May 12 
- CA/CA-SCINET-T2 >900 job f./Get er.:Failed to get LFC replicas.GGUS 58116 in progress updated. 

3) Thu May 13 
+ CA:valid1 rec.t.134292,95,96,99 job f/TRF_SEGVIO+killed.Savannah 67386 tasks set to "finished". 

4) Fri May 14 
 + CA/TRIUMF t.118006 job f./Segm.fault.Savannah 67440.Set to "finished". 

5) Sat May 15 
- CA/AUSTRALIA-ATLAS_MCDISK 500 FT f./NO_SPACE_LEFT.Savannah 67462, 114461 blacklisted.8TB added May 17.

-- BryanCaron - 17 May 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf ATLAS_Schedule_for_UVic-May17.pdf r1 manage 40.5 K 2010-05-18 - 14:34 BryanCaron  
PNGpng apel-statistics-aus-20100518.png r1 manage 832.9 K 2010-05-18 - 17:46 BryanCaron  
PNGpng apel-statistics-ca-20100518.png r1 manage 1302.5 K 2010-05-18 - 17:46 BryanCaron  
PNGpng apel-status-aus-20100518.png r1 manage 15.1 K 2010-05-18 - 17:54 BryanCaron  
PNGpng apel-status-ca-20100518.png r1 manage 24.3 K 2010-05-18 - 17:54 BryanCaron  
PNGpng dashboard-all-20100518.png r1 manage 960.2 K 2010-05-18 - 17:58 BryanCaron  
PNGpng dashboard-ca-20100518.png r1 manage 403.6 K 2010-05-18 - 17:59 BryanCaron  
PNGpng panda-dashboard-all-20100518.png r1 manage 1884.3 K 2010-05-18 - 17:37 BryanCaron  
PNGpng panda-dashboard-ca-20100518.png r1 manage 631.9 K 2010-05-18 - 17:38 BryanCaron  
PDFpdf t2-avail-reliab-diagnostics-042010.pdf r1 manage 382.5 K 2010-05-18 - 18:59 BryanCaron  
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r7 - 2010-05-25 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback