TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100511 (2010-05-12, RyanTaylor) EditAttach

Tier1 and Tier-2 Weekly Report 2010-05-11

General News

  • CA Cloud Squad Actions [20100511]
    • SFU-LC2 scheduled downtime added to ADC Site-Exclusion Savannah [20100510]
    • requested fix to ToA to include previously approved PHYS-EXOTICS_GROUPDISK area to be hosted at CA-ALBERTA-WESTGRID-T2 (20100503)
      • CA-ALBERTA-WESTGRID-T2_PHYS-EXOTICS token created and entered into ToA by DDM experts [20100507]
    • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216] RESOLVED 20100506
      • likely connected with the observed missing sites in the ATLAS SRMv2 SAM tests issue https://gus.fzk.de/ws/ticket_info.php?ticket=56196 (open)
      • need to follow-up with the schedconfig team as to why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts
        • RESOLVED 20100427* - fixed by experts in svn which is used to populate the schedconfig database
    • site information sanity checks [20100506]
      • sites contacted in regards to their published values of HEP-SPEC06, SI00, GlueHostArchitectureSMPSize and Cores from their site BDIIs
        • all corrections made
    • Open Issues and Ongoing Discussions:
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • http://savannah.cern.ch/bugs/?65355 TRIUMF-LCG2 Transfer failures; source file does not exist [20100406] (see TRIUMF site report below and ggus 56849)
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • ATLAS US VOMS server information [20100511]
    • vo.racf.bnl.gov is another ATLAS VOMS server at BNL supported by default in gLite for SL4 but not for SL5
    • sites should check for the appropriate .lsc file in the SL5 WN installation
    • file content (vo.racf.bnl.gov.lsc): 
/DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov
/DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1
    • for SL4 CE using yaim the following should be added to site-info.def prior to running yaim configuration step: 
atlas vo.racf.bnl.gov 15003 /DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov 24' into VO_ATLAS_VOMSES
DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1' into VO_ATLAS_VOMS_CA_DN

  • draft April WLCG Tier2 Accounting usage report issued (20100510)
    • see section on APEL accounting below

  • draft April WLCG Tier2 Reliability and Availability report issued (20100511)
    • see SAM availability and reliability section below

  • Squid Deployment [20100511]
    • New Squid rpm release announced [20100426] - See link
      • ACTION ITEM FOR SITES:
        • Sites should please prepare their local Squid cache installations for updating as soon as possible.
        • when configuring the squid please add the following networks as to be allowed to connect to your site squid:
          • your internal cluster network(s)
          • 192.12.15.0/255.255.255.0 for the BNL testing network to be confirmed
          • 206.12.25.0/255.255.255.128 for CA-ALBERTA-WESTGRID-T2
          • 142.150.188.0/255.255.255.0 for CA-SCINET-T2 to be confirmed
          • 206.12.48.0/255.255.255.0 for CA-VICTORIA-WESTGRID-T2 to be confirmed
          • 206.12.24.0/255.255.255.0 for SFU-LCG2
          • 142.104.21.0/255.255.255.128 for VICTORIA-LCG2
          • 206.12.1.0/255.255.255.0 for TRIUMF-LCG2
          • 192.231.127.0/255.255.255.0 for Australia-ATLAS
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100504]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2)
    • SFU-LCG2 ERROR (bugaboo CE) and WARN (snowpatch CE); local squid is OK; backup squid at Victoria fails
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (new PoolCondPFC installed to pickup the Frontier configuration; local Squid installation still shows errors but backup is OK)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

  • draft April WLCG Tier2 Reliability and Availability report issued (20100511)

wlcg-avail-table-header-20100511.png wlcg-avail-table-aus-20100511.png wlcg-avail-table-ca-20100511.png

APEL Accounting

wlcg-accounting-table-header-20100511.png wlcg-accounting-table-aus-20100511.png wlcg-accounting-table-ca-20100511.png

  • ACTION ITEM FOR SITES: Sites please confirm their historical APEL data that has been published looks correct when comparing with batch system utilization
    • See example steps circulated by D. Qing (20100406)
    • Confirmed checks performed to date have been done at: TRIUMF-LCG2, SFU-LCG2, CA-SCINET-T2 (20100504)
  • See EGEE Accounting Portal
    • Note: some information is not currently available via the portal or if it is there it may not be correctly displayed [20100511]

  • NOTE:
    • this week the APEL status information and results are omitted due to some recently observed changes within the APEL accounting portal, specifically:
      • 1) "Countries" view is now reporting "This COUNTRY has no accounting data available for the options selected!". I see this error message for nearly all countries (including CA) except for Croatia, Italy, Russia, UK and US.
      • 2) the "EGEE --> Production --> ROC_Canada" view is also empty, along with the sites directly under this branch giving the error listed above in 1).
      • 3) the "Tier2" view for Canada now shows CA-EAST-T2 and CA-WEST-T2 as per how out WLCG commitments are organized. However, the CA-WEST-T2 only currently lists CA-ALBERTA-WESTGRID-T2. The SFU-LCG2 and VICTORIA-LCG2 sites are listed at the same level as East and West within CA, and so not counted as part of the CA-WEST-T2 federation.
    • to fixing item 3) I will request that the SFU-LCG2 and VICTORIA-LCG2 sites be added into CA-WEST-T2.
    • I'll also submit a ticket regarding problems 1) and 2) since having the full CA cloud site view is useful.

APEL Publishing Status

  • omitted this week (see above note)

APEL Accounting Results

  • omitted this week (see above note)

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517 and subsequently suspended by CA_ROC until site re-certification occurs

  • CA-ALBERTA-WESTGRID-T2 site status [20100511]
    • APEL publishing to be corrected; already investigating to confirm that accounting info is at least reaching the local mySQL database
    • APEL statistics to be validated once publishing error is resolved
    • information system update
      • published value of GlueHostArchitectureSMPSize to be corrected during next yaim configuration
      • with next yaim configuration will also correct incorrect userid assigned to atcan006
    • additional open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • network cabling maintenance planned for lighpath connection [postponed from 20100319]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • VICTORIA-LCG2 site

Outstanding Issues

  • ADCoS shifter Savannah bug ticket 65596 issued [20100410]
    • Too little space left on local disk to run job
    • investigating to restrict the maximum numer of production and/or analysis jobs on the 4 cpu blades which share 29 GB of local disk space
    • solution being implemented and tested by D. Leske [20100416]

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

Outstanding Issues

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:
    • https://gus.fzk.de/ws/ticket_info.php?ticket=57909 Task 132887 fails with get error at TRIUMF-LCG2 (20100503) (verified 20100506)
    • https://gus.fzk.de/ws/ticket_info.php?ticket=57695 Release 15.6.4.1 missing for tasks 131413 and 131405 at TRIUMF-LCG2 (20100427) (unsolved)
    • https://gus.fzk.de/ws/ticket_info.php?ticket=57021 AtlasProduction/15.6.6.5 not found in CA/TRIUMF (20100404) (assigned)
    • https://gus.fzk.de/ws/ticket_info.php?ticket=56849 missing files at TRIUMF-LCG2 (20100330) (in progress)

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status report from Simon via email (20100511) 
I was in and out of the the office last week settling my famly after they arrived, 
hence not so much done for last week. However my hands are free now to 
have them both on the site configuration. I should be able to get the 'test" site 
(if i may call it so) online by end of May. I will keep you up to date.

concerning construction:

   - A company, TEQ, is now contracted to do the building. Basically the ball is 
now i this company's court. They have established the subcontracts and building 
should begin soon. No date mentioned but this is expected to begin this month as 
mentioned earlier.
   - For the systems, the Mcgill lawyers committee are still reviewing meaning it 
hasn't gone out to the vendors yet.

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
    dashboard-ca-20100511.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-all-20100511.png

  • Alternate Views (See link for last 3 days only):
    panda-dashboard-all-20100511.png

  • CA Cloud (last 3 days):
    panda-dashboard-ca-20100511.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard: 

name,rank,value
TRF_SEGVIO ,1,879
EXEPANDA_Release_MISSING,2,594
EXEPANDA_DQ2_STAGEIN,3,76
TRFERROR,4,27
EXEPANDA_DQ2PUT_FILECOPYERROR,5,21
TRF_UNKNOWN ,6,18
EXEPANDA_GET_NOSUCHFILE,7,13
EXEPANDA_JOBKILL_BYPILOT,8,12
TRF_SEGFAULT ,9,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,10,6
EXEPANDA_GET_SIZEMISMATCH,11,2
EXEPANDA_JOBKILL_NOLOCALSPACE,12,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,13,1

    • VICTORIA-LCG2 
name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,8
TRFERROR,2,1
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,3,1
EXEPANDA_DQ2GET_INFILE,4,1

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRFERROR,1,5
EXEPANDA_DQ2PUT_FILECOPYERROR,2,3
EXEPANDA_JOBKILL_SIGTERM,3,2
EXEPANDA_GET_SIZEMISMATCH,4,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,1

    • Australia-ATLAS 
name,rank,value
TRF_SEGVIO ,1,5
TRFERROR,2,4
EXEPANDA_DQ2PUT_FILECOPYERROR,3,1
TRF_UNKNOWN ,4,1

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes:
    • week (May 3 - May 10)
      • active tasks from PandaDB: 161
        • distribution by cloud: CA: 18 CERN: 6 DE: 17 ES: 3 FR: 14 IT: 7 ND: 3 NL: 13 TW: 3 UK: 6 US: 71 

1) Tue May 4
 + CA/CA-SCINET-T2 dq2-get/lcg-cp f.GGUS 57953 solved:Services restarted.

4) Fri May 7
- CA/TRIUMF MCDISK FT f./source file doesn't exist.GGUS 58059,56849 
   updated,in progress.Savannah 65122:got list of files registered in LFC
   ,but missing on dCache(del.Mar.24-Apr.29/FTS upgrade).Recovery not 
   done yet:correction is on-going.

5) Sat May 8
 - CA/CA-SCINET-T2 t.118006 job f./lost heartbeat.GGUS 58066 assigned.

7) Mon May 10
 - CA/CA-SCINET-T2 many job f./Get/Put:Bad magic number.GGUS 58116 
   assigned.

-- BryanCaron - 10 May 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf ATLAS_Schedule-May10.pdf r1 manage 40.5 K 2010-05-11 - 05:39 BryanCaron  
PNGpng dashboard-all-20100511.png r1 manage 951.2 K 2010-05-11 - 17:22 BryanCaron  
PNGpng dashboard-ca-20100511.png r1 manage 424.7 K 2010-05-11 - 17:21 BryanCaron  
PNGpng panda-dashboard-all-20100511.png r1 manage 1892.3 K 2010-05-11 - 17:10 BryanCaron  
PNGpng panda-dashboard-ca-20100511.png r1 manage 688.8 K 2010-05-11 - 17:10 BryanCaron  
PNGpng wlcg-accounting-table-aus-20100511.png r1 manage 107.4 K 2010-05-11 - 17:34 BryanCaron  
PNGpng wlcg-accounting-table-ca-20100511.png r1 manage 313.0 K 2010-05-11 - 17:34 BryanCaron  
PNGpng wlcg-accounting-table-header-20100511.png r1 manage 170.6 K 2010-05-11 - 17:33 BryanCaron  
PNGpng wlcg-avail-table-aus-20100511.png r1 manage 164.8 K 2010-05-11 - 17:42 BryanCaron  
PNGpng wlcg-avail-table-ca-20100511.png r1 manage 463.5 K 2010-05-11 - 17:43 BryanCaron  
PNGpng wlcg-avail-table-header-20100511.png r1 manage 197.5 K 2010-05-11 - 17:42 BryanCaron  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2010-05-12 - RyanTaylor
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback