TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100615 (2010-06-16, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-06-15

  • NOTE: unless indicated otherwise the report covers the last 3 weeks of activity

General News

  • CA Cloud Squad Actions [20100615]
    • siteTests updated [20100614]
      • now includes new site VICTORIA-WG corresponding to CA-VICTORIA-WESTGRID-T2 analysis queue
    • update to schedconfig to reduce value of maxinputsize from 14 GB to 8 GB for VICTORIA-LCG2 queues to match smaller available disk space local to the WNs [20100611]
    • ROOTSYS truncated at 127 characters issue [20100608]
      • http://savannah.cern.ch/bugs/?68546 (open)
      • currently affecting Victoria (Tier3) and CA-SCINET-T2
      • varying character lengths below and above 127 characters show variable success and failure
    • DDM subscriptions [20100611]
      • datasets to MCDISK at CA-VICTORIA-WESTGRID-T2 and CA-ALBERTA-WESTGRID-T2 (input for siteTests and GangaRobot)
    • transition of TRIUMF pilot factories to new autopilot factory [20100608]
    • new site of CA-VICTORIA-WESTGRID-T2 added to central ATLAS ToA configuration [20100609]
    • panda queues created for CA-VICTORIA-WESTGRID-T2 [20100609]
    • ANALY_ALBERTA-WG1 queue marked online after successful testing after site SCRATCHDISK became full [20100611]
    • Open Issues and Ongoing Discussions:
      • user build jobs failing at CA-SCINET-T2 [20100511]
        • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
        • this is related to the ROOTSYS truncation issue identified above and detailed in http://savannah.cern.ch/bugs/?68546
      • removal of CA-ALBERTA-WESTGRID-T2 in manual GangaRobot blacklist [20100511]
        • new subscription of "mc0*.merge.AOD.e*_r*" dataset required to MCDISK but initial subscription resulted in incorrect dataset naming
          • UPDATE (20100611): new subscriptions successfully completed via DaTRI without previous dataset name change
          • site removed from manual GangaRobot blacklist [20100615]
      • submitted GGUS in regards to SFU-LCG2 site storage information missing from CERN top-level bdii (lcg-bdii.cern.ch) [20100513]
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • ATLAS US VOMS server information [20100615]
    • ACTION ITEM FOR SITES: Sites please confirm if you have added support for this VOMS server (done for TRIUMF-LCG2, SFU-LCG2, CA-SCINET-T2, VICTORIA-LCG2, CA-ALBERTA-WESTGRID-T2)
    • vo.racf.bnl.gov is another ATLAS VOMS server at BNL supported by default in gLite for SL4 but not for SL5
    • sites should check for the appropriate .lsc file in the SL5 WN installation
    • file content (vo.racf.bnl.gov.lsc): 
/DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov
/DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1
    • for SL4 CE using yaim the following should be added to site-info.def prior to running yaim configuration step: 
atlas vo.racf.bnl.gov 15003 /DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov 24' into VO_ATLAS_VOMSES
DC=org/DC=DOEGrids/OU=Certificate Authorities/CN=DOEGrids CA 1' into VO_ATLAS_VOMS_CA_DN

  • Squid Deployment [20100518]
    • New Squid rpm release announced [20100426] - See link
      • ACTION ITEM FOR SITES:
        • Sites should please prepare their local Squid cache installations for updating as soon as possible, and confirm when the updated version is in place (Australia-ATLAS, CA-SCINET-T2 done)
        • when configuring the squid please add the following networks as to be allowed to connect to your site squid:
          • your internal cluster network(s)
          • 192.12.15.0/255.255.255.0 for the BNL testing network
          • 206.12.25.0/255.255.255.128 for CA-ALBERTA-WESTGRID-T2
          • 142.150.188.0/255.255.255.0 for CA-SCINET-T2
          • 206.12.48.0/255.255.255.0 for CA-VICTORIA-WESTGRID-T2
          • 206.12.24.0/255.255.255.0 for SFU-LCG2
          • 142.104.21.0/255.255.255.128 for VICTORIA-LCG2
          • 206.12.1.0/255.255.255.0 for TRIUMF-LCG2
          • 192.231.127.0/255.255.255.0 for Australia-ATLAS

    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100518]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2)
    • SFU-LCG2 WARN (both CEs); local squid is OK; backup squid at Victoria fails
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (both for local and connection to SFU squid; see error )
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

avail-reliab-headers-20100615.png avail-reliab-aus-20100615.png avail-reliab-ca-20100615.png

  • ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: investigate cause of lower measure of availability and reliability for May 2010

APEL Accounting

accounting-header-20100615.png accounting-ca-20100615.png

  • ACTION ITEM FOR SITES: Sites please confirm their historical APEL data that has been published looks correct when comparing with batch system utilization
    • See example steps circulated by D. Qing (20100406)
    • Confirmed checks performed to date have been done at: TRIUMF-LCG2, SFU-LCG2, CA-SCINET-T2, Australia-ATLAS (20100525)
  • See EGEE Accounting Portal

  • ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: correct APEL publishing backlog, correct advertised cpu capability (DONE 20100615)

APEL Publishing Status

apel-status-aus-20100615.png apel-status-ca-20100615.png

APEL Accounting Results

apel-statistics-aus-20100615.png apel-statistics-ca-20100615.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517 and subsequently suspended by CA_ROC until site re-certification occurs

  • CA-ALBERTA-WESTGRID-T2 site status [20100525]

Outstanding Issues

  • CA-ALBERTA-WESTGRID-T2 site status [20100511]
    • APEL publishing to be corrected; already investigating to confirm that accounting info is at least reaching the local mySQL database
    • APEL statistics to be validated once publishing error is resolved
    • information system update
      • published value of GlueHostArchitectureSMPSize to be corrected during next yaim configuration (DONE 20100615)
      • with next yaim configuration will also correct incorrect userid assigned to atcan006 (DONE 20100615)
      • will also correct published cpu resources to reflect NRAC allocation and WLCG MoU committment (DONE 20100615)
    • additional open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]
    • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • VICTORIA-LCG2 SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • large number of LFC errors continue this past week (see Outstanding Issues)

Outstanding Issues

  • build job issues and updated image deployment for missing rpms [20100512]
    • problem still present following missing rpm installation [20100518]
      • release installation recommended for 15.6.9 and 15.6.9.4 (action to JSteele/BCaron)

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • https://savannah.cern.ch/support/?114503 TRIUMF-LCG2 excluded as source in DDM transfers for CERN, ASGC, CNAF clouds (20100517)
    • TRIUMF actions successfully completed and now action is upon CNAF to update their FTS services in order to remove blacklist (20100911)
      • CNAF FTS upgrade to v 2.2.4 scheduled for 2010-06-17

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

Production

Summary

CA Cloud Statistics Reference Link (Last 3 Weeks) Overall Statistics Reference Link (Last 3 Weeks)

  • Within the CA Cloud:
    dashboard-ca-20100615.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-all-20100615.png

  • Alternate Views (See link for last 3 days only):
    panda-dashboard-all-20100615.png

  • CA Cloud (last 3 days):
    panda-dashboard-ca-20100615.png

Failed Jobs

  • From Panda Monitor (last week):

panda-dashboard-errors-australia2-20100515.png panda-dashboard-errors-alberta2-20100515.png panda-dashboard-errors-sfu2-20100515.png panda-dashboard-errors-triumf2-20100515.png panda-dashboard-errors-victoria2-20100515.png

  • From Dashboard (last 3 weeks): 

name,rank,value
TRFERROR,1,1022
EXEPANDA_DQ2PUT_FILECOPYERROR,2,327
TRF_UNKNOWN ,3,210
EXEPANDA_GET_NOSUCHFILE,4,175
EXEPANDA_DQ2_STAGEIN,5,76
TRF_SEGFAULT ,6,50
EXEPANDA_Release_MISSING,7,24
EXEPANDA_DQ2_SERVERERROR,8,19
EXEPANDA_JOBKILL_BYPILOT,9,15
EXEPANDA_JOBDISPATCHER_HEARTBEAT,10,10
EXEPANDA_GET_SIZEMISMATCH,11,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,12,4
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,13,2

    • VICTORIA-LCG2 
name,rank,value
EXEPANDA_DQ2_STAGEIN,1,231
EXEPANDA_DQ2GET_INFILE,2,6
TRFERROR,3,6
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,5
EXEPANDA_DQ2PUT_FILECOPYERROR,5,4
EXEPANDA_GET_GLOBUSSYSTEMERROR,6,3
EXEPANDA_PUT_LFCMODULEIMPORT,7,1
EXEPANDA_PUT_GLOBUSSYSTEMERROR,8,1
EXEPANDA_JOBKILL_NOLOCALSPACE,9,1

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRFERROR,1,27
EXEPANDA_GET_SIZEMISMATCH,2,11
EXEPANDA_JOBKILL_NOLOCALSPACE,3,3
EXEPANDA_DQ2PUT_FILECOPYERROR,4,1
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,1

    • Australia-ATLAS 
name,rank,value
EXEPANDA_DQ2_STAGEIN,1,1486
EXEPANDA_DQ2PUT_FILECOPYERROR,2,260
TRFERROR,3,52
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,25
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,16
EXEPANDA_DQ2GET_INFILE,6,10
EXEPANDA_GRID-PROXY_INVALID,7,6
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,8,1

You can find the meanings of the transform error codes on this link.

  • Yuri's notes:
    • week (May 25 - May 31)
      • active tasks from PandaDB: 109
        • distribution by cloud: CA:4 CERN:0 DE:10 ES:7 FR:20 IT:5 ND:7 NL:12 TW:3 UK:14 US:27

    • week (June 1 - June 7)
      • active tasks from PandaDB: 265
        • distribution by cloud: CA:13 DE:54 ES:13 FR:40 IT:15 ND:23 NL:14 TW:1 UK:23 US:69

    • week (June 8 - June 14)
      • active tasks from PandaDB: 123
        • distribution by cloud: CA:5 CERN:1 DE:3 ES:3 FR:1 IT:1 ND:8 NL:21 UK:2 US:78 

2) Wed May 26
 + CA/VICTORIA-LCG2 FT f./SOURCE er./certificate has been revoked.GGUS
   58502 verified:host cert.expired and was replaced.

5) Sat May 29
 - CA/CA-SCINET-T2 FT,job f./HTTP_TIMEOUT,SRM/stage-out er.GGUS 58615
   assigned:SRM space manager died,restarted.Test jobs failed May 31.
 + CA/Australia-ATLAS 200 job f./Put er.:registration failed.GGUS 58616
   solved:added pilot role into the ACL for /grid/atlas/dq2/prod_test/
   and its subdirectories.

3) Thu Jun.3
 - CA/CA-SCINET-T2 many job f./lfc_getreplicas failed with:2704, Bad magic 
   number.GGUS 58116 in progress updated.

5) Sat Jun.12
+ CA/Australia-ATLAS 237 job f./Get er.:BDII No entries for host.GGUS
   58991,58999 assigned:302 job f.Jun.14:BDII locked up(looks DOSed)?

6) Sun Jun.13
 - CA/CA-SCINET-T2 >600 job f./lfc_getreplicas failed.Bad magic number.
   GGUS 58116 in progress updated.Ticket opened since May 10.

-- BryanCaron - 14 Jun 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng accounting-ca-20100615.png r1 manage 24.4 K 2010-06-15 - 04:55 BryanCaron  
PNGpng accounting-header-20100615.png r1 manage 23.6 K 2010-06-15 - 04:55 BryanCaron  
PNGpng apel-statistics-aus-20100615.png r1 manage 79.8 K 2010-06-15 - 05:01 BryanCaron  
PNGpng apel-statistics-ca-20100615.png r1 manage 151.4 K 2010-06-15 - 05:02 BryanCaron  
PNGpng apel-status-aus-20100615.png r1 manage 14.9 K 2010-06-15 - 05:01 BryanCaron  
PNGpng apel-status-ca-20100615.png r1 manage 24.1 K 2010-06-15 - 05:01 BryanCaron  
PNGpng avail-reliab-aus-20100615.png r1 manage 13.5 K 2010-06-15 - 04:42 BryanCaron  
PNGpng avail-reliab-ca-20100615.png r1 manage 41.0 K 2010-06-15 - 04:42 BryanCaron  
PNGpng avail-reliab-headers-20100615.png r1 manage 16.5 K 2010-06-15 - 04:42 BryanCaron  
PNGpng dashboard-all-20100615.png r1 manage 962.6 K 2010-06-15 - 14:45 BryanCaron  
PNGpng dashboard-ca-20100615.png r1 manage 494.6 K 2010-06-15 - 14:45 BryanCaron  
PNGpng panda-dashboard-all-20100615.png r1 manage 1828.5 K 2010-06-15 - 14:24 BryanCaron  
PNGpng panda-dashboard-ca-20100615.png r1 manage 730.0 K 2010-06-15 - 14:25 BryanCaron  
PNGpng panda-dashboard-errors-alberta2-20100515.png r1 manage 1198.1 K 2010-06-15 - 15:13 BryanCaron  
PNGpng panda-dashboard-errors-australia2-20100515.png r1 manage 785.7 K 2010-06-15 - 15:13 BryanCaron  
PNGpng panda-dashboard-errors-sfu2-20100515.png r1 manage 491.7 K 2010-06-15 - 15:14 BryanCaron  
PNGpng panda-dashboard-errors-triumf2-20100515.png r1 manage 924.1 K 2010-06-15 - 15:15 BryanCaron  
PNGpng panda-dashboard-errors-victoria2-20100515.png r1 manage 563.2 K 2010-06-15 - 15:15 BryanCaron  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2010-06-16 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback