Tier1 and Tier-2 Weekly Report 2011-09-20

General News

• ADC and ATLAS Computing News [20110913]
  • ADC Weekly meeting
  • ATLAS operations report to GDB from Ueda I https://indico.cern.ch/getFile.py/access?contribId=6&sessionId=0&resId=0&materialId=slides&confId=106648
  • Memo to GDB from ATLAS on gLExec see gLExec_ATLAS_GDB_Memo.pdf or https://indico.cern.ch/getFile.py/access?contribId=8&resId=0&materialId=0&confId=115406
    • Some versions of Athena hang under gLExec - not fully understood. Also increased instabilities observed.
    • Focus on security questions with stolen proxies
    • "Considering all the above, ATLAS believes that the amount of security that gLexec would bring to the infrastructure does not motivate the operational cost. In addition, while gLexec covers some vulnerabilities, it creates very serious ones in case of compromised credentials. The conclusion is that ATLAS does not intend to invoke gLexec on the worker node"
      • Use ARGUS eventually. Job traceability available through PaNDa.
      • Comment: What about user job that causes problem? Could use callout to ARGUS to check user before running. Being considered.
  • Issue with user jobs requiring 4GB memory.
    • Statement: Sites should publish their memory limits in schedconfig and Panda should broker jobs accordingly
    • Panda server was updated today to ignore sites with memory=0 in schedconfig when job is submitted with memory requirement
    • Questions:
      • Should high memory jobs run only at T2s? If so this should be taken into account in data distribution (for instance ESDs and RAW should be available in T2s if necessary)
      • If input data is available only at T1, would DaTRI work to request replication? If the size is reasonable (O(TB)) DaTRI would work.
      • Could high memory jobs be limited to people with group role=production?
      • Is the use case is fixed and predictable enough to make any hardware request - changes in ATLAS VO card, approval via WLCG?
    • General feeling: Very few real cases where this required so far - go through CREM if necessary to make request and deal with case-by-case.
    • Some believe that high pile-up environment will push us in this direction.
    • Comment by Rod: If specify memory limit then will not get brokered to sites with limit=0. Also pilot sets ulimit which will kill a lot of jobs at 2GB and some at 3GB. Memory has usually not an issue at a site.
    • Conclusion: Not clear if this will be implemented in job matching or in pilot yet as potential to break a lot of jobs.
  • Reprocessing
    • First multi-cloud repro done, FZK taking jobs from NL cloud Tuesday, 20
    • CERN factories patched so pilotcode.tar.gz retrieved via squid:25085, will deploy new factory to non-cern factories.
    • Reprocessing phase II - Periods I,J,K,
      • 370k RAW files in 638 reco tasks (about half of Part I). Started 11th Sep
      • SARA, NIKHEF and PIC had scheduled downtime
  • MC11 G4 status
    • Current status for priority 1 samples:
      • 130M events with wide beam spot done.
      • 110M events with narrow beam spot done (FS+pipe killer)
      • Further 420M events (most of remaining priority 1 samples) submitted last week with the validated setup,40% of these done by today.
      • Speed seems very good (10M-15M events/day); so hopefully we can recover some time.
      • Details in the production wiki: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AtlasProductionGroupMC11a
      • MC11a for priority-1 samples launched last Wednesday, so far 239/646M done (37%). We should be done in 1-2 weeks with the HITS available.
• CA Cloud Squad Actions [20110816]
  • DaTRI requests approved / processed (last week)
  • 

• • Declared Downtimes
  • ATLAS Site Exclusion Savannah entries [20110621]
    • HammerCloud autoexclusions
      • none
  • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • !PD2P (dynamic data placement) in CA cloud [20110830]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
• Open Issues and Ongoing Discussions
  • Sites should be providing the April 2011 storage pledge. [see 20110315 report.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/
  • CREAM CE deployment preparations
    • TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20110426]
  • HammerCloud [20110531]
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20110621]

• gLite News [20110705]
  • [20110705] [https://operations-portal.egi.eu/broadcast/archive/id/420][EGI Trust Anchor release 1.40-1] is now final - please install.
  • [20110504] [https://operations-portal.egi.eu/broadcast/archive/id/378 ][gLite 3.2 WN release] has an important known issue detailed here: http://glite.cern.ch/glite-WN/known_issues
  • [20110627] New IGTF distribution (certificates) and version 3 of the CRL retrieval tool Fetch-crl https://dist.eugridpma.info/distribution/util/fetch-crl3/. Version 1.39 was updated to 1.40 on 2011-06-28.
  • [20110628] [https://operations-portal.egi.eu/broadcast/archive/id/410][gLite 3.2 update 30 release] The highlights of the update are:
    • New version of glite-BDII_top
    • New version of glite-CREAM (to support changes for SGE)
    • New version of glite-LB
    • New version of glite-SGE_utils

• ROC_Canada News [20110809]
  • Current shift -Ashok Argawal 19-24 Sep
  • Next Week - Ryan Taylor

• Squid Deployment [20110329]
  • CA cloud Frontier and Squid deployment information (versions and network information)
    • Frontier-Squid Monitoring http://frontier.cern.ch/squidstats/indexatlas.html
  • SAM tests: (CA Link), (Aus Link):

• General Computing News

• SONAR Tests [20110426]
  • Performance monitored at http://bourricot.cern.ch/dq2/ftsmon/sonar_view/cached/ with historical views at http://bourricot.cern.ch/dq2/ftsmon/ . Canadian cloud now tested on Fridays. All CA Tier-2's are tested with small (20MB) and medium (200MB) files, but only TRIUMF, CA-SCINET-T2 and CA-VICTORIA-WESTGRID-T2 are currently tested with large (2GB) files outside of the cloud.

SAM Results

• site service status via Gridview - Please check your site

• September 1-13, 2011
• September 2011 Detailed Site Information (valid until 20110831):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

Outstanding Issues

TRIUMF

Recent Issues

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2011-05-24%2020:59:59&start-date=2011-05-31%2020:00:00

   World-wide weekly ATLAS ADCoS summary (Sep.13-19,2011)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Sep.13-19,2011) Panda production service
   - completed 2,087,601 managed group, MC production, validation and reprocessing jobs
   - average ~298,229 jobs per day (new weekly record!)
   - failed 145,937 jobs
   - average efficiency:
      -- jobs     - 93.5%
   - active tasks: 1053
      -- distribution by cloud:
    CA:73 CERN:32 DE:203 ES:37 FR:92 IT:47 ND:79 NL:273 TW:3 UK:105 US:109

B) - CERN migration Castor->EOS (some space tokens), Sep.13.
   - CERN ORACLE Streams repl.update ATLAS_CONF_TRIGGER_REPR->T1s, Sep.14
   - new DB Releases 17.0.1 subscribed to sites, Sep.13.
   - new default DQ2 Clients' setup.py script implemented, Sep.15.
   - Phase II of the release-17 reprocessing campaign started, Sep.12.
   - MC11a digi+reco production runs in a final configuration, Sep.14.

3) Thu Sep.15
   -> Cloud/site PROD efficiency ever<50% : CA/CA-SCINET-T2

-- 17.0.3.4 mc11_7TeV Digi+REco task 515195 failures: TRF_SEGFAULT | "out
 of memory".Savannah 86832:MC11 production - TOP.Many (~1300) job failures
 in CA cloud.The same problem with t.515222(MC11 production - TAU) in US
 cloud.Tasks are still "running".No comments published.

• Stats:
  • See Key for what the headers mean