Tier1 and Tier-2 Weekly Report 2013-07-09

General News

• Availability and Accounting reports

• ADC and ATLAS Computing News [20130709]
  • ADC Weekly meeting
  • AMOD Report
    • DDM moving about 0.5PB a day
    • DDM dashboard stuck at one point - fixed
    • RAL pointed out bug in CVMFS 2.1.11.
      • Fix 2.1.12. Follow up with Task Force
    • Proposal for change of analysis queues draining (pointed out by Alastair@RAL)
      • Set brokeroff 8 hrs (currently is: 6 hrs) in advance of a downtime
      • Set offline 4 hrs (currently is: 2hrs)
    • FZK had GPFS issue affecting 1/3 of pools
    • Free space on T2 DATADISKs getting critical
      • http://voatlas161.cern.ch/monitoring/global_storage/T2/table.html
      • Deletion of secondaries, reshuffling of primaries
    • During BNL file recovery
      • AK47 AOD distributions stopped on Thursday
      • Resumed yesterday
      • PD2P stopped for critical sites
      • Deletion stopped from US T2 PRODDISKs
• BNL File Recovery
  • Recovery prototype based on procedure for extending MC samples:
    • Good: several datasets from the same MC sample are registered in the same container
    • Bad: MC samples are extended from the beginning of the chain of tasks, i.e. from evgen
    • Ugly: new production step .recov. avoids data recovery from the beginning of the chain
      • The .recov. dataset is similar to Tier-­0 datasets, which are external to the production system
    • Attempted procedure on 5 merged AOD files in delayed B-­Physics processing
      • Both TAG datasets were registered in the same container
      • Production system was unable to process LIST request for TAG merging since the re\x{2010}produced merged TAG file had non-consecutive sequence number
    • Thus, use of an external dataset (i.e. unrelated to the original task) did not work
      • Because of that, the recovery task must extend the whole chain of tasks with lost files
    • A general recovery procedure requires keeping track of the lumiblocks It is not clear yet how to handle lumiblocks in the prototyped procedure
    • Recoverable and lost files are known
      • Production coordinators should examine the affected datasets and decide on the actions
        • Recover lost files (?)
        • Rerun the tasks
        • Ignore and keep the incomplete datasets
      • Lost files in DATADISK
        • ~450k in total
        • ~107k recoverable, ¾ are really lost (no replicas)
        • About ½ are log files, most of them can be ignored
        • 689 HITS tasks affected,where merging cannot be done (but it might be done already)
        • 23,978 tasks affected in total
        • 31,789 datasets with lost files (including logs)
          • 9425 data datasets
          • 22364 log datasets
      • Recoverable:
        • 20,260 data datasets
        • 88 log datasets (log datasets not replicated)
      • Most heavily affected projects:
        • mc12_8TeV
        • data12_8TeV
        • data09_900GeV
        • mc11_7TeV
        • groupmc08
      • Only 715 big datasets affected at >1% level

• CA Cloud Squad Actions [20120626]
  • User DaTRI requests approved / processed (last week)
  • Declared Downtimes

• • HammerCloud autoexclusions

• • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
• Open Issues and Ongoing Discussions
  • Sites should be providing the 2013 storage pledge.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/ http://gstat-wlcg.cern.ch/apps/pledges/resources
  • HammerCloud [20120214]
    • [20120214] Ongoing tests with McGill https://twiki.atlas-canada.ca/bin/view/AtlasCanada/McGill_Feb2012
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20120801]

• gLite/EMI News [20121127]
  • [20121029] UMD v.2.2.1 has been released
  • [20120924] Retirement calendar of gLite 3.1 and currently unsupported gLite 3.2 products
  • [20120911] EMI-2 WN & UI rpms (SL5/x86_64) beta testing version available

• ROC_Canada News [20130625]
  • Current shift - Erming Pei June 24-28, 2013
  • Next Week - Leslie Groer

• WLCG and General Computing News [20120911]

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

• site service status via Gridview - Please check your site

• April 2012 Detailed Site Information (valid until 20120430):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-MCGILL-CLUMEQ-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2013-02-01%2020:59:59&start-date=2013-02-07%2020:00:00

-- LeslieGroer - 2013-07-09