Tier1 and Tier-2 Weekly Report 2013-06-18

General News

• removing data12_8TeV.00209995.physics_JetTauEtmiss.merge.AOD.r4065_p1278_tid01068539_00 from LOCALGROUPDISK as not needed for HC tests
  • 149 files, 4949126 events
  • total size: 1328318810128 (1.3TB)
  • currently pinned at CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK, SFU-LCG2_LOCALGROUPDISK, TRIUMF-LCG2_LOCALGROUPDISK

• ADC and ATLAS Computing News [20130618]
  • ADC Weekly meeting
  • ATLAS SW Releases
    • June 11 - switch from old WMS tagging system to new Panda based ATLAS SW Releases Validation System
    • Follow up on queues that are not successfully validating
  • New procedure for lost files
    • Laborious process to add files to frozen dataset (files can be removed)
    • Implementing new automatic procedure to remove file from dataset, rerun a recovery job and make new recov output dataset that gets added to container
      • e.g. data12_8TeV.00214553.physics_Bphysics.recov.AOD.r4065_000008
      • Fits the workflow of DQ2 much better
      • Some concern if need to recover many jobs as each recovered job currently a separate dataset
  • T1 DATADISKs
    • Filling up at a few sites
      • FZK-LCG2 (48TB free), SARA-MATRIX (77TB free), IN2P3-CC (117TB free), INFN (145TB free)
      • Redistribute mc*AOD* to other T1s
      • IN2P3 added 100TB
      • Estimate, for e.g. at FZK, that add about 20-30TB/week normally and 80TB during reprocessing campaigns
      • In case of sudden crisis: unmerged inputs will be manually deleted
      • Working with CREM to identify data/replication policy that could be changed
    • Storage Area Automatic Blacklisting (SAAB)
      • Activate this week
      • Update the SSB views to include an SAAB column
      • 'SAAB actions' log file to be prepared (similar to panda 'incidents' page)
      • See Twiki: https://twiki.cern.ch/twiki/bin/viewauth/Atlas/StorageAreaAutomaticBlacklisting
    • Migration of Pilot to Rucio naming convention (S. Jezequel)
      • Pilot factories adapted to give possibility to write output files with rucio naming convention (P. Nilsson)
      • Procedure to migrate queues (both prod and analysis queues) :
      • Panda queue updated per site with AGIS interface (add 'rucio' in sepath or seprodpath)
      • Find errors manually from Panda monitoring and rollback if necessary
      • All sites migrated except :
        • US : Wait for feedback on compatibility of US specific tools (Hiro ?)
        • NDGF + arc queues : Request update from Andrej
        • IAAS/NECTAR/CERN-P1 (Cloud ) : Issue with hashlib
        • PSNC/DRESDEN : Problem with hashlib (GGUS issued)
        • CERN : Problematic path (specific code ?)
      • PRODDISK and SCRATCHDISK will be populated with 'rucio' files
        • PRODDISK : No 'pre-rucio' file older than 15 days (if no dark data)
        • SCRATCHDISK : Can take a month or more to purge

• CA Cloud Squad Actions [20120626]
  • User DaTRI requests approved / processed (last week)
  • Declared Downtimes

• • HammerCloud autoexclusions

• • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
• Open Issues and Ongoing Discussions
  • Sites should be providing the 2013 storage pledge.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/ http://gstat-wlcg.cern.ch/apps/pledges/resources
  • HammerCloud [20120214]
    • [20120214] Ongoing tests with McGill https://twiki.atlas-canada.ca/bin/view/AtlasCanada/McGill_Feb2012
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20120801]

• gLite/EMI News [20121127]
  • [20121029] UMD v.2.2.1 has been released
  • [20120924] Retirement calendar of gLite 3.1 and currently unsupported gLite 3.2 products
  • [20120911] EMI-2 WN & UI rpms (SL5/x86_64) beta testing version available

• ROC_Canada News [20130423]
  • Current shift - Sergey Chelsky April 22-26, 2012
  • Next Week - Erming Pei

• WLCG and General Computing News [20120911]

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

• site service status via Gridview - Please check your site

• April 2012 Detailed Site Information (valid until 20120430):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-MCGILL-CLUMEQ-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2013-02-01%2020:59:59&start-date=2013-02-07%2020:00:00

-- LeslieGroer - 2013-06-18