Tier1 and Tier-2 Weekly Report 2011-06-21

General News

• ADC and ATLAS Computing News [20110620] https://indico.cern.ch/conferenceDisplay.py?confId=143804
  • LHC delivered over 1 pb^-1 to ATLAS and CMS
  • LHC Machine studies next week and Technical stop July 4-8. Time for interventions at T1's if necessary.
  • FTS 2 developments coming - no developments for 1.5 years
  • New CVMFS server at CERN - necessary to support more sites
• CA Cloud Squad Actions [20110607]
  • DaTRI requests approved / processed (last week)

SITE	NumRequests	NumDatasets	NumSubscribed	SizeofAllDatasets (GB)
AUSTRALIA-ATLAS_LOCALGROUPDISK	3	22	22	28
AUSTRALIA-ATLAS_SCRATCHDISK	9	769	697	1763
CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK	11	17	17	2319
CA-SCINET-T2_PHYS-TOP	2	2	2	0.07
SFU-LCG2_LOCALGROUPDISK	7	7	7	492
TRIUMF-LCG2_LOCALGROUPDISK	14	417	414	9493
TRIUMF-LCG2_PERF-TAU	3	984	984	11521

• • Declared Downtimes
    • SFU-LCG2/ SCHEDULED downtime (OUTAGE) [27-06-2011 14:00 to 01-07-2011 18:00 UTC ] - UPS upgrade
    • CA-SCINET-T2/ UNSCHEDULED downtime (OUTAGE) [21-06-2011 15:10 to 21-06-2011 22:00 UTC] - Chiller failure.
  • ATLAS Site Exclusion Savannah entries [20110621]
    • HammerCloud autoexclusions
      • ANALY_SFU Auto-Excluded at Wed Jun 15 06:00:03 2011 CET
      • ANALY_SFU reset online at Wed Jun 15 19:00:03 2011 CET
  • other cloud squad related savannah and / or ROC ggus tickets [20110607]
    • !PD2P (dynamic data placement) in CA cloud [20110607]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)

• Open Issues and Ongoing Discussions
  • Sites should be providing the April 2011 storage pledge. [see 20110315 report.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/
  • CREAM CE deployment preparations
    • TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20110426]
  • HammerCloud [20110531]
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20110621]
  • [20110524] UVic and Toronto participated in SSC5 (23 May - 03 June 2011) - final reports to be sent to SSC5
  • [20110616] EGI CSIRT ADVISORY [EGI-ADV-20110615-02] Vulnerability for LRMS (details to be posted later)

• gLite News [20110531]
  • [20110504] [ https://operations-portal.egi.eu/broadcast/archive/id/378 ][gLite 3.2 WN release] has an important known issue detailed here: http://glite.cern.ch/glite-WN/known_issues

• ROC_Canada News [20110621]
  • Current shift - Reda Tafirout June 20-24
  • Next Week - Sergey Chelsky

• Squid Deployment [20110329]
  • CA cloud Frontier and Squid deployment information (versions and network information)
    • Frontier-Squid Monitoring http://frontier.cern.ch/squidstats/indexatlas.html
  • SAM tests: (CA Link), (Aus Link):

• General Computing News

• SONAR Tests [20110426]
  • Performance monitored at http://bourricot.cern.ch/dq2/ftsmon/sonar_view/cached/ with historical views at http://bourricot.cern.ch/dq2/ftsmon/ . Canadian cloud now tested on Fridays. All CA Tier-2's are tested with small (20MB) and medium (200MB) files, but only TRIUMF, CA-SCINET-T2 and CA-VICTORIA-WESTGRID-T2 are currently tested with large (2GB) files outside of the cloud.

SAM Results

• site service status via Gridview - Please check your site

• June 15-21, 2011

• June 2011 Detailed Site Information (valid until 20110631):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

Outstanding Issues

TRIUMF

Recent Issues

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2011-05-24%2020:59:59&start-date=2011-05-31%2020:00:00

Failed Jobs

• From Panda Monitor (last day)

You can find the meanings of the transform error codes on this link.

• My Notes:

• Highlight's of Yuri's notes: (May 31-June 6, 2011)

   World-wide weekly ATLAS ADCoS summary (Jun.14-20,2011)

I. General summary:
   ---------------

A)  During the past week (Jun.14-20,2011) Panda production service
   - completed 1,576,318 managed group, MC production, validation
     and reprocessing jobs
   - average ~225,188 jobs per day
   - failed 56,194 jobs
   - average efficiency:
      -- jobs     ~96.6%
   - active tasks from Panda-monitor: 430
      -- distribution by cloud:
      CA:6 CERN:40 DE:14 ES:12 FR:134 IT:29 ND:28 NL:28 TW:4 UK:33 US:102

B) - CERN/CERN-PROD downtime (SE,SRM warning/OUTAGE), Jun.14,20.
    - New DB Release 15.8.1 subscription done, Jun.20.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    -------------------------------------------------

1) Tue Jun.14

- CA/AUSTRALIA-ATLAS 1000s (eff.71%) FT f./SOURCE:VOMS error when
  processing cert.GGUS 71502 oh hold,updated:there was an errant .lsc
  file in the vomsdir on the SE and 2 of the 12 disk servers had an extra
  legacy lsc for altas.No err.on Jun.20.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
    (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)

++ 16.6.2.6 data11_7TeV Reco merge task 423378 failures: ATH_FAILURE -
 Athena non-zero exit.Savannah 83387.Group production LAr.All jobs failed
 10 attempts in CA.Task ABORTED.Similar to #83386 below

++ 16.6.4.1 data11_7TeV Reco merge task 423381 failures: TileCellToTTL1
 ERROR unable to retrieve Cell container.Savannah 83386.Group production
 TRIGGER L1Calo.All jobs failed 10 up to attempts in CA cloud.Understood:
 needs ESD as input,instead of AOD.Task ABORTED.


 

• Stats:
  • See Key for what the headers mean

-- LeslieGroer - 2011-06-21