Tier1 and Tier-2 Weekly Report 2012-01-10

General News

• ADC and ATLAS Computing News [20120110]
  • ADC Weekly meeting
  • Most sites functioned well over the holidays - about 100k+ jobs per day
  • Some issues with group production overwhelming MC reprocessing at some T1s (TRIUMF mentioned). Wanted 20/80 split. Also T2 not contributed as much as expected due to T1 weights (reduce data transfers and limit PRODDISK usage). Lowered weights (CA included) and helps but brokerage is still slow.
    • Possible mitigation scenarios discussed and hope to tweak this within the current tools rather than moving data by hand
  • FTS 2.8.8 under validation at CERN (needed for Oracle 11g switch) - expect full deployment soon after some more testing (T0-T1)
  • Also using gsiftp/xrdcp for EOS directly rather than SRM. Load lighter but some issues with surls etc to be resolved.
  • Hammercloud issue lat week due to expired proxy. AMOD is not included in the emails. Squads asked to respond.
  • One of the issues of getting sites online again automatically was not full list of datasets available. Will change dq2 to retrieve list of datasets.
  • Slots open from Feb for AMOD shifters
  • New DDM Data Subscription Monitoring page in SSB
    • http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteview#currentView=DDM_subscriptions&highlight=false
    • Monitor DATADISK, GROUPDISK, LOCALGROUPDISK subscriptions (DATATAPE to be added)
    • OK if subscriptions < 100, WARNING for 100-300 and ALERT for >300.
    • Not ranking sites on this but cloud squads asked to monitor
    • Discussion about improvements to this monitoring (e.g. draining rate is more important perhaps than absolute etc)
• CA Cloud Squad Actions [20120110]
  • DaTRI requests approved / processed (last week)

• • Declared Downtimes
    • CA-ALBERTA-WESTGRID-T2 SCHEDULED downtime (OUTAGE) [28-12-2011 23:30 to 04-01-2012 23:30 UTC] - chilled water outage
  • ATLAS Site Exclusion Savannah entries [20110621]
    • HammerCloud autoexclusions
      • ANALY_SFU Auto-Excluded at Tue Jan 3 14:03:31 2012 CET
      • ANALY_MCGILL Auto-Excluded at Tue Jan 3 14:03:32 2012 CET
      • ANALY_VICTORIA-WG1 Auto-Excluded at Tue Jan 3 14:03:39 2012 CET
      • ANALY_SCINET Auto-Excluded at Tue Jan 3 14:33:30 2012 CET
      • ANALY_AUSTRALIA Auto-Excluded at Tue Jan 3 14:33:34 2012 CET
      • ANALY_SFU_bugaboo Auto-Excluded at Tue Jan 3 15:02:37 2012 CET
      • ANALY_TRIUMF Auto-Excluded at Tue Jan 3 15:02:44 2012 CET
      • ANALY_ALBERTA-WG1 reset online at Thu Jan 5 02:00:41 2012 CET
  • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
    • 
• Open Issues and Ongoing Discussions
  • Sites should be providing the April 2011 storage pledge. [see 20110315 report.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/
  • HammerCloud [20110531]
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20120110]

• gLite News [20120110]

• ROC_Canada News [20120110]
  • Current shift - Neil Knecht 9-13 Jan
  • Next Week - ?

• WLCG and General Computing News
• 
• 

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

• site service status via Gridview - Please check your site

• January 2012 Detailed Site Information (valid until 20120131):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2012-01-10%2020:59:59&start-date=2012-01-03%2020:00:00

   World-wide weekly ATLAS ADCoS summary (Jan.3-9,2012)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past 2 weeks (Dec.27-Jan.9) Panda production service
   - completed 2,983,151 managed group, MC production, validation and reprocessing jobs
   - average ~213,082 jobs per day
   - failed 166,217 jobs
   - average efficiency:
      -- jobs  ~94.7%
   - active tasks:
      -- distribution by cloud: 3130
  CA:222 CERN:64 DE:562 ES:324 FR:867 IT:94 ND:145 NL:190 TW:57 UK:238 US:367

B)
   - UK/RAL-LCG2 DT (CE OUTAGE:lcgce07-9 unavailable.),Jan.4-5.
   - UK/RAL-LCG2 DT (FTS,SE,SRM OUTAGE:CASTOR ORACLE DB migration),Jan.5.
   - ES/PIC DT (WMS,FTS warning,Savannah 125516),Jan.9.
   - CERN/Elog maintenance, restart at 15:10 CET, Jan.9.
   - CERN/old Panda Server nodes (voatlas57,58,59)removed,no support.Jan.9

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)

1) Tue Jan.3 
- TW/TAIWAN-LCG2_DATADISK->CA-SCINET-T2 67 FT f./SOURCE file doesn't exist
  Savannah 90206:There is a long list fo datasets to be sent to CA-SCINET_
  PHYS-TOP.It is why it takes time and the pinning needs to be renewed. 

4) Fri Jan 6
+ CA/CA-MCGILL-CLUMEQ-T2 job f./HwSusy.so:can't open shared object file:No
  such file or dir.GGUS 77924 verified:seems general CVMFS issue,not site-
  related,see Savannah 90241. 

 5) Sat Jan 7
+ CA/CA-SCINET-T2 800 job f./Get er.:file with GUID..not found.GGUS 77950
  solved:files not repicated,most "lost hesrtbeat" job f.not related.

-- LeslieGroer - 2012-01-10