Tier1 and Tier-2 Weekly Report 2011-08-09

General News

• ADC and ATLAS Computing News [20110808]
  • ADC Weekly meeting
  • Change to meeting times - Move ADC ops from Monday to Tuesday 16:00 CERN time.
  • ADC development meeting moves to Mondays 15:30, biweekly.
  • LFC migration - improved performance with Data pump. Testing at SARA during technical stop end of Aug?
  • Some downtimes in worldwide production last week due to ADCR database problem during h/ware migration (Wed) and DDM central catalogue problems (Thu)
  • STORM back-end bug identified - does not crash cleanly. Patch in September.

• CA Cloud Squad Actions [20110809]
  • DaTRI requests approved / processed (last month)

• • Declared Downtimes
    • CA-MCGILL-CLUMEQ-T2 UNSCHEDULED downtime (OUTAGE) [08-08-2011 14:10 to 15-08-2011 00:00 UTC ] - site still down for re-installation and recovery.
  • ATLAS Site Exclusion Savannah entries [20110621]
    • HammerCloud autoexclusions
      • ANALY_ALBERTA-WG1 Auto-Excluded at Fri Aug 5 04:00:05 2011 CET
      • ANALY_VICTORIA-WG1 Auto-Excluded at Sat Aug 6 06:30:04 2011 CET
      • ANALY_ALBERTA-WG1 Auto-Excluded at Tue Aug 9 02:30:04 2011 CET
  • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • !PD2P (dynamic data placement) in CA cloud [20110607]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)

• Open Issues and Ongoing Discussions
  • Sites should be providing the April 2011 storage pledge. [see 20110315 report.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/
  • CREAM CE deployment preparations
    • TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20110426]
  • HammerCloud [20110531]
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20110621]
  • [20110524] UVic and Toronto participated in SSC5 (23 May - 03 June 2011) - final reports to be sent to SSC5
  • [20110616] EGI CSIRT ADVISORY [EGI-ADV-20110615-02] Vulnerability for LRMS (details to be posted later)

• gLite News [20110705]
  • [20110705] [https://operations-portal.egi.eu/broadcast/archive/id/420][EGI Trust Anchor release 1.40-1] is now final - please install.
  • [20110504] [https://operations-portal.egi.eu/broadcast/archive/id/378 ][gLite 3.2 WN release] has an important known issue detailed here: http://glite.cern.ch/glite-WN/known_issues
  • [20110627] New IGTF distribution (certificates) and version 3 of the CRL retrieval tool Fetch-crl https://dist.eugridpma.info/distribution/util/fetch-crl3/. Version 1.39 was updated to 1.40 on 2011-06-28.
  • [20110628] [https://operations-portal.egi.eu/broadcast/archive/id/410][gLite 3.2 update 30 release] The highlights of the update are:
    • New version of glite-BDII_top
    • New version of glite-CREAM (to support changes for SGE)
    • New version of glite-LB
    • New version of glite-SGE_utils

• ROC_Canada News [20110809]
  • Current shift -Reda Tafirout 8-12 Aug
  • Next Week -Di Qing

• Squid Deployment [20110329]
  • CA cloud Frontier and Squid deployment information (versions and network information)
    • Frontier-Squid Monitoring http://frontier.cern.ch/squidstats/indexatlas.html
  • SAM tests: (CA Link), (Aus Link):

• General Computing News

• SONAR Tests [20110426]
  • Performance monitored at http://bourricot.cern.ch/dq2/ftsmon/sonar_view/cached/ with historical views at http://bourricot.cern.ch/dq2/ftsmon/ . Canadian cloud now tested on Fridays. All CA Tier-2's are tested with small (20MB) and medium (200MB) files, but only TRIUMF, CA-SCINET-T2 and CA-VICTORIA-WESTGRID-T2 are currently tested with large (2GB) files outside of the cloud.

SAM Results

• site service status via Gridview - Please check your site

• July 10-August 9, 2011

• August 2011 Detailed Site Information (valid until 20110831):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

Outstanding Issues

TRIUMF

Recent Issues

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2011-05-24%2020:59:59&start-date=2011-05-31%2020:00:00

  World-wide weekly ATLAS ADCoS summary (Aug.2-8,2011)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Aug.2-8,2011) Panda production service
   - completed 822,231 managed group, MC production, validation
     and reprocessing jobs
   - average ~117,461 jobs per day
   - failed 52,939 jobs
   - average efficiency:
      -- jobs  ~93.9
   - active tasks from Panda-monitor: 2407
      -- distribution by cloud:
CA:183 CERN:76 DE:251 ES:95 FR:430 IT:210 ND:136 NL:264 TW:93 UK:238 US:431

B) - CA/TRIUMF ORACLE DB intervention (transparent, <5min.), Aug.4. 
    - CERN/CVMFS namespace upgrade, ~50min., Aug.8.
   - CERN/ATLAS SSB upgrade rolled back at 7:30pm Aug.8.To be done Aug.9.
   - new DQ2 Clients Testing Release 0.1.37_rc4 announced, Aug.4.
   - new DB Release 16.4.1,16.5.1.1,16.6.1 subscribed to sites, Aug.3,5,8.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    ------------------------------------------------- 
6) Sun Aug.7 
- CA/CA-SCINET-T2 FT f./SOURCE:CONN.:SRM.GGUS 73238 assigned:SRM process
  restarted.Second failure in a week after upgrading to CentOS5.6.Stll 1K+
  f. after restart,Aug.8.Blacklisted in DDM:Savannah 122683.

• Stats:
  • See Key for what the headers mean