Tier-2 Weekly Report 2009-01-13

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

• ops-availability-20090120.png:
  

• ops-reliability-20090120.png:
  

• atlas-availability-20090120.png:
  

• atlas-reliability-20090120.png:
  

APEL Accounting Publishing Status

• apel-gant-20090120.png:
  

APEL Accounting Results

• apel-table-20090120.png:
  

General News

• Site Test Package (A. de Silva)
  • site admins are requested to install the following package to aid in the debugging of job submission
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests
  • Which sites have installed the Test Package?
    • Alberta - not yet
    • SFU -
    • Victoria -
    • Toronto -
  • Please do so as this is useful for debugging purposes.

• Tier1/2 face-to-face meeting at TRIUMF scheduled for January 26 (full day) and 27 (half day) agenda link

Site Reports

Alberta

Outstanding and Recent Issues

• scheduled downtime for Thursday January 15 for 6 hours starting at 08:00 MST due to interruption of chilled water service to HVAC unit in our computing centre
• long-latency observed on smokeping traffic from TRIUMF to Alberta router interface on e2e lightpath; rebooted switch (not affecting RTT to LCG services) [January 15]
• FTS channel between TRIUMF-Alberta problem reported last week via GGUS ticket - non-functioning for a few weeks now resolved; site in production test before full resumption of production jobs [January 20]
• similar to Victoria, the disk space issues as per settings in new panda update; resolved by update to Panda dBase [January 14]

• installing secondary site infrastructure using Xen for gLite updates deployment testing prior to moving any service changes into production
  • Xen VMs running (4); new host certificates created; CE glite middleware installed so far only
• stderr/stdout direction with a handful of remaining WNs
• various gLite updates to CE, SE and MON pending (see above)

SFU

Outstanding and Recent Issues

• atcan (analysis) user jobs being queued non-starting ; resolved [January 14]
• stdout/stderr still on a few nodes even with proper ssh keys installed
• occasional SAM test failures due to queues being full since CE node removed from accepting SAM jobs

Victoria

Outstanding and Recent Issues

• SE gridmapfile update issues - missing DN for JSteele - manual update still required [January 6,7]
  • automatic cross-check with CE grid-mapfile would be good to have in place
• caching of globus-job-run output seen (from December 15) - problem persisting [January 9]
  • problem related to different source job submitter systems (one real the other a VM); resolved?
• Victoria disk space issues as per settings in new panda update[January 14]
• production exceeding queue limit of 2 days at site ; resolved [January 16]
• added a new pool node with 2TB pool ; operations seem ok [January 20]

Toronto

Outstanding and Recent Issues

• SE gridmapfile update issues - missing DN for JSteele - manual update still required [January 6,7]
  • automatic cross-check with CE grid-mapfile would be good to have in place * Toronto queues issue [January 20]
• production exceeding queue limit of 2 days at site; still unresolved; Leslie made changes Jan20 [January 16]

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	19026	1018	359989310	28249685	94.9%	92.7%
SFU-LCG2	4052	360	152737885	2664704	91.8%	98.3%
TORONTO-LCG2	884	1280	21166121	96370080	40.9%	18%
VICTORIA-LCG2	581	357	15057934	41890506	61.9%	26.4%
ALBERTA-LCG2	480	115	7667264	2672840	80.7%	74.2%
total	25023	3130	556618514	171847815	88.9%	76.4%

dashboard.pdf

Most of the failed jobs at Toronto and Victoria were because jobs were killed by the system when a wallclock time of 2 days was reached. Most of the failed Alberta jobs had a transfer timeout (2 weeks) error.

Failed Jobs

FailedJobs-20jan09.pdf

Notes

-- AsokaDeSilva - 20 Jan 2009