Tier1 and Tier-2 Weekly Report 2009-05-05

General News

• draft March availability and reliability report issued
  • https://twiki.cern.ch/twiki/bin/viewfile/LCG/SamMbReports?filename=Tier2_Reliab_200903.pdf
  • CA sites generally showed decreased availability and reliability

                                                     Reliability History                  
                Reliability  Availability    Dec08     Jan09     Feb09   
Toronto        34 %          15 %          41 %      97 %      67 %     
Alberta        83 %          82 %          98 %      94 %      98 %     
SFU            77 %         77 %           92 %      89 %      85 %     
Victoria       91 %         89 %           95 %      93 %      96 %  

• Hammercloud test 277 started today [2009-05-05] on CA sites (without Filestager)
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/DAChallenge#May_2009

• Advanced Notice: STEP09 exercise (May 25-31 Setup, June 1-14 Running)
  • https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
  • T2s please keep these periods in mind when planning upgrades, scheduled downtimes

• REMINDER #1: siteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests

• REMINDER #2: remaining Sites (Toronto) please provide dCache checksum information (csm info) for you pools
  • From the dCache admin interface for each pool run the 'csm info' command and return the output to the list

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

* ops-availability-20090505.png:
* ops-reliability-20090505.png:

* atlas-availability-20090505.png:
* atlas-reliability-20090505.png:

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: None.

Site Reports

Alberta

Recent Issues

• too many globus-job-mnnagers running causing problem with non-executing jobs backlogging queues starting around 2009-04-24 identified via GGUS ticket
  • CE services restarted 2009-04-28 and correct operations resumed

• new T2 on Westgrid - UA cluster (storage + cpu) being commissioned
  • connectivity of ATLAS service nodes to new network is complete as of [2009-04-16]
    • awaiting confirmation of BGP configuration from TRIUMF T1 networking people before attempting any failover tests [20009-04-21]
  • SAN storage volume mounting on dCache pool nodes resolved with vendor - configuration of multipath IO to be done with TRIUMF Tier-1 as reference
  • torque/Moab installation is in place for the cluster
  • general ldap issues being resolved

Outstanding Issues

• site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
• full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
• pnfsd.log to be added to log-rotate [2009-03-31]
• few WNs to be debugged with respect to missing stderr/stdout
• still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• odd fairshare behavior observed comparing priority of atcan jobs versus atprd jobs when an overall share is provided to the single atlas group (atcan + atprd)
  • to be discussed and investigated further

• several SE related errors over the weekend [2009-05-04]
  • currently moving all SE files from old storage (gridstore) to new one across 1 GigE link of which is shared with general job access to the write pools
  • some error levels expected but should still be reported

Outstanding Issues

Victoria

Recent Issues

• SE errors reported through failing test jobs and also affecting production jobs [2009-04-28]
  • stale movers causing others to wait until timeout (for last month?); removed and problem resolved [2009-04-29]
  • timeouts can be configured for movers

• APEL publishing status corrected and up-to-date (see above plot) ; missing statistics reported last meeting for April 09 [2009-04-21]

Outstanding Issues

Toronto

Recent Issues

• out of space for production releases reported [2009-04-29]
  • deleted sl3 production releases (no longer needed) to clear up space [2009-04-30]

• downtime for dcache upgrade extended to [2009-04-28]
  • site back in operation 2009-04-30 ; reentered into DDM for the functional tests over the weekend
  • site set online by ADC oS shifters [2009-04-30] * some information services issues still to be resolved

Outstanding Issues

• new dCache almost complete but still to be fully debugged; target return date is end of this week [2009-04-24]
  • srm space reservations failure being debugged

• scheduled maintenance downtime between 2009-03-25 and 2009-03-30 for ongoing dCache 1.9 and chimera upgrade
  • failing SFT for job submission since 2009-03-10 with general SE problems since early March (GGUS ticket 46825)
  • split head node with upgrade to chimera from PNFS ; scheduled downtimes followed

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• Blades shutdown over the weekend because of cooling concerns.
• NFS software mount switched on Monday (4 May) and some jobs may have failed for this (may be flagged with errors such as release not installed / found.)

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	17229	2655	132208889	16677237	86.6%	88.8%
SFU-LCG2	2201	319	63682002	3169017	87.3%	95.3%
ALBERTA-LCG2	1821	282	54410961	407236	86.6%	99.3%
VICTORIA-LCG2	949	737	44715423	1723093	56.3%	96.3%
total	22200	3993	295017275	21976583	84.8%	93.1%

DashboardSummary.pdf

Failed Jobs

FailedJobs-05May09.pdf

Notes

• Not trivial to get information from Panda today - monitor not showing the failed job details.
• TRIUMF: Blades shutdown over the weekend because of cooling concerns. Also nfs software mount switched on Monday (4 May) and some jobs may have failed for this (may be flagged with errors such as release not installed / found.)
• A major part of the failures across all sites is the "Service Manager unable to Initialize service error" which I suspect is this GGUS ticket. (48362)
• SFU has some SE errors (86 of the 449 failed jobs)
• TRIUMF had some SE failures (1148 of 2656 failed jobs) - last of which happened at 04-30 18:42 UTC.
• Victoria had SE errors as well (297 of 696 failed jobs) - last of which happened at 04-29 16:22 UTC.

-- BryanCaron - 05 May 2009