Tier1 and Tier-2 Weekly Report 2009-05-12

General News

• Hammercloud test 277 completed last week on CA sites (without Filestager)
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/DAChallenge#May_2009

• Detection/deletion of files registered in the LFC in the non space token area
  • schedule to be set for this cleanup to be applied to the CA SEs
    • https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsScripts#Detection_deletion_of_files_regi
    • List of files (from April 23)

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 1-14 Running)
  • https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
  • T2s please keep these periods in mind when planning upgrades, scheduled downtimes

• REMINDER #2: siteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests
  • Please confirm results of daily siteTests package execution to operations mailing list (and new TWiki pages to soon follow)
    • This is one component of the CA ROC functionality that is foreseen to come online later this year as a result of the transition from EGEE to EGI and the end of the CERN ROC operations.

• REMINDER #3: remaining Sites (Toronto) please provide dCache checksum information (csm info) for you pools
  • From the dCache admin interface for each pool run the 'csm info' command and return the output to the list

• May GDB Agenda (May 13, 2009) http://indico.cern.ch/conferenceDisplay.py?confId=45475

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• t1-ops-availability-20090512.jpg:
  

• t1-ops-reliability-20090512.jpg:
  

• t2-ops-availability-20090512.jpg:
  

• t2-ops-reliability-20090512.jpg:
  

ATLAS VO

• t1-atlas-availability-20090512.jpg:
  

• t1-atlas-reliability-20090512.jpg:
  

• t2-atlas-availability-20090512.jpg:
  

• t2-atlas-reliability-20090512.jpg:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090512.jpg:
  

APEL Accounting Results

• apel-statistics-20090512.jpg:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: 1.84 TB (HITS) to TRIUMF_LOCALGROUPDISK (pending approval)

Site Reports

Alberta

Recent Issues

• ggus ticket to indicate the PRODDISK area became full (1.5 TB) [2009-05-10] and increased in size to 2.0 TB [2009-05-11] but continues to grow in occupancy

• new T2 on Westgrid - UA cluster (storage + cpu) being commissioned
  • confirmation of BGP configuration from TRIUMF T1 networking people received [2009-05-07] ; failover tests to be scheduled
  • SAN storage volumes mounted on dCache pools using multipath IO [2009-04-11] ; services installation to begin this week

Outstanding Issues

• few minor issues pending correction:
  • site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
    • Update [2009-05-12]: Filestager issues that are causing Gangarobot to fail at Alberta should be cleared up in Ganga 5.2.1 when released
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• SE failures reported (55 for SE out of 433 failed jobs ; 384 successful) [2009-05-08]
  • heavy load on dCache with many transfers and write requests to new pools, not unexpected with the migration from gridstore

Outstanding Issues

• several SE related errors over the weekend [2009-05-04]
  • currently moving all SE files from old storage (gridstore) to new one across 1 GigE? link of which is shared with general job access to the write pools
  • some error levels expected but should still be reported

Victoria

Recent Issues

• long running (greater than 12 hours) atcan ganga jobs observed [2009-05-08]

• SE failures with hanging srmcp for pool read and write [2009-05-10]
  • pnfs using gdbm reached 2 GB limit ; migration to postgres for pnfs needed ; completed (?) as SAM tests are passing
  • entered scheduled downtime [2009-05-10 to 2009-05-11]

Outstanding Issues

Toronto

Recent Issues

• site status ok but many (130) hanging ganga jobs noticed (few minutes of CPU then killed by 80h timeout from batch system) [2009-05-11]
  • possible leftovers from Hammercloud 277 - safe to be killed

Outstanding Issues

• out of space for production releases reported [2009-04-29]
  • deleted sl3 production releases (no longer needed) to clear up space [2009-04-30]
  • migration to larger software installation area?

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• scheduled FTS outage for oracle patches [2009-05-07]

• switch to larger NFS area for atlas software exports [2009-05-06]

• failed export to DATADISK from CERN and T1s [2009-05-08]
  • transient error only and can be ignored (srm connection spike?)

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	44763	553	228945381	3656644	98.8%	98.4%
SFU-LCG2	2525	846	15116899	19326216	74.9%	43.9%
VICTORIA-LCG2	1452	434	34296373	9987991	77%	77.4%
ALBERTA-LCG2	1191	327	29024328	6040647	78.5%	82.8%
total	49931	2160	307382981	39011498	95.9%	88.7%

DashboardSummary.pdf

Failed Jobs

FailedJobs.pdf

Notes

• Victoria, SFU: mainly SE failures.
• Alberta: looks like DDM errors (registration, DQ2) among others.

-- BryanCaron - 12 May 2009