Tier1 and Tier-2 Weekly Report 2009-05-26

General News

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 2-14 Running) * NOTE: NEW START DATE OF JUNE 2 *
  • ATLAS STEP09: https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
  • ATLAS CA STEP09: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Step09
    • proposed CA T2 shares for merge.AOD and data distribution exercise (PDF)
  • for further information regarding the types of jobs expected during STEP09 please see https://twiki.cern.ch/twiki/bin/view/Atlas/StressTestJobs

• REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
  • Please confirm results of daily siteTests package execution to operations mailing list and on TWiki results pages

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-20090526.png:
  

• ops-reliability-20090526.png:
  

ATLAS VO

• atlas-availability-20090526.png:
  

• atlas-reliability-20090526.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090526.png:
  

APEL Accounting Results

• apel-statistics-20090526.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

• completed srmv1 cleanup of SE [2009-05-24]; total of 4.3 TB of space was cleared for reallocation to MCDISK
• temporarily added 4 additional pools from WestGrid-UofA T2 dCache systems to existing T2 pool nodes via NFS export [2009-05-25]
  • added to default Pool group so some pool-to-pool migration will be needed after STEP09 to free up for final T2 configuration
• updated space tokens for STEP09 to accommodate shares of merge.AOD and data distribution exercise
  • USERDISK_SIZE = 0.191 TB, MCDISK_SIZE = 17.008 TB, PRODDISK_SIZE = 1.819 TB, LOCALGROUPDISK_SIZE = 0.005 TB, SCRATCHDISK_SIZE = 0.227 TB, DATADISK_SIZE = 30.559 TB (TOTAL_SIZE = 49.809 TB)

Outstanding Issues

• few minor issues pending correction:
  • site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
    • Update [2009-05-12]: Filestager issues that are causing Gangarobot to fail at Alberta should be cleared up in Ganga 5.2.1 when released
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

Outstanding Issues

Victoria

Recent Issues

Outstanding Issues

Toronto

Recent Issues

• Debugging APEL issues - issued GGUS ticket #48841 https://gus.fzk.de/ws/ticket_info.php?ticket=48841
• Issued ticket https://gus.fzk.de/ws/ticket_info.php?ticket=48882 to address why no production jobs have been assigned to Toronto

Outstanding Issues

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	11325	463	303802278	1921529	96.1%	99.4%
VICTORIA-LCG2	2007	1740	111559362	4342478	53.6%	96.3%
SFU-LCG2	1070	1403	27516836	28980553	43.3%	48.7%
ALBERTA-LCG2	1535	187	55364925	0	89.1%	100%
total	15937	3793	498243401	35244560	80.8%	93.4%

Dashboard.pdf

Failed Jobs

FailedJobs.pdf

Again we have a discrepancy between the Panda monitor and the dashboard so these are the failure breakdowns from the dashboard:

TRIUMF:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,176
EXEPANDA_DQ2_STAGEIN,2,98
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,77
EXEPANDA_JOBKILL_SIGTERM,4,66
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,5,39
EXEPANDA_DQ2GET_INFILE,6,3
TRF_SEGVIO
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,8,1
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,9,1
TRF_OUTFILE

ALBERTA:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,187

SFU:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,982
EXEPANDA_DQ2PUT_FILECOPYERROR,2,389
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,3,26
EXEPANDA_GET_REPLICANOTFOUND,4,4
EXEPANDA_PUT_GLOBUSSYSTEMERROR,5,1
TRF_SEGVIO

VICTORIA:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,1264
EXEPANDA_JOBEXPIRED_SIXDAYS,2,291
EXEPANDA_DQ2_STAGEIN,3,144
EXEPANDA_DQ2PUT_FILECOPYERROR,4,23
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,8
TRF_UNKNOWN
TRF_OUTFILE
EXEPANDA_JOBDISPATCHER_HEARTBEAT,8,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,9,1
EXEPANDA_GET_REPLICANOTFOUND,10,1

The major causes of failures at SFU and Victoria are SE related.

Notes

-- BryanCaron - 26 May 2009