Tier1 and Tier-2 Weekly Report 2009-05-19

General News

• Detection/deletion of files registered in the LFC in the non space token area
  • schedule to be set for this cleanup to be applied to the CA SEs
    • https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsScripts#Detection_deletion_of_files_regi
    • List of files (from April 23)

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 1-14 Running)
  • https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
  • T2s please keep these periods in mind when planning upgrades, scheduled downtimes
  • CA preparations and actions for STEP09: caron-STEP09_20090519.pdf: caron-STEP09_20090519.pdf

• REMINDER #2: siteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests
  • Please confirm results of daily siteTests package execution to operations mailing list (and new TWiki pages to soon follow)
    • This is one component of the CA ROC functionality that is foreseen to come online later this year as a result of the transition from EGEE to EGI and the end of the CERN ROC operations.

• REMINDER #3: remaining Sites (Toronto) please provide dCache checksum information (csm info) for you pools
  • From the dCache admin interface for each pool run the 'csm info' command and return the output to the list
  • [2009-05-19 L. Groer] This information was sent to the list

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

Outstanding Issues

• few minor issues pending correction:
  • site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
    • Update [2009-05-12]: Filestager issues that are causing Gangarobot to fail at Alberta should be cleared up in Ganga 5.2.1 when released
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

Outstanding Issues

• several SE related errors over the weekend [2009-05-04]
  • currently moving all SE files from old storage (gridstore) to new one across 1 GigE? link of which is shared with general job access to the write pools
  • some error levels expected but should still be reported

Victoria

Recent Issues

Outstanding Issues

Toronto

Recent Issues

• Debugging APEL issues - issued GGUS ticket #48841 https://gus.fzk.de/ws/ticket_info.php?ticket=48841
• Issued ticket https://gus.fzk.de/ws/ticket_info.php?ticket=48882 to address why no production jobs have been assigned to Toronto

Outstanding Issues

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	22234	1003	231940112	2616713	95.7%	98.9%
VICTORIA-LCG2	1698	624	71171958	14661198	73.1%	82.9%
SFU-LCG2	1061	783	14388430	18943717	57.5%	43.2%
ALBERTA-LCG2	1658	96	52331715	2293158	94.5%	95.8%
total	26651	2506	369832215	38514786	91.4%	90.6%

DashboardSummary.pdf

Failed Jobs

FailedJobs.pdf

Notes

Panda monitor and Dashboard have discrepancies again; here are the errors from Dashboard for each site:

TRIUMF

name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,697
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,2,189
TRF_SVRINIT ,3,42
TRF_UNKNOWN ,4,21
TRF_OUTFILE ,5,14
EXEPANDA_JOBEXPIRED_SIXDAYS,6,11
TRF_SEGFAULT ,7,10
EXEPANDA_GET_NOSUCHFILE,8,8
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,9,5
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,10,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,11,1
EXEPANDA_JOBKILL_BYPILOT,12,1


ALBERTA

name,rank,value
TRFERROR,1,51
EXEPANDA_JOBEXPIRED_SIXDAYS,2,42
TRF_SEGFAULT ,3,1
TRF_UNKNOWN ,4,1
EXEPANDA_GET_NOSUCHFILE,5,1


SFU

name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,615
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,152
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,3,7
TRF_SEGFAULT ,4,4
EXEPANDA_GET_REPLICANOTFOUND,5,3
EXEPANDA_JOBKILL_BYPILOT,6,1
TRF_SVRINIT ,7,1


VICTORIA
name,rank,value
EXEPANDA_DQ2GET_INFILE,1,292
EXEPANDA_DQ2PUT_FILECOPYERROR,2,237
EXEPANDA_DQ2_STAGEIN,3,33
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,15
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,5,14
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,9
TRF_UNKNOWN ,7,5
EXEPANDA_JOBKILL_SIGTERM,8,4
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,9,4
EXEPANDA_JOBEXPIRED_SIXDAYS,10,3
TRF_SEGFAULT ,11,2
EXEPANDA_DQ2_SERVERERROR,12,2
TRF_SVRINIT ,13,2
EXEPANDA_JOBKILL_BYPILOT,14,2

SFU and Victoria had SE issues that caused the low efficiencies.

-- BryanCaron - 19 May 2009