Tier1 and Tier-2 Weekly Report 2009-06-23

General News

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 2-14 Running) Now Ended
  • ATLAS STEP09 Daily Logbook: https://twiki.cern.ch/twiki/bin/view/Atlas/Step09Logbook
  • ATLAS CA STEP09: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Step09

• • see CA STEP09 for daily and overall summary

• REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • So far only ALBERTA and TORONTO with some reporting of SiteTests results (see for templates)
  • Please confirm results of daily siteTests package execution to operations mailing list and on TWiki results pages given above

• site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

• Advance Notice: ATLAS Cosmic Run June 22 - July 5
  • data export will be to T1 and T2s
  • CA cloud shares to be submitted Wednesday (see email for proposed shares)
  • STEP09 cleaning is in progress
  • Expected data volumes across the entire cloud:
    • ESD (T1): ~ 80 TB
    • AOD: ~ 8 TB
    • DPD: ~ 8 TB

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

Outstanding Issues

• installation of grid services on new Westgrid-UA T2 site still to done

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

Outstanding Issues

Victoria

Recent Issues

Outstanding Issues

Toronto

Recent Issues

Outstanding Issues

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	33613	145	317123694	513914	99.6%	99.8%
TORONTO-LCG2	6496	233	209132604	2167385	96.5%	99%
SFU-LCG2	6368	101	68876066	369044	98.4%	99.5%
VICTORIA-LCG2	1780	30	45864875	205394	98.3%	99.6%
ALBERTA-LCG2	239	10	4049256	20986	96%	99.5%
total	48496	519	645046495	3276723	98.9%	99.5%

Dashboard_Summary.pdf

Failed Jobs

Panda monitor not providing information today so Information taken from the production dashboard (see above site links).

TRIUMF:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,46
EXEPANDA_DQ2_STAGEIN,2,40
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,3,25
TRF_UNKNOWN ,4,16
EXEPANDA_GET_NOSUCHDBRELEASEFILE,5,11
TRF_SEGFAULT ,6,4
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,7,2
EXEPANDA_PUT_GLOBUSSYSTEMERROR,8,1

Alberta:

name,rank,value
TRF_EXC ,1,6
EXEPANDA_DQ2PUT_LOCAL-OUTPUT-FILE-MISSING,2,4

SFU:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,22
TRF_OUTFILE ,2,19
EXEPANDA_GET_NOSUCHFILE,3,18
EXEPANDA_JOBKILL_SIGTERM,4,16
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,5,14
EXEPANDA_DQ2_STAGEIN,6,6
EXEPANDA_DQ2GET_INFILE,7,4
EXEPANDA_DQ2GET_CANTCALL,8,1
EXEPANDA_DQ2PUT_FILECOPYERROR,9,1

Toronto:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,112
EXEPANDA_DQ2PUT_FILECOPYERROR,2,39
EXEPANDA_DQ2_STAGEIN,3,22
TRF_EXC ,4,21
TRF_SEGFAULT ,5,9
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,6,9
TRF_UNKNOWN ,7,7
EXEPANDA_JOBDISPATCHER_HEARTBEAT,8,6
EXEPANDA_GET_NOSUCHFILE,9,4
EXEPANDA_GET_REPLICANOTFOUND,10,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,11,1
EXEPANDA_JOBKILL_BYPILOT,12,1

Victoria:

name,rank,value
TRF_EXC ,1,23
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,3
EXEPANDA_DQ2_STAGEIN,3,2
EXEPANDA_JOBKILL_BYPILOT,4,1
EXEPANDA_DQ2GET_INFILE,5,1

-- BryanCaron - 23 Jun 2009