Tier1 and Tier-2 Weekly Report 2009-06-09

General News

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 2-14 Running)
  • ATLAS STEP09 Daily Logbook: https://twiki.cern.ch/twiki/bin/view/Atlas/Step09Logbook
  • ATLAS CA STEP09: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Step09

• REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • So far only ALBERTA and TORONTO with some reporting of SiteTests results (see for templates)
  • Please confirm results of daily siteTests package execution to operations mailing list and on TWiki results pages given above

• site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

• draft May 2009 accounting report released for comments
  • http://lcg.web.cern.ch/LCG/accounting/Tier-2/2009/may-09/Tier2_Accounting_Report_May2009.pdf

T2 CA East =  62% of MoU pledge
      TORONTO-LCG2 --->  55282 kSI2K-Hrs

T2 CA West = 166% of MoU pledge
      ALBERTA-LCG2 --->  85282 kSI2K-Hrs
          SFU-LCG2 --->  17499 kSI2K-Hrs
     VICTORIA-LCG2 ---> 119415 kSI2K-Hrs

• Advance Notice: ATLAS Cosmic Run June 22 - July 5
  • data export will be to T1 and T2s
  • clouds need to determine shares to be distributed to T2s (timescale of approximately 2 weeks, so just after STEP09 + a few days)
  • STEP09 cleaning will be done in advance
  • Expected data volumes across the entire cloud:
    • ESD (T1): ~ 80 TB
    • AOD: ~ 10 TB
    • DPD: ~ 10 TB

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-20090609.png:
  

• ops-reliability-20090609.png:
  

ATLAS VO

• atlas-availability-20090609.png:
  

• atlas-reliability-20090609.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090609.png:
  

APEL Accounting Results

• apel-statistics-20090609.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

• single occurrence errors received for SAM tests
  • SRMv2-put, CE-sft-lcg-rm [2009-06-03]
  • sBDII-sanity test [2009-06-08] (bdii-fwd.log overflowed /opt partition) with ggus ticket following

• SiteTests package test failure observed [2009-06-08] with 'request expired'; successful the next day

• 4 (smaller) of the 6 total dcache pools became full [2009-06-07] causing failed writes to local SE
  • full pool marked as read-only to successfully restore write access (see STEP09 page)

• user analysis (via pilots) job errors
  • 69 failures (191 successful) [2009-06-08]
    • example: http://condorg.triumf.ca/analysisPilots/2009-06-08/lcgce01.cpp.ualberta.ca_2119_jobmanager-lcgpbs-atlas-CERN/11628724.8.out
  • 6 failures (346 successful) [2009-06-09] ; write failure with no write pools - leftover from weekend job retries?
    • example: http://condorg.triumf.ca/analysisPilots/2009-06-08/lcgce01.cpp.ualberta.ca_2119_jobmanager-lcgpbs-atlas-CERN/11631649.19.out

Outstanding Issues

• installation of grid services on new Westgrid-UA T2 site still to done

• few minor issues pending correction:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• SFU production restored to online status [2009-06-03]

• user analysis (pilot) job failures [2009-06-04] (14 jobs)
  • possible problematic files (.lib.tgz and .root. identified)

• user analysis (pilot) job failures [2009-06-05] (9 failures)
  • problem staging input files ; successfully accessed offline

• user analysis (pilot) job failures [2009-06-09] (5 failures)
  • problem staging input files ; investigating

Outstanding Issues

Victoria

Recent Issues

• SE failures resulting in no space available on MCDISK [2009-06-04] - see STEP09 summary pages

• user analysis (pilot) job failures reported [2009-06-05] and to be investigated by site
  • put timeout from server certificate error ; missing or not installed (6 failures)
  • replica not found (14 failures)
  • input could not be staged (234 failures)

• user analysis (pilot) job failures reported [2009-06-09] and to be investigated by site
  • put timeout from server certificate error ; missing or not installed (3 failures)

Outstanding Issues

Toronto

Recent Issues

• SE high load and errors observed [2009-06-08]
  • 467 atlas job failures reported (ggus ticket and daily CA operations)
  • FTS tranfers halted ; restart of srm and door services done to correct problem

Outstanding Issues

• Debugging APEL issues - issued GGUS ticket #48841 https://gus.fzk.de/ws/ticket_info.php?ticket=48841 RESOLVED
• Issued ticket https://gus.fzk.de/ws/ticket_info.php?ticket=48882 to address why no production jobs have been assigned to Toronto RESOLVED

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	42652	1229	281103809	5737710	97.2%	98%
TORONTO-LCG2	2525	874	88149262	19182047	74.3%	82.1%
VICTORIA-LCG2	1387	381	51243254	3337336	78.5%	93.9%
ALBERTA-LCG2	1275	98	31458116	1227594	92.9%	96.2%
SFU-LCG2	1003	275	18979968	156557	78.5%	99.2%
total	15243	1337	478322231	19502513	91.9%	96.1%

* dashboard-20090609.pdf: dashboard-20090609.pdf

Failed Jobs

Information taken from the production dashboard (see above site links).

TRIUMF
name,rank,value
EXEPANDA_DQ2_SERVERERROR,1,375
TRF_EXC ,2,338
TRFERROR,3,265
TRF_OUTFILE ,4,148
EXEPANDA_JOBEXPIRED_SIXDAYS,5,40
EXEPANDA_JOBKILL_BYPILOT,6,25
TRF_UNKNOWN ,7,8
EXEPANDA_GET_ADLER32MISMATCH,8,7
TRF_SEGFAULT ,9,6
EXEPANDA_DQ2_STAGEIN,10,4
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,11,4
EXEPANDA_GET_NOSUCHFILE,12,3
EXEPANDA_PUT_GLOBUSSYSTEMERROR,13,2
EXEPANDA_JOBKILL_BYSERVER,14,2
TRF_SEGVIO ,15,1
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,16,1

TORONTO
name,rank,value
EXEPANDA_GET_GLOBUSSYSTEMERROR,1,524
EXEPANDA_PUT_GLOBUSSYSTEMERROR,2,295
EXEPANDA_DQ2GET_INFILE,3,36
EXEPANDA_JOBKILL_BYPILOT,4,6
TRF_SEGVIO ,5,5
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,6,5
TRF_UNKNOWN ,7,1
TRF_ATHENACRASH ,8,1
EXEPANDA_DQ2PUT_FILECOPYERROR,9,1

VICTORIA
name,rank,value
TRF_ARG ,1,199
EXEPANDA_DQ2GET_INFILE,2,78
EXEPANDA_DQ2PUT_FILECOPYERROR,3,44
EXEPANDA_GET_GLOBUSSYSTEMERROR,4,21
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,15
EXEPANDA_JOBEXPIRED_SIXDAYS,6,15
EXEPANDA_UNKNOWNERROR_JOBWRAPPERCRASH,7,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,8,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,9,2
EXEPANDA_JOBKILL_BYPILOT,10,1
TRF_UNKNOWN ,11,1
TRF_SEGVIO ,12,1

ALBERTA
name,rank,value
TRFERROR,1,64
EXEPANDA_JOBEXPIRED_SIXDAYS,2,27
EXEPANDA_DQ2_SERVERERROR,3,4
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,4,2
EXEPANDA_DQ2GET_INFILE,5,1

SFU
name,rank,value
EXEPANDA_GET_NOSUCHDBRELEASEFILE,1,264
EXEPANDA_JOBKILL_SIGTERM,2,6
EXEPANDA_DQ2_STAGEIN,3,2
TRF_OUTFILE ,4,1
EXEPANDA_DQ2GET_CANTCALL,5,1
TRF_SEGVIO ,6,1
 

-- BryanCaron - 09 Jun 2009