Tier1 and Tier-2 Weekly Report 2009-06-02

General News

• REMINDER #1: STEP09 exercise (May 25-31 Setup, June 2-14 Running) * NOTE: NEW START DATE OF JUNE 2 *
  • ATLAS STEP09: https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
  • ATLAS CA STEP09: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Step09
    • submitted CA T2 shares for merge.AOD and data distribution exercise (PDF)
  • for further information regarding the types of jobs expected during STEP09 please see https://twiki.cern.ch/twiki/bin/view/Atlas/StressTestJobs
  • T1 and T2 information for feedback see https://twiki.cern.ch/twiki/bin/view/Atlas/Step09Feedback

• REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
  • Please confirm results of daily siteTests package execution to operations mailing list and on TWiki results pages given above

• Advance Notice: ATLAS Cosmic Run June 22 - July 5
  • data export will be to T1 and T2s
  • clouds need to determine shares to be distributed to T2s (timescale of approximately 2 weeks, so just after STEP09 + a few days)
  • STEP09 cleaning will be done in advance
  • Expected data volumes across the entire cloud:
    • ESD (T1): ~ 80 TB
    • AOD: ~ 10 TB
    • DPD: ~ 10 TB

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-t1-availability-20090602.png:
  

• ops-t1-reliability-20090602.png:
  

• ops-availability-20090602.png:
  

• ops-reliability-20090602.png:
  

ATLAS VO

• atlas-t1-availability-20090602.png:
  

• atlas-t1-reliability-20090602.png:
  

• atlas-availability-20090602.png:
  

• atlas-reliability-20090602.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090602.png:
  

APEL Accounting Results

• apel-statistics-20090602.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

• four single occurrence errors received for SAM tests
  • SRMv2-ls-dir and SRMv2-put [2009-05-27]
  • CE-sft-lcg-rm [2009-05-29]
  • sBDII-sanity [2009-05-30]

• otherwise site services were stable
  • additional borrowed pools functioning fine during merge.AOD replications

• preliminary STEP09 fairshare analysis
  • during ramp up of STEP09 preparations of the weekend noticed that the assigned fairshares between production, analysis (pilot) and analysis (wms) were not being achieved (no running production jobs)
    • modified maui.cfg to create soft/hard limits on number of jobs in addition to fairshare targets; enabled production jobs to run but hard limits were not being obeyed
    • in general balance between production (60 jobs), analysis (WMS) (22 jobs) and analysis (pilots) (30 jobs) behaving as desired but will continue to monitor

Outstanding Issues

• few minor issues pending correction:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• SE related failures [20090527] in 12 of 1866 jobs reported (incl. communication error on send)

• SE related failures [20090529] in 42 of 1065 jobs reported (majority dccp timeouts)

• SFU offline in production since approx. May 23 due to https://gus.fzk.de/ws/ticket_info.php?ticket=48973&from=search ; ticket closed though
  • needs to be followed-up with ADC shifters

• "low" transfer rate between TRIUMF and SFU reported by ADC ops [20090531] ; TRIUMF to increase the FTS channel (?)

• 24 of 2786 analysis job report errors [20090602]
  • during dccp "exception caught: Get function can not be called for staging input files: " - under investigation

Outstanding Issues

Victoria

Recent Issues

• SE failures reported [20090527] in 17 of 341 analysis jobs prior 24 hours
  • lcg_cp: communication error on send
  • lcg_cr: permission denied
  • get_data failed
  • failed to get lfc replicas (timeout)

• SE machine certificate revoked after request to grid canada for new certificate [2009-05-27]
  • ongoing issue with regards to coordination with Grid Canada when transitioning service node to new host certificate

• SE failures reported [2009-05-29] 253 of 483 analysis jobs prior 24 hours
  • majority were 'abnormal termination: ecode=28169 ... etext not set) ...'

• disk pool failure [2009-05-31] causing no free space to be available in current token areas
  • jobs failing including SAM
  • unscheduled downtime entered to repair system
  • panda pilots temporarily halted from being sent to UVic during outage ; however this is not necessary
    • policy for future such outages at sites will be to continue submitting pilots since brokerage will not assign payloads during outages in gocdb

• SE failures reported [20090602] on both dashboard and SAM srm tests
  • 'server responded with an error 451 operation failed: non-null return code from ... Copy Failed'

Outstanding Issues

Toronto

Recent Issues

• false alarm regarding deletion made of database files as they were not used for production [20090527]
• correction within panda settings for site location for release check so as to restore production jobs to site [20090526]
• SE failures reported [20090527 and 20090528] in 18 of 220 analysis jobs
  • transport endpoint not connected
  • lcg-cp failed with abnormal termination (etext not set) and 'get_data failed'
    • access to DBRelease files ; extra replicas created in site dCache

• storage pool node errors observed during attempted pool-to-pool transfer with poolDomain restart appearing to have helped [20090528]

• pool server node became unresponsive and could not be restarted out-of-band [20090531]
  • restarted manually following day

• SE errors observed [20090602] in SAM tests and dashboard
  • globus_xio: system error in connect: connection refused
  • also observed FTS errors

• preliminary fairshare analysis [as of 20090529]
  • targets: atlasprd 45%, atcan 22%, atlas 22%
  • current [20090529]: atlasprd 49%, atcan 3%, atlas 46%

Outstanding Issues

• Debugging APEL issues - issued GGUS ticket #48841 https://gus.fzk.de/ws/ticket_info.php?ticket=48841
• Issued ticket https://gus.fzk.de/ws/ticket_info.php?ticket=48882 to address why no production jobs have been assigned to Toronto

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

We've been seeing a larger SAM test failures than usual. All the failures were due to timeouts on the SRM where client failed to get a TURL (lcg-gt in particular but also timeouts observed in FTS logs at getrequest stage). The overall system was not busy in terms of system's load. After some debugging, we've changed various SRM queue parameters to make them larger and restarted the SRM/Tomcat service.

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	11759	400	271589091	4550351	96.7%	98.4%
TORONTO-LCG2	1627	415	118091742	2091783	79.7%	98.3%
VICTORIA-LCG2	979	519	57785392	12859259	65.4%	81.8%
ALBERTA-LCG2	878	3	30856006	1120	99.7%	100%
total	15243	1337	478322231	19502513	91.9%	96.1%

• dashboard-20090602.pdf
• no production jobs to SFU this past week ; GGUS ticket 48973 ; will be followed up with production team

Failed Jobs

Information taken from the production dashboard.

TRIUMF:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,149
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,2,146
EXEPANDA_DQ2_STAGEIN,3,72
EXEPANDA_JOBKILL_SIGTERM,4,19
EXEPANDA_GET_NOSUCHFILE,5,7
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,6,4
TRF_SEGVIO ,7,1
EXEPANDA_DQ2GET_INFILE,8,1
EXEPANDA_JOBKILL_BYPILOT,9,1

ALBERTA:

name,rank,value
TRF_SEGVIO ,1,1
EXEPANDA_GET_REPLICANOTFOUND,2,1
EXEPANDA_GET_NOSUCHFILE,3,1

SFU:

VICTORIA:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,129
EXEPANDA_DQ2PUT_FILECOPYERROR,2,96
EXEPANDA_JOBKILL_SIGTERM,3,96
TRF_ARG ,4,90
EXEPANDA_DQ2_STAGEIN,5,65
EXEPANDA_GET_GLOBUSSYSTEMERROR,6,21
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,7,7
EXEPANDA_GET_REPLICANOTFOUND,8,5
TRF_SEGVIO ,9,3
EXEPANDA_PUT_LFCMODULEIMPORT,10,3
EXEPANDA_DQ2GET_INFILE,11,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,12,2

TORONTO:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,382
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,14
TRF_SEGVIO ,3,8
EXEPANDA_DQ2_STAGEIN,4,5
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,3
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,6,2
TRF_SEGFAULT ,7,1

-- BryanCaron - 02 Jun 2009