Tier1 and Tier-2 Weekly Report 2009-04-21

General News

• Draft March 2009 WLCG Accounting report is now available Tier-2_Accounting_Report_March2009.pdf
  • CA-TIER2-EAST: 1324 kSI2k-Hours (1 % of pledge)
  • CA-TIER2-WEST: 207120 kSI2k-Hours (155 % of pledge)

• CA cloud (production) moved to run from the Oracle Panda servers at CERN [2009-04-20]
  • production pilots switched to Oracle
  • analysis pilots still to be sent to Panda server at BNL (mySQL)

• REMINDER #1: siteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests

• REMINDER #2: remaining Sites (Toronto) please provide dCache checksum information (csm info) for you pools
  • From the dCache admin interface for each pool run the 'csm info' command and return the output to the list

• reminders ongoing with Panda team so as to move forward in implementing ATLAS Canada computing model within Panda framework

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

• ops-availability-20090421.png:
  

• ops-reliability-20090421.png:
  

• atlas-availability-20090421.png:
  

• atlas-reliability-20090421.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090421.png:
  

APEL Accounting Results

• apel-statistics-20090421.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:
  • 995 GB of HITS to TRIUMF_LOCALGROUPDISK

Site Reports

Alberta

Recent Issues

• stable site operations
  • single SAM SFT failure for CE reported (auto-email) [2009-04-19]

• new T2 on Westgrid - UA cluster (storage + cpu) being commissioned
  • water cooling loop issues resolved last week and is functioning properly under load [2009-04-16]
  • connectivity of ATLAS service nodes to new network is complete via a single-mode fiber connection between Westgrid and HEPnet switches [2009-04-16]
    • BGP configuration within router setup over lightpath to TRIUMF T1 being confirmed with T1 networking people [2009-04-21]
  • SGI image installation issues for head and atlas nodes unresolved; manually installed SL47 onto all ATLAS nodes [2009-04-17]
    • SAN storage volume mounting on dCache pool nodes - details to be resolved with vendor today
  • torque/Moab installation to be done this week

Outstanding Issues

• bug report filed by Asoka with Ganga in regards to failing robot filestager tests using Ganga 5.1.9 compared to those which were successful during Hammercloud (Ganga 5.1.8) [2009-04-14] - *is this resolved with 5.1.10 ? No. Need to wait for 5.2.0 release *
  • issue first observed [2009-04-07] (see below)
• site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
• full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
• pnfsd.log to be added to log-rotate [2009-03-31]
• few WNs to be debugged with respect to missing stderr/stdout
• still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• single job failure in writing user log to SE in the last 24 hour period (time out error) [2009-04-21]

Outstanding Issues

• current SE instabilities as observed during Hammercloud test 214 in Filestager [2009-04-08]
  • plan to move some dcache services to other server(s) for existing SE ? [2009-04-08]

Victoria

Recent Issues

• SE errors reported for 10% of analysis jobs (lcg_cp communication error on send) [2009-04-15]
• production and analysis job problems reported via team-ticket [2009-04-20]
  • possible problematic nodes under investigation as it does not look to be Oracle transition or pilot related

• APEL publishing status not up-to-date (see above plot) ; missing statistics for April 09 [2009-04-21]

Outstanding Issues

• bug report filed by Asoka with Ganga in regards to failing robot filestager tests using Ganga 5.1.9 compared to those which were successful during Hammercloud (Ganga 5.1.8) [2009-04-14] - *Is this resolved with 5.1.10 ? No. Need to wait for 5.2.0 release *

Toronto

Recent Issues

• site downtime extended from original (as described below)
• new dCache almost complete but still to be fully debugged; target return date is end of this week [2009-04-24]
  • srm space reservations failure being debugged
  • DDM operations will temporarily remove site from DDM Functional Tests if site doesn't come back to service before May 1

Outstanding Issues

• scheduled maintenance downtime between 2009-03-25 and 2009-03-30 for ongoing dCache 1.9 and chimera upgrade
  • failing SFT for job submission since 2009-03-10 with general SE problems since early March (GGUS ticket 46825)
  • split head node with upgrade to chimera from PNFS ; scheduled downtimes followed

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• network intervention interrupting network traffic into and out of TRIUMF - lightpaths to T1 and T2s unaffected [2009-04-15]

Outstanding Issues

• partial cooling failure [2009-04-11] with lower cpu (wn) capacity of about 30% until middle of following week - status?

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	10530	6286	111899234	124143616	62.6%	47.4%
VICTORIA-LCG2	2605	6611	62146494	39200119	28.3%	61.3%
SFU-LCG2	1214	960	20143720	19350572	55.8%	51%
ALBERTA-LCG2	1134	482	28425336	23069740	70.2%	55.2%
total	15483	14339	222614784	205764047	51.9%	52%

Dashboard.pdf

Failed Jobs

(From Panda Monitor - the numbers do not match the ATLAS Dashboard).

FailedJobs-21Apr09.pdf

(From ATLAS Dashboard)

TRIUMF
name,rank,value
TRF_UNKNOWN ,1,6161
EXEPANDA_JOBEXPIRED_SIXDAYS,2,179
EXEPANDA_JOBKILL_BYPILOT,3,15
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,9
EXEPANDA_DQ2_STAGEIN,5,5
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,6,4
EXEPANDA_JOBKILL_SIGTERM,7,2
EXEPANDA_GET_NOSUCHFILE,8,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,9,2
TRF_SEGFAULT ,10,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,11,1
EXEPANDA_PILOTEXCEPTION,12,1

ALBERTA
name,rank,value
TRF_UNKNOWN ,1,474
EXEPANDA_JOBEXPIRED_SIXDAYS,2,6
EXEPANDA_JOBKILL_BYPILOT,3,3
EXEPANDA_DQ2_SERVERERROR,4,2
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,5,1
EXEPANDA_GET_FAILEDTOGETLFCREPLICA,6,1

SFU
name,rank,value
TRF_UNKNOWN ,1,924
EXEPANDA_DQ2PUT_FILECOPYERROR,2,12
EXEPANDA_JOBKILL_SIGTERM,3,12
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,9
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,8
EXEPANDA_JOBKILL_BYPILOT,6,4
TRF_OUTFILE ,7,2
EXEPANDA_DQ2_STAGEIN,8,1

VICTORIA
name,rank,value
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,1,4752
TRF_UNKNOWN ,2,1802
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,36
TRF_SVRINIT ,4,13
EXEPANDA_DQ2GET_INFILE,5,12
EXEPANDA_DQ2_STAGEIN,6,8
EXEPANDA_JOBKILL_BYPILOT,7,8
EXEPANDA_GET_FAILEDTOGETLFCREPLICA,8,2
TRF_OUTFILE ,9,1
EXEPANDA_JOBEXPIRED_SIXDAYS,10,1
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,11,1
EXEPANDA_RUNJOBEXCEPTION,12,1

Notes

All of the CA cloud had had a high failure rate for task 59222 valid2.108867.CosSimIDVolSolOnTorOn.simul.s533.task; Rod is aware of it and contacted the appropriate people about killing the task.

UVic, in addition, has a possible site-related problem which showed up since 18 Apr where the pilots quit prematurely before fetching a payload.

-- BryanCaron - 21 Apr 2009