Tier1 and Tier-2 Weekly Report 2009-03-31

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

• availability-ops-20090331.png:
  

• reliability-ops-20090331.png:
  

• availability-atlas-20090331.png:
  

• reliability-atlas-20090331.png:
  

APEL Accounting Publishing Status

• apel-gantt-20090331.png:
  

APEL Accounting Results

• apel-statistics-20090331.png:
  

General News

• requested TiersofAtlas.py update to reflect new SCRATCHDISK space token in the sit dache at Victoria and SFU (to complement that already done at Alberta and TRIUMF) [20090331].
• critical VOMS server certificate update needs to be applied before April 1 at the latest (see EGEE broadcast message of 2009-03-30) on all site services
  • https://cic.gridops.org/index.php?section=roc&page=broadcastretrieval&step=2&typeb=C&idbroadcast=39901

• Westgrid TRIUMF and SFU network intervention for restructuring [2009-03-30]
  • status ?

• Hammercloud tests are planned for restart on 2009-04-07 with FileStager

Site Reports

Alberta

Recent Issues

• various SRM errors reported in SAM tests (started 2009-03-28) with GGUS ticket received 2009-03-30 (all file transfers failing)
  • entered unscheduled downtime into GOCDB for 2 days (started 2009-03-30) but ended after 1 day downtime
    • failure due to pnfsd.log growing too large (> 2 GBytes) which causes the 32-bit application to fail as well as not restart until the file is removed.
    • rerunning information provider script for storage capacity corrected reported site storage GlueSA info; SAM tests subsequently passed
    • will add pnfsd.log to log-rotate for future handling.

• new Westgrid - UA cluster (storage + cpu) arrived at the main campus computing centre and installation is underway [2009-03-30]

Outstanding Issues

• few WNs to be debugged with respect to missing stderr/stdout
• still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• SE problems observed in user and production jobs [2009-03-24] and identified by Asoka and ADC Ops shifters
  • high load on newly added 5 TB pool nodes; individually rebooted but errors persisted ; restarted entire dCache setup [2009-03-25] which corrected the problem
• created missing atlasscratchdisk area in pnfs for the new ATLASSCRATCHDISK space token [2009-03-26]
• 1 reported failure of 68 analysis pilot jobs with dccp error (executed on node s25) [2009-03-30]

Outstanding Issues

Victoria

Recent Issues

• PRODDISK area became full [2009-03-23]
  • increased PRODDISK from 1.5 to 2 TB [2009-03-23]

• long-running ganga jobs reported [2009-03-23]
  • some correlation believed with failing SAM CE SFT failures (see below)
  • panda and ganga jobs running successfully as of 2009-03-24

• GGUS ticket 47331 received about SAM CE SFT test failures [2009-03-24]
  • site outage for maintenance to correct temporarily removed it from information system

• single file stage failure for user analysis pilot (out of 100 user analysis jobs) [2009-03-31]

Outstanding Issues

Toronto

Recent Issues

• created SCRATCHDISK space token [2009-03-12]

Outstanding Issues

• scheduled maintenance downtime between 2009-03-25 and 2009-03-30 for ongoing dCache 1.9 and chimera upgrade
  • failing SFT for job submission since 2009-03-10 with general SE problems since early March (GGUS ticket 46825)
  • split head node with upgrade to chimera from PNFS ; scheduled downtimes followed
  • status and plans for restart?

McGill

Recent Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

Outstanding Issues

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	31373	838	296633752	36765853	97.4%	89%
SFU-LCG2	5600	777	63905239	13099345	87.8%	83%
VICTORIA-LCG2	3059	245	114412841	770668	92.6%	99.3%
ALBERTA-LCG2	812	201	17623817	3625806	80.2%	82.9%
total	40844	2061	492575649	54261672	95.2%	90.1%

DashboardSummary.pdf

Failed Jobs

Panda Monitor is still flakey but this is the report for the past 3 days .

FailedJobs-Panda.pdf

Dashboard shows the errors for the past 7 days as:

TRIUMF:
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,383
EXEPANDA_JOBKILL_BYSERVER,2,166
EXEPANDA_JOBKILL_BYPILOT,3,112
EXEPANDA_JOBKILL_BYEXTIF,4,78
EXEPANDA_DQ2_STAGEIN,5,41
TRF_SEGFAULT ,6,22
TRF_EXC ,7,11
EXEPANDA_JOBKILL_SIGTERM,8,7
TRF_UNKNOWN ,9,6
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,10,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,11,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,12,2
EXEPANDA_PUT_GLOBUSSYSTEMERROR,13,2
EXEPANDA_DQ2PUT_FILECOPYERROR,14,1

Alberta:
name,rank,value
EXEPANDA_DQ2GET_INFILE,1,92
TRFERROR,2,51
EXEPANDA_JOBEXPIRED_SIXDAYS,3,24
EXEPANDA_DQ2_STAGEIN,4,21
EXEPANDA_GET_REPLICANOTFOUND,5,14

SFU:
name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,544
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,197
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,33
EXEPANDA_DQ2_STAGEIN,4,22
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,5,4
EXEPANDA_DQ2GET_INFILE,6,3
TRF_UNKNOWN ,7,3
TRF_SEGFAULT ,8,3

VICTORIA:
name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,148
EXEPANDA_JOBEXPIRED_SIXDAYS,2,144
EXEPANDA_GET_REPLICANOTFOUND,3,13
EXEPANDA_DQ2_STAGEIN,4,1
EXEPANDA_PILOTEXCEPTION,5,1

Notes

-- BryanCaron - 31 Mar 2009