Tier1 and Tier-2 Weekly Report 2009-04-14

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

• ops-availability-20090414.png:
  

• ops-reliability-20090414.png:
  

• atlas-availability-20090414.png:
  

• atlas-reliability-20090414.png:
  

APEL Accounting Publishing Status

• apel-gantt-20090414.png:
  

APEL Accounting Results

• apel-statistics-20090414.png:
  

General News

• Hammercloud tests (Test 214 and 221) completed at Victoria, SFU, Alberta and TRIUMF with and without FileStager [2009-04-07 and 2009-04-09]
  • Results at : https://twiki.atlas-canada.ca/bin/view/AtlasCanada/DAChallenge#April_2009

• REMINDER #1: siteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTests

• REMINDER #2: Sites please provide dCache checksum information (csm info) for you pools

• reminders ongoing with Panda team so as to move forward in implementing ATLAS Canada computing model within Panda framework

• TRIUMF network intervention scheduled for 1 hour on 2009-04-15 to replace 2 router cards
  • interruption of traffic into and out of TRIUMF (including Tier-1) except for lightpath traffic to/from CERN and T2s.

• notice from ATLAS DDM of central cleaning of incomplete MC AOD.v12 and AOD.v13 datasets [2009-04-12]
  • policy target for the near future has:
    • for AOD.v12, only 2 complete replicas will be kept worldwide (from 2 different clouds)
    • for AOD.v13, only 4 complete replicas will be kept
  • need to discuss distribution policy within CA cloud for any such complete replicas (ie. replica at T1 can be deleted if a replica exists at any T2 in the same cloud)

Site Reports

Alberta

Recent Issues

• bug report filed by Asoka with Ganga in regards to failing robot filestager tests using Ganga 5.1.9 compared to those which were successful during Hammercloud (Ganga 5.1.8) [2009-04-14]
  • issue first observed [2009-04-07] (see below)

• various SAM test failures Fri [2009-04-10] and early Sat [2009-04-11] resulting from cluster NIS server going offline (accessed by WNs)
  • hard restart required to recover from kernel panic on server
  • to be followed-up as LCG accounts were to be fully decoupled from cluster NIS

• increased datadisk from 5 to 6 TB [2009-04-12]

• new Westgrid - UA cluster (storage + cpu) being commissioned
  • water cooling loop issues being debugged by campus facilities services
  • new ATLAS Tier-2 network for Grid services nodes defined and configured in Hepnet router at UofA
    • BGP configuration within router setup over lightpath to TRIUMF-T1 to be done this week
    • connectivity of ATLAS service nodes to new network to be done this week along with single-mode fiber connection between Westgrid and HEPnet switches
  • SGI debugging image install issues as part of ATLAS service node OS installation

Outstanding Issues

• site currently failing ganga filestager tests with 14.2.25 and is blacklisted . This is under investigation (AdS) [2009-04-07]
• pnfsd.log to be added to log-rotate [2009-03-31]
• few WNs to be debugged with respect to missing stderr/stdout
• still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• SE problems observed in user analysis jobs as part of Hammercloud test 214 in Filestager [2009-04-08]
  • failed dccp and transfer terminations
    • load related problem with pnfs; restarted entire dcache instance [2009-05-03]
    • plan to move some dcache services to other server(s) for existing SE [2009-04-08]

• 45 failed of 209 Panda analysis (non-Hammercloud) jobs [2009-04-09]
  • wormhole SE and site BDII disk space became full halting bdii ; restarted

• pnfs write permission error to datadisk from FTS transfers reported via ggus ticket and resolved [2009-04-13]

Outstanding Issues

• current SE instabilities

Victoria

Recent Issues

• long running jobs observed during Hammercloud runs [2009-04-08]
• bug report filed by Asoka with Ganga in regards to failing robot filestager tests using Ganga 5.1.9 compared to those which were successful during Hammercloud (Ganga 5.1.8) [2009-04-14]
• increased datadisk to 5 TB [2009-04-11]

Outstanding Issues

Toronto

Recent Issues

• site downtime extended from original (as described below)
• new dCache almost complete but still to be fully debugged

Outstanding Issues

• scheduled maintenance downtime between 2009-03-25 and 2009-03-30 for ongoing dCache 1.9 and chimera upgrade
  • failing SFT for job submission since 2009-03-10 with general SE problems since early March (GGUS ticket 46825)
  • split head node with upgrade to chimera from PNFS ; scheduled downtimes followed

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• partial cooling failure [2009-04-11] with lower cpu (wn) capacity of about 30% until middle of next week

Outstanding Issues

• corrupt files reported likely as transient error during transfer [2009-04-04] - resolved upon retry?

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	16513	2539	134244497	9834872	86.7%	93.2%
VICTORIA-LCG2	1332	408	41003841	12977554	76.6%	76%
ALBERTA-LCG2	1096	362	20153344	2938070	75.2%	87.3%
SFU-LCG2	552	264	12355558	56123835	67.6%	18%
total	19493	3573	207757240	81874331	84.5%	71.7%

DashboardSummary.pdf

Failed Jobs

Still having difficulty getting 7 day information from Panda monitor so this is from the Dashboard:

TRIUMF
name,rank,value
EXEPANDA_JOBKILL_BYEXTIF,1,892
EXEPANDA_JOBKILL_BYSERVER,2,633
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,428
EXEPANDA_DQ2PUT_FILECOPYERROR,4,298
TRF_EXC ,5,215
TRF_OUTFILE ,6,21
EXEPANDA_JOBKILL_BYPILOT,7,20
EXEPANDA_JOBEXPIRED_SIXDAYS,8,8
TRF_UNKNOWN ,9,7
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,10,5
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,11,5
EXEPANDA_DQ2_STAGEIN,12,3
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,13,2
EXEPANDA_GET_SIZEMISMATCH,14,1
TRFERROR,15,1

VICTORIA
name,rank,value
EXEPANDA_JOBKILL_BYEXTIF,1,225
EXEPANDA_JOBKILL_BYSERVER,2,145
EXEPANDA_DQ2GET_INFILE,3,19
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,9
TRF_UNKNOWN ,5,4
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,6,4
EXEPANDA_DQ2_STAGEIN,7,2

ALBERTA
name,rank,value
EXEPANDA_JOBKILL_BYEXTIF,1,260
EXEPANDA_JOBKILL_BYSERVER,2,97
TRFERROR,3,2
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,4,1
EXEPANDA_GET_FAILEDTOGETLFCREPLICA,5,1
EXEPANDA_DQ2GET_INFILE,6,1

SFU
name,rank,value
EXEPANDA_JOBKILL_BYEXTIF,1,212
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,46
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,3,3
TRF_UNKNOWN ,4,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,1

Notes

-- BryanCaron - 14 Apr 2009