Tier1 and Tier-2 Weekly Report 2009-02-24

SAM Results

• Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

• ops-availability-20090224.png:
  

• ops-reliability-20090224.png:
  

• atlas-availability-20090224.png:
  

• atlas-reliability-20090224.png:
  

APEL Accounting Publishing Status

• apel-gantt-20090224.png:
  

APEL Accounting Results

• apel-table-20090224.png:
  

General News

• distribution of reprocessed DPD datasets started by ATLAS DDM operations following clear-up of registrations backlog [2009-02-21]
  • subscriptions to TRIUMF and within CA cloud to T2s were on hold for 3 - 4 days to first clear up backlog of file registrations in the TRIUMF LFC (not a site issue)
  • allocated to T2s as per usual AOD distribution (55% SFU, 20% Alberta, 15% Toronto, 10% Victoria)
  • site transfer errors observed (see comments below for individual sites)
  • atlas-ddm-datadisk-2009-02-24.png:
    

• general outage of Alberta, Toronto and McGill lightpaths due to CANARIE Calgary ROADM POP reconfiguration for 8 hours [2009-02-20]
  • auto-failover to regular routed connections during this time
  • 123549828021295_now_1234634220.png (Right-click on image to enlarge):
    

• REMINDER TO T2 SITE ADMINS: Request for information for CA sites to WLCG Collaboration Board about installed resources
  • current resources only (non-NPF) needed for storage (TBs) and cpu (kSI2k)
  • please reply to Bryan with this info for Friday 2009-02-27

• updated SiteTests package from Asoka [2009-02-18]

• next round of Distributed Analysis tests for CA cloud postponed until 2009-03-03 to first work on resolving open issues at some sites (see below)
  • in the future will continue as planned with option for sites to opt-out if problems are occurring

Site Reports

Alberta

Recent Issues

• transfer failures between ALBERTA and TRIUMF reported [2009-02-23]
  • 3 of 4 pool nodes had memory problems since approximately 1 - 2 days (Out of Memory Error: Java heap space)
  • restarted dCache pool services on all 4 pools with increased memory allocation to 1024MB from 512MB. No issues observed since that time but transfers have been quiet.

• brief delay in upload of APEL statistics observed (see above); reran parser/publisher to generate new statistics update [2009-02-24]
• submitted request to CERN ROC for registration of new Alberta-WestGrid T2 site (CA-ALBERTA-WESTGRID-T2) [2009-02-19]

Outstanding Issues

• few WNs to be debugged with respect to missing stderr/stdout
• still need to add 2nd GigE nics to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• 1 powersupply out of 5 failed on one of the HP blade chassis - still operating but no redundancy [2009-02-17]

• network routing issues at SFU - resulting in approximately 60% of transfers to SFU failing (timeout when contacting SE) [2009-02-19]
  • wormhole quiet during this period - needs investigation
• investigations of failing ganga jobs/nodes via SiteTests ongoing - rpm and library packaging issues observed [2009-02-19]

• lock-file error during installation of 14.2.25.6 ; cleared [2009-02-21]
• site shutdown due to water leak in machine room ; restored services same day [2009-02-23]

Outstanding Issues

• Ganga jobs failing at SFU a large fraction of the time and outstanding since 2008-11-10 - 100% failure of previous weeks hammercloud DA tests - under ongoing investigation

Victoria

Recent Issues

• software directory issues for release area 14.5.1 - permissions problems fixed by site admins but release general use still failing [2009-02-23]

Outstanding Issues

• A. Agarwal now away on holiday (return after CHEP?); local admins keeping watch (and A. Agarwal remotely when possible)

Toronto

Recent Issues

• dCache pnfs migration from gdbm to postgres made during unscheduled downtime outage to overcome 2 GB file limit [2009-02-18]
  • FTS channels between TRIUMF and TORONTO paused during this period
• Panda test job errors reported along with a few SE errors during previous 24 hour period [2009-02-19]
• team-ticket submitted by ADC Ops regarind permissions issue when attempting writes Toronto SE [2009-02-20]
  • no free space on Toronto SE (MCDISK) ; updated to 10.9 TB [2009-02-20]
• SE problems still observed by analysis pilots [2009-02-23]
  • suspected load issue on SE due to now running additional postgres processes for pnfs
    • plus site added 58% additional WNs that were now accessing the dCache pools giving 300 concurrently running jobs
  • reduced allowed concurrent FTS transfers to Toronto from 10 to 3 ; no real change as a result
  • postgres tuning suggestions provided by TRIUMF
  • will change from lcg-cp to dccp for stage-in (R. Walker) [2009-02-24]

Outstanding Issues

• scripts for auto-updating grid-mapfile on glite 3.0 dCache admin node [2009-02-17]

TRIUMF

Recent Issues

• decrease in available MCDISK space observed by ADC Production Ops (221 of 250 TB) [2009-02-18]
  • actually dominated by AOD transfers between T1s and not MC production in CA cloud due to improved FTS performance following the upgrade
  • slow clean-up of space observed - questions regarding location of site services as a result
  • effective management of disk resources under discussion (ADC Ops)

• distribution of reprocessed DPD datasets started by ATLAS DDM operations following clear-up of registrations backlog [2009-02-21]

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	20830	380	316279086	4076347	98.2%	98.7%
SFU-LCG2	4456	268	115485080	3272213	94.3%	97.2%
TORONTO-LCG2	1703	1978	114843867	9524568	46.3%	92.3%
VICTORIA-LCG2	2026	89	106250301	338965	95.8%	99.7%
ALBERTA-LCG2	1268	173	38621590	2911995	88%	93%
total	30283	2888	691479924	20124088	91.3%	97.2%

DashboardSummary.pdf

Failed Jobs

Notes

-- BryanCaron - 24 Feb 2009