Tier1 and Tier-2 Weekly Report 2009-09-08

General News

• ADC News (20090908)
  • note-worthy items coming out of various ADC sessions during the ATLAS Software and Computing Week (text)

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • Sites NOT running daily SiteTests currently are TORONTO and VICTORIA - please take action and correct this - thanks

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090908.png:
  

• ops-reliability-t2-20090908.png:
  

ATLAS VO

• atlas-availability-t2-20090908.png:
  

• atlas-reliability-t2-20090908.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090908.png:
  

APEL Accounting Results

• apel-statistics-20090908.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: None

Site Reports

Alberta

Recent Issues

• installed lcg-CE on SL48 node of WestGrid-Alberta cluster [2009-08-14]
  • successful setup of site-BDII and submissions of jobs via gatekeeper to WNs (including ATLAS Canada VOMS roles)
  • job submissions from CE to WNs successful (stderr/stdout delivery) following tested fix to pbs_mom config (thanks Sergey for suggested fix)
    • pbs_mom config being applied to remaining WNs by WestGrid-UofA admins
  • baseline dCache admin and pool services installation performed on three nodes (admin + 2 pools) [20090824]
    • clean 1.9.1-11 installation with chimera ; in preparation for update to 1.9.3-3 on system before further validation
  • APEL-MON services installation targetted for this week (Wed/Thurs)

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• required production releases installed on bugaboo (SL5 WNs) and tags published [20090824]
  • missing x86_64 and i386 dependencies for root to be corrected

• unscheduled downtime announced for 20090826 to 20090831 for various upgrades (lustre, kernel, torque/moab) [20090820]

• analysis errors observed in regards to missing files (see Outstanding Issues entry below)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

Outstanding Issues

• savannah ticket for approximately 20k missing files at SFU dCache that are still registered in DDM [2009-08-14]
  • https://savannah.cern.ch/bugs/?54369
  • clean-up of LFC done by DDM Ops and completed [20090825]
    • files being transferred again to MCDISK for the whole dataset that was deleted since significant fraction was lost

• recurrent problems with missing files expected to exist on SE but unavailable during analysis and production jobs [2009-07-22]
  • see previous item

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• unscheduled downtime for approx. 4 hours [20090821]
  • unscheduled power outage to the University brought down site services and nodes

• analysis job failures with timeouts in file copy from SE (27) [20090819]
  • network bandwidth to SE (which nodes?) saturated during HammerCloud
  • maximum job slots for atlas pilots user (atlaspt) misconfigured to accept too high (100) number of concurrently running jobs ; reduced to 40

• some analysis job failures observed with input file staging timeouts (32) and put file copy timeouts (4) [2009-08-11]
  • example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-CA/12870924.5.out
  • example: http://condorg.triumf.ca/analysisPilots/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-GEN/12870111.7.out

Outstanding Issues

Toronto

Recent Issues

• SE issues [20090819]
  • failing SiteTests, production and analysis jobs, HammerCloud tests observed [20090819]
  • restart of srm services corrected the problem and was confirmed with successful SiteTests run by site admin [20090819]
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51019 [solved]

• timeouts on transfers from TRIUMF [20090820]
  • restarted hung pool node which is flakey and cannot be accessed remotely via control box
  • site additionally marked offline for production by ADCoS shifters * GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51026 [solved]

• migration of contents from faulty dCache pool (approx 800 GB) to other pools [20090824]

• issues with connectivity to TRIUMF via end-to-end lightpath from Toronto (and other sites) [20090824]
  • network outage affecting Toronto and Alberta lightpaths to TRIUMF for approx. 2 minutes
  • routing issue with the BCnet primary DNS server was the cause

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• request to clean some dCache directories [20090823]
  • central cleaning not working properly so request made to site
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51079 [open]

• LFC permissions issue fixed [20090820]
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51036 [closed]

• brief 1 hour TRIUMF scheduled downtime for oracle cpu and linux package update [20090820]

Outstanding Issues

• TRIUMF without ATLAS Atlas-15.3.1/i686-slc4-gcc34-opt [2009-08-17]
  • However it appears to truly be a Panda issue as the release is installed at the site
  • issue resolution in progress by Panda people
    • GUS: https://gus.fzk.de/ws/ticket_info.php?ticket=50947&from=search (still open but should be closed??)
    • Savannah: http://savannah.cern.ch/bugs/?54373

• scheduled downtime for dCache patch application [2009-07-30]
  • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	13635	707	207360694	584048	95.1%	99.7%
SFU-LCG2	3107	4	50442429	145615	99.9%	99.7%
TORONTO-LCG2	2788	34	114168551	460650	98.8%	99.6%
VICTORIA-LCG2	1239	37	40874582	277066	97.1%	99.3%
ALBERTA-LCG2	1178	42	23289034	0	96.6%	100%
total	21947	824	436135290	1467379	96.4%	99.7%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,655
TRF_SVRINIT ,2,40
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,3,7
EXEPANDA_JOBKILL_BYPILOT,4,3
TRF_UNKNOWN ,5,1
EXEPANDA_RUNJOBEXCEPTION,6,1

Alberta:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,42

SFU:

name,rank,value
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,1,1
EXEPANDA_DQ2GET_CANTCALL,2,1
EXEPANDA_PUT_GLOBUSSYSTEMERROR,3,1
EXEPANDA_JOBKILL_BYPILOT,4,1

Toronto:

name,rank,value
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,1,21
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,6
EXEPANDA_JOBKILL_BYPILOT,3,3
TRF_SEGFAULT ,4,2
TRFERROR,5,1
TRF_UNKNOWN ,6,1

Victoria:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,22
EXEPANDA_DQ2GET_INFILE,2,8
EXEPANDA_DQ2_STAGEIN,3,3
TRFERROR,4,2
EXEPANDA_DQ2PUT_FILECOPYERROR,5,1
EXEPANDA_JOBKILL_BYPILOT,6,1

Notes

-- BryanCaron - 08 Sep 2009