Tier1 and Tier-2 Weekly Report 2009-08-18

General News

• ADC Operations News [2009-08-13] (notes)
  • ongoing discussion re: transition from SL(C)4 to SL(C)5
    • ATLAS has given confirmation to WLCG MB that sites can go ahead and upgrade WNs to SL(C)5
      • however the transition to SL(C)5 is not urgent as production releases will work with SL4 and SL(C)5 for quite some time
      • See the following for more details: https://twiki.cern.ch/twiki/bin/view/LCG/SL4toSL5wnMigration
  • premature DB release file deletion and scratch cleanup on WNs
    • distributed tarball deletion before job completion seen at some US T2s coincident with site scratch area cleanup via crontab
    • ongoing discussion to evolve to a job-oriented cleanup policy and mechanism
  • ATLAS Tier-1/2/3 Jamboree @ CERN Tuesday October 13, 2009 Agenda Link
    • purpose is to discuss planning, setup and operations in advance of the LHC restart and will be available via EVO

• HammerCloud: next iteration of HammerCloud tests of CA to start tomorrow (Wednesday) [20090818] (Asoka)

• Security notification (Sergey): local privilege escalation on Linux kernels
  • sites should have received notice and details of this exploit through the EGEE SECURITY CSIRTS list

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • Sites NOT running daily SiteTests currently are TORONTO and VICTORIA - please take action and correct this - thanks

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090818.png:
  

• ops-reliability-t2-20090818.png:
  

ATLAS VO

• atlas-availability-t2-20090818.png:
  

• atlas-reliability-t2-20090818.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090818.png:
  

APEL Accounting Results

• apel-statistics-20090818.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: central distribution of some unmerged e/gamma AOD samples due to inavailability of merged samples
  • ALBERTA-LCG2_MCDISK: mc09_valid*AOD*r738*
  • SFU-LCG2_MCDISK: mc09_valid*recon*AOD*r729* (3.7 TB)

Site Reports

Alberta

Recent Issues

• single SAM test failure for OPS tests (CE-sft-lcg-rm) [2009-08-11]

• installed lcg-CE on SL48 node of WestGrid-Alberta cluster [2009-08-14]
  • successful setup of site-BDII and submissions of jobs via gatekeeper to WNs (including ATLAS Canada VOMS roles)
  • currently debugging with WestGrid admins some peculiarities of job submissions to test nodes assigned to ATLAS
  • APEL-MON services installation (Wednesday) and dCache (Thurs/Fri) this week with target of next week for completion and extensive tests

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• installation tags forced on bugaboo CE (for SL5 WNs) following permissions fix [2009-08-11]
  • confirmed success of SiteTests package execution [2009-08-13]
  • full installation of release tags expected to take several days
• some analysis job failures observed with dccp errors of 'get_data_failed' and checksum errors on several occasions in the passed week
  • example: http://condorg.triumf.ca/analysisPilots/2009-08-16/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-GEN/13051201.0.out
• • example: http://condorg.triumf.ca/analysisPilots/2009-08-15/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-GEN/13012336.8.out
• observed failure to use scratch on 5 specific WNs of snowpatch [2009-08-17]
  • missing /scratch/atlas directory on WNs - recreated
• savannah ticket for approximately 20k missing files at SFU dCache that are still registered in DDM [2009-08-14]
  • https://savannah.cern.ch/bugs/?54369

Outstanding Issues

• recurrent problems with missing files expected to exist on SE but unavailable during analysis and production jobs [2009-07-22]
• list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
  • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
• list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
• Sergey creating list of files that exist in pnfs but are not on any pools
• Sergey additionally creating list of files that exist in pnfs but are not on any pools
  • Oroginal reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• some analysis job failures observed with input file staging timeouts (32) and put file copy timeouts (4) [2009-08-11]
  • example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-CA/12870924.5.out
  • example: http://condorg.triumf.ca/analysisPilots/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-GEN/12870111.7.out

Outstanding Issues

Toronto

Recent Issues

• site CE problems reported associated with lcg-CA updates (1.31-1 release) [2009-08-13]
  • GUS ticket https://gus.fzk.de/ws/ticket_info.php?ticket=50885&from=search

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? No. Requires further follow-up (LG)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? No. Requires further follow-up (LG)

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• TRIUMF without ATLAS Atlas-15.3.1/i686-slc4-gcc34-opt [2009-08-17]
  • However it appears to truly be a Panda issue as the release is installed at the site
  • issue resolution in progress by Panda people
    • GUS: https://gus.fzk.de/ws/ticket_info.php?ticket=50947&from=search
    • Savannah: http://savannah.cern.ch/bugs/?54373

Outstanding Issues

• scheduled downtime for dCache patch application [2009-07-30]
  • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	1731	3951	16545475	2244438	30.5%	88.1%
VICTORIA-LCG2	137	31	4511764	431952	81.5%	91.3%
TORONTO-LCG2	56	47	754921	323258	54.4%	70%
ALBERTA-LCG2	83	16	956350	79620	83.8%	92.3%
SFU-LCG2	22	14	129911	111729	61.1%	53.8%
total	2029	4059	22898421	3190997	33.3%	87.8%

DashboardSummary.pdf

Failed Jobs

From Panda:

FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
EXEPANDA_Release_MISSING,1,3740
TRF_SVRINIT ,2,152
TRF_SEGFAULT ,3,25
TRF_GEO_MISMATCH ,4,10
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,9
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,6,6
EXEPANDA_JOBKILL_SIGTERM,7,3
EXEPANDA_JOBKILL_BYPILOT,8,3
TRF_UNKNOWN ,9,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,10,1

Alberta:

name,rank,value
TRF_SVRINIT ,1,16

SFU:

name,rank,value
TRF_SVRINIT ,1,13
EXEPANDA_JOBKILL_BYPILOT,2,1

Toronto:

name,rank,value
TRF_SVRINIT ,1,47

Victoria:

name,rank,value
TRF_SVRINIT ,1,27
EXEPANDA_JOBKILL_BYPILOT,2,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,3,1
TRF_OUTFILE ,4,1

Notes

There has not been many jobs on the CA cloud and the primary reason is a problem with the Panda pilot controller which has just been fixed this morning. See https://savannah.cern.ch/bugs/?54373 for details.

-- BryanCaron - 18 Aug 2009