Tier1 and Tier-2 Weekly Report 2009-08-25

General News

• ADC Operations News [2009-08-20] (agenda)
  • no significant news this past week
  • ATLAS Tier-1/2/3 Jamboree @ CERN Tuesday October 13, 2009 Agenda Link
    • purpose is to discuss planning, setup and operations in advance of the LHC restart and will be available via EVO

• HammerCloud Run from 20090819 to 20090820
  • See comments and links to results at: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/DAChallenge#August_2009

• updated kernels announced today for SL4 and SL5 with fix to advertised exploit [CVE-2009-2692]

• SL5:

If you recall, I brought up the issue of the compiler used in SL5 for the Athena Kits where it was set to gcc 4.1 instead of what is needed (gcc34 for some kits). I can confirm that from release 15.4.0 (which will not be installed at sites), this issue has been resolved. (The 15.4.0 kit is being shipped with their own gcc34 compiler.)
Be aware; to quote D. Quarrie:

"We have decided to drop the SLC4/gcc43 builds and to focus on the SLC5/gcc43 builds (both 32-bit and 64-bit) as the next major platforms. This is to reduce the burden on build machines, disk space and most particularly release builders. It also reflects the rapid deployment of SLC5 at CERN and throughout the GRID.
We are also increasing the priority on being able to validate the SLC5/gcc43 builds prior to ATLAS turn-on for physics later this year. Whether the 32-bit SLC5/gcc43 build will in fact become the primary production platform is a decision that will be taken after the validation is completed.
Given this increased priority, can I ask developers to regularly test all their code for these platforms as well as for the existing SLC4/gcc34 platform before making tag submissions. Thank you for your cooperation.
David
"

So sites could end up soon having to support both gcc34 and gcc43 kits on SL5 depending on job requirements.

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • Sites NOT running daily SiteTests currently are TORONTO and VICTORIA - please take action and correct this - thanks

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090825.png:
  

• ops-reliability-t2-20090825.png:
  

ATLAS VO

• atlas-availability-t2-20090825.png:
  

• atlas-reliability-t2-20090825.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090825.png:
  

APEL Accounting Results

• apel-statistics-20090825.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: None

Site Reports

Alberta

Recent Issues

• stale movers associated with dcap doors briefly (~ 2 hours) following reboot of NFS servers on WestGrid-UofA hosting 2 x 2 pools on current dCache Alberta setup [20090819]
  • restart of pool services corrected the problem

• issues with connectivity to TRIUMF via end-to-end lightpath from Alberta (and other sites) [20090824]
  • network outage affecting Toronto and Alberta lightpaths to TRIUMF for approx. 2 minutes
  • routing issue with the BCnet primary DNS server was the cause

• installed lcg-CE on SL48 node of WestGrid-Alberta cluster [2009-08-14]
  • successful setup of site-BDII and submissions of jobs via gatekeeper to WNs (including ATLAS Canada VOMS roles)
  • job submissions from CE to WNs successful (stderr/stdout delivery) following tested fix to pbs_mom config (thanks Sergey for suggested fix)
    • pbs_mom config being applied to remaining WNs by WestGrid-UofA admins
  • baseline dCache admin and pool services installation performed on three nodes (admin + 2 pools) [20090824]
    • clean 1.9.1-11 installation with chimera ; in preparation for update to 1.9.3-3 on system before further validation
  • APEL-MON services installation targetted for this week (Wed/Thurs)

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• required production releases installed on bugaboo (SL5 WNs) and tags published [20090824]
  • missing x86_64 and i386 dependencies for root to be corrected

• unscheduled downtime announced for 20090826 to 20090831 for various upgrades (lustre, kernel, torque/moab) [20090820]

• analysis errors observed in regards to missing files (see Outstanding Issues entry below)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

Outstanding Issues

• savannah ticket for approximately 20k missing files at SFU dCache that are still registered in DDM [2009-08-14]
  • https://savannah.cern.ch/bugs/?54369
  • clean-up of LFC done by DDM Ops and completed [20090825]
    • files being transferred again to MCDISK for the whole dataset that was deleted since significant fraction was lost

• recurrent problems with missing files expected to exist on SE but unavailable during analysis and production jobs [2009-07-22]
  • see previous item

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• unscheduled downtime for approx. 4 hours [20090821]
  • unscheduled power outage to the University brought down site services and nodes

• analysis job failures with timeouts in file copy from SE (27) [20090819]
  • network bandwidth to SE (which nodes?) saturated during HammerCloud
  • maximum job slots for atlas pilots user (atlaspt) misconfigured to accept too high (100) number of concurrently running jobs ; reduced to 40

• some analysis job failures observed with input file staging timeouts (32) and put file copy timeouts (4) [2009-08-11]
  • example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-CA/12870924.5.out
  • example: http://condorg.triumf.ca/analysisPilots/2009-08-11/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-GEN/12870111.7.out

Outstanding Issues

Toronto

Recent Issues

• SE issues [20090819]
  • failing SiteTests, production and analysis jobs, HammerCloud tests observed [20090819]
  • restart of srm services corrected the problem and was confirmed with successful SiteTests run by site admin [20090819]
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51019 [solved]

• timeouts on transfers from TRIUMF [20090820]
  • restarted hung pool node which is flakey and cannot be accessed remotely via control box
  • site additionally marked offline for production by ADCoS shifters * GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51026 [solved]

• migration of contents from faulty dCache pool (approx 800 GB) to other pools [20090824]

• issues with connectivity to TRIUMF via end-to-end lightpath from Toronto (and other sites) [20090824]
  • network outage affecting Toronto and Alberta lightpaths to TRIUMF for approx. 2 minutes
  • routing issue with the BCnet primary DNS server was the cause

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• hostname resolution problem [20090824]
  • problem with BCnet primary DNS server
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51123 [closed]

• request to clean some dCache directories [20090823]
  • central cleaning not working properly so request made to site
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51079 [open]

• LFC permissions issue fixed [20090820]
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51036 [closed]

• brief 1 hour TRIUMF scheduled downtime for oracle cpu and linux package update [20090820]

Outstanding Issues

• TRIUMF without ATLAS Atlas-15.3.1/i686-slc4-gcc34-opt [2009-08-17]
  • However it appears to truly be a Panda issue as the release is installed at the site
  • issue resolution in progress by Panda people
    • GUS: https://gus.fzk.de/ws/ticket_info.php?ticket=50947&from=search (still open but should be closed??)
    • Savannah: http://savannah.cern.ch/bugs/?54373

• scheduled downtime for dCache patch application [2009-07-30]
  • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	16123	1337	95982355	8573129	92.3%	91.8%
TORONTO-LCG2	1157	5731	15063203	35800670	16.8%	29.6%
SFU-LCG2	5464	301	30068865	1278550	94.8%	95.9%
VICTORIA-LCG2	2692	448	55720756	9592296	85.7%	85.3%
ALBERTA-LCG2	1661	25	23759416	908580	98.5%	96.3%
total	27097	7842	220594595	56153225	77.6%	79.7%

DashboardSummary.pdf

Failed Jobs

From Panda: FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
TRF_UNKNOWN ,1,1029
TRF_SEGFAULT ,2,202
EXEPANDA_JOBKILL_BYSERVER,3,48
TRFERROR,4,37
EXEPANDA_JOBKILL_BYPILOT,5,13
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,6,6
EXEPANDA_JOBKILL_SIGTERM,7,3

Alberta:

name,rank,value
TRFERROR,1,20
EXEPANDA_DQ2_STAGEIN,2,3
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,2

SFU:

name,rank,value
TRFERROR,1,100
TRF_SEGFAULT ,2,74
TRF_UNKNOWN ,3,66
EXEPANDA_DQ2PUT_FILECOPYERROR,4,18
EXEPANDA_PUT_LFCMODULEIMPORT,5,16
EXEPANDA_JOBKILL_BYPILOT,6,9
EXEPANDA_JOBKILL_BYSERVER,7,9
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,8,4
EXEPANDA_DQ2_STAGEIN,9,3
EXEPANDA_DQ2GET_CANTCALL,10,2

Toronto:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,5092
EXEPANDA_GET_GLOBUSSYSTEMERROR,2,417
EXEPANDA_JOBKILL_BYEXTIF,3,118
EXEPANDA_DQ2_STAGEIN,4,59
EXEPANDA_DQ2PUT_FILECOPYERROR,5,29
EXEPANDA_PUT_GLOBUSSYSTEMERROR,6,12
TRF_SEGFAULT ,7,1
TRFERROR,8,1
EXEPANDA_JOBKILL_BYSERVER,9,1
EXEPANDA_JOBKILL_BYPILOT,10,1

Victoria:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,391
EXEPANDA_JOBKILL_BYEXTIF,2,25
EXEPANDA_JOBKILL_BYPILOT,3,14
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,4,10
EXEPANDA_JOBKILL_BYSERVER,5,3
EXEPANDA_DQ2PUT_FILECOPYERROR,6,2
EXEPANDA_DQ2GET_INFILE,7,1
TRFERROR,8,1
TRF_OUTFILE ,9,1

Notes

Toronto is offline since 20 Aug; see

https://gus.fzk.de/ws/ticket_info.php?ticket=51019&from=ID
https://gus.fzk.de/ws/ticket_info.php?ticket=51026&from=ID

Victoria: experienced unexpected power outage which affected some jobs (heartbeat.) - Aug 20/21.

TRIUMF: some errors are 15.3.1 failures (now resolved) and transform errors (persistent to transient errors).

-- BryanCaron - 25 Aug 2009