Tier1 and Tier-2 Weekly Report 2009-09-15

General News

• ADC News (20090910)
  • no ADC ops meeting last week (CERN holiday)
  • ESD-to-ESD reprocessing campaign commenced (2009-09-11) See link for details.

• Cloud operations [20090908]
  • to work around uncertainties in how the site queues are manually or automatically disabled within panda relative to the downtime status announced in GOCDB, if a CA site goes down for any reason cloud contacts (Bryan/Asoka/Di) will:
    • initiate setting the site queue(s) offline ourselves using the standard ADC procedure
    • email ADCoS shifters and ATCAN operations list for notification

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • Sites NOT running daily SiteTests currently are TORONTO and VICTORIA - please take action and correct this - thanks

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090915.png:
  

• ops-reliability-t2-20090915.png:
  

ATLAS VO

• atlas-availability-t2-20090915.png:
  

• atlas-reliability-t2-20090915.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090915.png:
  

APEL Accounting Results

• apel-statistics-20090915.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: None

Site Reports

Alberta

Recent Issues

• site BDII missing storage information due to filled partition on SE admin [20090908]
  • GGUS ticket https://gus.fzk.de/ws/ticket_info.php?ticket=51386 [solved]

• installation status: WLCG on WestGrid-Alberta cluster [2009-09-15]
  • dCache upgraded to 1.9.3-3 and under test
  • APEL-MON services to be installed this week

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• commissioning of bugaboo (SL5 WNs) within Panda for production underway [20090908]
  • information system updates/fixes re: atlas releases - done
  • jobmanager issues overcome by yaim reconfigure of lcg-CE - done
  • python and version dependency issues via Ganga currently under investigation
  • production queue online in Panda [20090914]

Outstanding Issues

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• unscheduled downtime for approx. 4 hours [20090912]
  • unscheduled power outage to the University brought down site services and nodes

• MCDISK reported full ADCoS [20090914]
  • GGUS https://gus.fzk.de/ws/ticket_info.php?ticket=51522
  • Savannah ticket https://savannah.cern.ch/bugs/?55642

Outstanding Issues

Toronto

Recent Issues

• SE access issues reported by ADCoS [20090914]
  • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51524

• FileStager issues observed (GangaRobot and SiteTests) [20090909]
  • restarted srm and door services [20090909] [solved]

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• 2h scheduled downtime to apply latest kernel updates [20090909]
• scheduled downtime announced for [20090916] for dcache upgrade and other middleware reconfiguration

Outstanding Issues

• scheduled downtime for dCache patch application [2009-07-30]
  • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	958	49	14958916	490031	95.1%	96.8%
TORONTO-LCG2	684	56	27269735	4732766	92.4%	85.2%
VICTORIA-LCG2	521	46	20242075	1103359	91.9%	94.8%
ALBERTA-LCG2	266	4	5985891	16461	98.5%	99.7%
SFU-LCG2	116	9	1729867	109900	92.8%	94%
total	2545	164	70186484	6452517	93.9%	91.6%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor: FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
TRF_SVRINIT ,1,33
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,2,9
EXEPANDA_JOBKILL_SIGTERM,3,2
EXEPANDA_JOBKILL_BYPILOT,4,2
TRF_UNKNOWN ,5,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,1

Alberta:

name,rank,value
TRF_SVRINIT ,1,4

SFU:

name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,4
TRF_SVRINIT ,2,2
EXEPANDA_DQ2_STAGEIN,3,1
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,1
EXEPANDA_PUT_LFCMODULEIMPORT,5,1

Toronto:

name,rank,value
TRFERROR,1,38
TRF_SVRINIT ,2,14
TRF_UNKNOWN ,3,2
EXEPANDA_DQ2PUT_FILECOPYERROR,4,1
EXEPANDA_DQ2_STAGEIN,5,1

Victoria:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,22
TRF_SVRINIT ,2,13
EXEPANDA_DQ2_STAGEIN,3,5
EXEPANDA_JOBKILL_BYPILOT,4,4
EXEPANDA_DQ2GET_INFILE,5,1
TRFERROR,6,1

Notes

• Very few jobs for the past week.
• Toronto had SE issues GGUS 51524
• Victoria - unexpected power outage (~4 hours, 12 Sep).

-- BryanCaron - 14 Sep 2009