Tier1 and Tier-2 Weekly Report 2009-09-29

General News

• ADC News [20090924]
  • DDM
    • throughput tests scheduled for around Oct 4 between T0-T1 for tests of FTS 2.2; details to be circulated
    • grey/dark data cleaning - scripts for dCache to go into SVN some instructions possibly on twiki already
    • pre-MC09 disk cleanup - deletion of 14 TeV samples from release 12 and 13 planned along reducing numbers of replicas for data not accessed in the last 6+ months
    • new data transfer request interface (DaTRI) in preproduction (link)
  • Reprocessing from ESD
    • around 98% jobs done as of 20090928 with bulk replication in progress
  • ATLAS UAT (User Analysis Test) planned for 21-23 Oct (28-30 Oct as backup) (twiki link)
    • objective to get as many user analysis jobs running over world-wide resources concurrent with normal analysis activities, with metrics from pathena and ganga
    • follow-on test to STEP09 exercise and last one before datataking
    • 5 AOD container datasets (500M events) to be distributed to T1/T2s MCDISK - estimated at 55 TB - temporary space only to be deleted at the end of UAT

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3
    • TRIUMF: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-t2-availability-20090929.png:
  

• ops-t2-reliability-20090929.png:
  

ATLAS VO

• atlas-t2-availability-20090929.png:
  

• atlas-t2-reliability-20090929.png:
  

APEL Accounting

APEL Publishing Status

• apel-status-20090929.png:
  

APEL Accounting Results

• apel-statistics-20090929.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue (Link to be fixed)

• This past week:

Site Reports

Alberta

Recent Issues

• scheduled downtime planned and booked for Wed October 7 for 4 hours
  • system maintenance and updates to be applied (torque, maui, kernel, ...)

• handful of intermittent SAM errors observed
  • CE-sft-job, sBDII-sanity [20090926] and CE-sft-lcg-rm, SRMv2-get-SURLS, SRMv2-host-cert-valid [20090928]

• installation status: WLCG on WestGrid-Alberta cluster [2009-09-29]
  • request submitted to CERN ROC to begin certification process for CA-ALBERTA-WESTGRID-T2 site [20090918]
    • site entered into SAM tests [200909023]
  • APEL-MON services installed and tomcat/java issues debugged [20090923]
  • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• site unscheduled downtime [20090928]
  • DDN hardware failure

• reported SAM failures
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51645 [closed 20090929]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51784 [closed 20090929]

• bugaboo (SL5) commissioning within atlas
  • releases patched to use gcc34:
    • bugaboo - 14.2.10, 14.2.20, 14.2.21, 14.2.24, 14.2.25, 14.5.1
    • ce3 - 14.2.10, 14.2.20, 14.2.21, 14.2.24, 14.2.25
  • remaining releases still requiring patches:
    • bugaboo
    • ce3 - 14.0.40.x (failing KV on both clusters)
  • under test:
    • bugaboo - 14.1.0, 15.1.0
    • ce3 - 15.0.0, 15.1.0, 15.2.0
  • incorrect compiler still observed for releases 15.3.1 and 13.0.40 with newer AtlasLogin (no patch required)
  • different sets of release caches when comparing sites the two sites to be reconciled
  • release tagging within the information system: what is different about the process at each location? Tag procedure should be the same.
    • bugaboo: tags forced
    • ce3: tags not forced; waiting feedback so as to obtain successful KV
  • yum group install of glite-WN to resolve 32-bit v 64-bit problems ; clean-up of 32-bit prior required to install 64-bit (libvomsc_gcc64dbg.so.0) [20090922]

Outstanding Issues

• snowpatch - fairshare status wrt different pilots not running [20090918]
  • all are members of the overall atlas group - sub-group fairshare (?)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• DATADISK full from distribution of reprocessed 08cosmics [20090928]
  • token increased in size by +500 GB and FTS transfers restarted

Outstanding Issues

Toronto

Recent Issues

• lfc_getreplicas error reported by ADCoS [20090928]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51844

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• dccp error: no pool candidates ... reported by ADCoS [20090929]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51882
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51863

• new CE (ce3)
  • see SFU section above for notes regarding ce3 commissioning

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	34948	3160	97837688	12960551	91.7%	88.3%
TORONTO-LCG2	1824	236	25984829	290408	88.5%	98.9%
SFU-LCG2	1171	23	4019409	766768	98.1%	84%
ALBERTA-LCG2	1178	0	11450831	0	100%	100%
VICTORIA-LCG2	663	0	7879154	0	100%	100%
total	39784	3419	147171911	14017727	92.1%	91.3%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor: FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
TRF_UNKNOWN ,1,1964
EXEPANDA_DQ2_STAGEIN,2,591
EXEPANDA_DQ2PUT_FILECOPYERROR,3,308
TRFERROR,4,157
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,56
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,6,49
TRF_SEGFAULT ,7,12
TRF_EXC ,8,9
TRF_SVRINIT ,9,6
EXEPANDA_PUT_GLOBUSSYSTEMERROR,10,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,11,2
TRF_OUTFILE ,12,2
EXEPANDA_DQ2GET_INFILE,13,1
EXEPANDA_JOBKILL_BYPILOT,14,1

Alberta:

SFU:

name,rank,value
EXEPANDA_GRID-PROXY_INVALID,1,21
EXEPANDA_DQ2_STAGEIN,2,1
EXEPANDA_DQ2PUT_FILECOPYERROR,3,1

Toronto:

name,rank,value
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,1,206
EXEPANDA_DQ2_STAGEIN,2,20
TRF_SEGFAULT ,3,5
EXEPANDA_DQ2GET_INFILE,4,5

Victoria:

Notes

How we fared compared to other clouds:

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
BNL	278126	50129	2858362233	971719803	84.7%	74.6%
None	71691	18000	423176771	124977549	79.9%	77.2%
FZK	37780	17006	251804345	44784543	69%	84.9%
SARA	44953	5444	296887460	22351527	89.2%	93%
TRIUMF	39638	3404	146645575	13846449	92.1%	91.4%
NDGF	28419	2341	456785820	19423020	92.4%	95.9%
CNAF	22896	5607	233586013	51104988	80.3%	82%
RAL	6415	52	55207707	382091	99.2%	99.3%
PIC	456	2644	15272226	107183	14.7%	99.3%
CERN	634	980	6682525	367699	39.3%	94.8%
ASGC	1260	63	2541474	458336	95.2%	84.7%
LYON	0	171	0	4995560	0%	0%
total	532268	105841	4.746952149e+09	1254518748	83.4%	79.1%

From ADCoS meeting today:

active tasks from PandaDB: 742

by cloud: CA:13 CERN:1 DE:24 ES:4 FR:20 IT:9 ND:6 NL:14 UK:19 US:632

Issues:

• Toronto: SE issue with node018 reported on production issue 20090928 and also as GGUS ticket 51844.
• TRIUMF: SE issues since 20090928 - reported as GGUS ticket 51863.
• SFU: grid proxy issues seen when changing gLite versions from 32 to 64 bit - glitch in a small time window.

-- BryanCaron - 29 Sep 2009