Tier1 and Tier-2 Weekly Report 2009-10-06

General News

• ADC News [20091001]
  • Squid servers to be installed at all T1s and T2s formally announced today [20091006]
  • MCDISK cleanup last few days (approx 20 TB at T2s and 80 TB at T1) - see http://atlddm02.cern.ch/dq2/accounting/cloud_view/CANADASITES/30/
  • DDM Throughput Test (T0-T1)
    • Mon-Tues - approx 5 TB per day to each T1 (DATADISK)
    • Wed-Thur - larger file sizes adding up to approx 50 TB per day
    • TRIUMF-T1 DATADISK has 54 TB free [20091006]
  • ATLAS UAT (User Analysis Test) planned for 21-23 Oct (28-30 Oct as backup) (twiki link)
    • all container datasets now estimated at +80 TB - temporary space to be deleted at the end of UAT
      • ASGC, CNAF,PIC,SARA,TRIUMF will have medium and small containers - all containers will have project 'uat09'
      • proposed distribution within CA T2s given current constraints: SFU (65%), Alberta (30%), Toronto (3.5 %), Victoria (1.5%).

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3
    • TRIUMF: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20091006.png:
  

• ops-reliability-t2-20091006.png:
  

ATLAS VO

• atlas-availability-t2-20091006.png:
  

• atlas-reliability-t2-20091006.png:
  

APEL Accounting

APEL Publishing Status

• apel-status-2009006.png:
  

APEL Accounting Results

• apel-statistics-2009006.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25980/server/pandamon/query?mode=ddm_req&filter=on&dpat=*&ver=ANY&status=ANY&cloud=CA&site=ALL&period=30&req_id=&action=List
  • This past week: mc09*ExoGraviton*AOD to TRIUMF LOCALGROUPDISK (31 GB)

Site Reports

Alberta

Recent Issues

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 and the CA-ALBERTA-WESTGRID-T2 site [20091005]
  • ggus ticket https://gus.fzk.de/ws/ticket_info.php?ticket=52100
  • already replied to indicate all WNs previously updated in august to non-vulnerable kernel version 2.6.30.5

• single SRMv2-put test failure message received for ALBERTA-LCG2 [20091003]
• single CE-Sft-lcg-rm test failure message received for ALBERTA-LCG2 [20091006]

• ALBERTA-LCG2 site scheduled downtime planned and booked for Wed October 7 for 4 hours
  • system maintenance and updates to be applied (torque, maui, kernel, ...)

• installation status: WLCG on WestGrid-Alberta cluster [20091006]
  • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
  • 3-day cycle of SAM tests spawned by CERN ROC at the site request
  • dCache configuration and information provider tested independently with ATLAS VO credentials - successful [20091005]
    • created needed ATLAS spacetokens and awaiting arrival again of SAM ops jobs to validate site to production status
    • looking to modify dCache configuration to have multihomed pools to separate internal and external traffic

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 [20091005]
  • ggus ticket ?
  • 122 WNs updated so far with the rest done maximally in 9 days

• site unscheduled downtime [20090928]
  • DDN hardware failure
  • replacement controller arrived from the US and was installed 20091001 with Lustre services restored that night

Outstanding Issues

• bugaboo (SL5) commissioning within atlas
  • ongoing resolution of rpm dependencies when comparing 32-bit v 64-bit, ce3 v bugaboo ; 64-bit rpm versions installed on bugaboo
    • release caches to be tested for revalidation to determine if other 32-bit versions also required

• snowpatch - fairshare status wrt different pilots not running [20090918]
  • all are members of the overall atlas group - sub-group fairshare (?)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 [20091005]
  • ggus ticket https://gus.fzk.de/ws/ticket_info.php?ticket=52099

• DATADISK reported full once again
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52093
  • BC send reminder via Savannah repeating the request to reduce the CR share being sent to Victoria

• DATADISK full from distribution of reprocessed 08cosmics [20091001]
  • token again increased in size by +500 GB and requested that the CR share be adjusted by -5%.
  • http://savannah.cern.ch/bugs/?56239

• permission error writing to directory in DATADISK [20091002]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52009 (solved)

Outstanding Issues

Toronto

Recent Issues

• datadisk full [20091001]
  • increased in size from 4.5 to 4.8 TB
  • BC requested CR share to be adjusted by -5% since disk still nearly full
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51945 and http://savannah.cern.ch/bugs/?56363

• permissions issues in writing to SE (hotdisk) [20090930]
  • FTS errors observed and reported by DQ and ADCoS shifter: https://gus.fzk.de/ws/ticket_info.php?ticket=51912 [solved]

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• new CE (ce3) for SL5 cluster commissioning
  • From Asoka email [20091005]:
    • all releases ready for production (validated and correct compiler) except for:
      • 15.1.0, 14.4.0 and 13.0.40 - KV failure requiring input from Alessandro
      • 15.3.1, 14.5.2, 14.1.0 - caches being revalidated (compiler already ok) and then tag published
      • 14.2.22 - tag to be forced

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	19094	1438	125124506	8427432	93%	93.7%
SFU-LCG2	3022	201	26089975	364360	93.8%	98.6%
TORONTO-LCG2	2733	78	57550935	331187	97.2%	99.4%
VICTORIA-LCG2	1754	20	35606272	15823	98.9%	100%
ALBERTA-LCG2	1188	8	14245566	150322	99.3%	99%
total	27791	1745	258617254	9289124	94.1%	96.5%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,1314
TRF_SEGFAULT ,2,85
TRFERROR,3,8
TRF_OUTFILE ,4,8
TRF_SEGVIO ,5,6
TRF_EXC ,6,6
EXEPANDA_DQ2GET_INFILE,7,5
TRF_UNKNOWN ,8,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,9,1
EXEPANDA_JOBKILL_BYPILOT,10,1

Alberta:

name,rank,value
TRF_SEGFAULT ,1,7
TRFERROR,2,1

SFU:

name,rank,value
TRF_SEGFAULT ,1,166
TRFERROR,2,14
TRF_OUTFILE ,3,10
EXEPANDA_DQ2_STAGEIN,4,7
TRF_SEGVIO ,5,4

Toronto:

name,rank,value
TRF_SEGFAULT ,1,27
EXEPANDA_GET_NOSUCHFILE,2,22
EXEPANDA_DQ2_STAGEIN,3,15
EXEPANDA_JOBKILL_NOLOCALSPACE,4,4
TRF_OUTFILE ,5,4
TRFERROR,6,3
EXEPANDA_DQ2PUT_FILECOPYERROR,7,2
EXEPANDA_DQ2GET_INFILE,8,1

Victoria:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,9
TRF_SEGFAULT ,2,6
TRFERROR,3,2
TRF_OUTFILE ,4,1
EXEPANDA_DQ2GET_INFILE,5,1
EXEPANDA_ATHENA_START-FAILED-MISSING-INSTALLATION,6,1

You can find the meanings of the transform error codes on this link.

Notes

• How we fared to other ATLAS sites ...

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
BNL	190078	3244	4296509444	28382053	98.3%	99.3%
None	73704	8595	1233903572	117981746	89.6%	91.3%
FZK	70920	8575	1441829662	100440716	89.2%	93.5%
SARA	34401	4641	305660254	9607083	88.1%	97%
CNAF	33501	4282	531290252	26818172	88.7%	95.2%
NDGF	33393	1791	591172560	15307620	94.9%	97.5%
TRIUMF	27791	1745	258617254	9289124	94.1%	96.5%
RAL	26564	1830	391865805	15120611	93.6%	96.3%
LYON	16749	1685	101368402	6526164	90.9%	94%
ASGC	11405	195	172075795	1176202	98.3%	99.3%
PIC	323	3314	11001058	0	8.9%	100%
CERN	180	141	544067	299864	56.1%	64.5%
total	519009	40038	9.335838125e+09	330949355	92.8%	96.6%

• Issues:
  • TRIUMF: 29 Sep SE issues (also a GGUS ticket 51863)

-- BryanCaron - 29 Sep 2009