Tier1 and Tier-2 Weekly Report 2009-09-22

General News

• ADC News (20090917) (link)
  • DDM operations: auto-deny of user dq2-get read access to tape, user support, user quotas needed, HOTDISK, filled MCDISKs and share/pledge matching, SCRATCHDISK cleaning, site federation accounting/pledges
  • Cloud squads

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3
    • TRIUMF: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • Sites NOT running daily SiteTests currently are TORONTO and VICTORIA - please take action and correct this - thanks

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090922.png:
  

• ops-reliability-t2-20090922.png:
  

ATLAS VO

• atlas-availability-t2-20090922.png:
  

• atlas-reliability-t2-20090922.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090922.png:
  

APEL Accounting Results

• apel-stats-20090922.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: 2 datasets
  • mc08.105001.pythia_minbias.simul.HITS.e357_s462_tid025492 and mc08.105001.pythia_minbias.simul.HITS.e357_s462_tid025499 to TRIUMF-LCG2_LOCALGROUPDISK (350 GB)

Site Reports

Alberta

Recent Issues

• deployed requested HOTDISK spacetoken and area (1 TB) [20090915]

• installation status: WLCG on WestGrid-Alberta cluster [2009-09-22]
  • request submitted to CERN ROC to begin certification process for CA-ALBERTA-WESTGRID-T2 site [20090918]
    • enter site into SAM tests
  • APEL-MON services installed and under post-yaim debugging of tomcat [20090922]
  • site certification by CERN ROC started [20090921]
    • https://gus.fzk.de/ws/ticket_info.php?ticket=51713

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• deployed requested HOTDISK spacetoken and area (1 TB) [20090915]

• snowpatch - fairshare status wrt different pilots not running [20090918]

• bugaboo (SL5) commissioning within atlas
  • new CE added manually to central ganga blacklist to avoid default submission to these CEs during testing phase
  • added missing certificates in vomdir on WNs - successful execution of DQ2_COPY [20090915]
  • various missing packages based upon production and direct test submissions identified and installed (freetype, popt, ghostscript, ...)
  • installation of 32-bit and 64-bit glite-WN software for 64-bit WNs hardware with gLite packaging of conflicting executables - cleanup script from Denice
  • redhat-lsb required on WNs [20090918]
  • marked offline in panda production to resolve compilation issues associated with gcc34 via panda [20090916]
    • new AtlasLogin patch for pre-15.4.0 releases needed to correctly setup the appropriate compiler; checks of various releases in progress for bugaboo and ce03
      • removed previous manual change to enable gcc34 access on bugaboo

Outstanding Issues

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• deployed requested HOTDISK spacetoken and area (1 TB) [20090918]

• failed jobs reported by ADCoS shifter - error copy command self timed out [20090917]
  • ggus ticket https://gus.fzk.de/ws/ticket_info.php?ticket=51621

Outstanding Issues

Toronto

Recent Issues

• deployed requested HOTDISK spacetoken and area (1 TB) [20090920]

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• deployed requested HOTDISK spacetoken and area (1 TB)

• scheduled downtime for site maintenance [2009-09-16]
  • dcache upgrade, CE updae and LFC reconfiguration
  • panda queues marked offline by CA Cloud (DQ) during outage and restored afterward
• new CE (ce3) added manually to central ganga blacklist to avoid default submission to these CEs (with corresponding SL5 WNs) during testing phase
  • similar testing coincident with bugaboo commissioning

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	2270	8	44563747	1653	99.6%	100%
VICTORIA-LCG2	428	18	16212091	946015	96%	94.5%
TORONTO-LCG2	372	31	20618392	3323223	92.3%	86.1%
ALBERTA-LCG2	273	1	8250015	5031	99.6%	99.9%
SFU-LCG2	74	0	2133833	0	100%	100%
total	3417	58	91778078	4275922	98.3%	95.5%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor: FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,8

Alberta:

name,rank,value
TRFERROR,1,1

SFU:

Toronto:

name,rank,value
TRFERROR,1,22
EXEPANDA_DQ2_STAGEIN,2,7
EXEPANDA_DQ2PUT_FILECOPYERROR,3,2

Victoria:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,8
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,4
TRFERROR,3,3
TRF_OUTFILE ,4,2
EXEPANDA_JOBKILL_BYPILOT,5,1

Notes

Issues:

Toronto: SE problem (reported 18 Sep).

How do we compare with the other ATLAS sites:

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
BNL	194792	33392	3906062296	71162694	85.4%	98.2%
CNAF	9906	3932	181437427	25594941	71.6%	87.6%
NDGF	8395	734	125680920	15135930	92%	89.3%
None	6228	2336	258725319	40337462	72.7%	86.5%
FZK	7639	499	382541377	15317548	93.9%	96.2%
LYON	5040	1761	259259434	61464895	74.1%	80.8%
SARA	3398	692	190640008	28885513	83.1%	86.8%
TRIUMF	3417	58	91778078	4275922	98.3%	95.5%
PIC	2859	450	103175075	52468805	86.4%	66.3%
RAL	2332	262	58808781	15296270	89.9%	79.4%
CERN	480	401	3410893	341820	54.5%	90.9%
ASGC	336	30	5478986	444269	91.8%	92.5%
total	244822	44547	5.566998594e+09	330726069	84.6%	94.4%

From today's ADCoS weekly meeting Yuri's report

active tasks from PandaDB: 99
by cloud: CA:4 DE:3 ES:4 FR:4 IT:3 ND:7 NL:8 UK:2 US:64

-- BryanCaron - 16 Sep 2009