Tier1 and Tier-2 Weekly Report 2009-10-13

General News

• ADC News [20091008]
  • ATLAS Tier-1-2-3 Jamboree [20091013] See agenda http://indico.cern.ch/conferenceDisplay.py?confId=66012
  • batch of HammerCloud tests scheduled for this week on CA sites to determine optimal file access methods for each site (and enter into central table)

• end-of-life announced for dCache versions < 1.9.4 (see dCache user-forum announcement)

• announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3
    • TRIUMF: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-t2-availability-20091013.png:
  

• ops-t2-reliability-20091013.png:
  

ATLAS VO

• atlas-t2-availability-20091013.png:
  

• atlas-t2-reliability-20091013.png:
  

APEL Accounting

APEL Publishing Status

• apel-status-20091013.png:
  

APEL Accounting Results

• apel-statistics-20091013.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25980/server/pandamon/query?mode=ddm_req&filter=on&dpat=*&ver=ANY&status=ANY&cloud=CA&site=ALL&period=30&req_id=&action=List
  • This past week:

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 and the CA-ALBERTA-WESTGRID-T2 site [20091005]
  • ggus ticket https://gus.fzk.de/ws/ticket_info.php?ticket=52100
  • already replied to indicate all WNs previously updated in august to non-vulnerable kernel version 2.6.30.5
  • ticket incorrectly selected for CA-ALBERTA-WESTGRID-T2 site and subsequently reassigned by CERN ROC to correct site

• single instance SAM failures reported via automatic email notification
  • CE-sft-lcg-rm [20091012]
  • sBDII-sanity [20091011]

• SE BDII outage following scheduled downtime temporarily resulting in site added to GangaRobot blacklist (notification from Asoka) [20091008]
  • incorrectly attempting to access SE via gfal instead of dcap
  • coincident with single reported CE-sft-lcg-rm error notification

• user analysis job errors reported (54 finished, 44 failed) [20091007]
  • http://panda.cern.ch:25980/server/pandamon/query?job=1023589412
  • temporary network error (?)

• T2 squid cache installation [20091013]
  • assigned ip address to service node: scache01.atlas.ualberta.ca (142.244.105.82)
  • test installation started thus far with base OS now through VM on ATLAS admin node; squid rpms to follow this week

• installation status: WLCG on WestGrid-Alberta cluster [20091006]
  • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
  • 3-day cycle of SAM tests spawned by CERN ROC at the site request
  • dCache configuration and information provider tested independently with ATLAS VO credentials - successful [20091005]
    • looking to modify dCache configuration to have multihomed pools to separate internal and external traffic

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52165 bugaboo CE and SAM tests (awaiting confirmation from site before ticket closure)

• apel publishing information missing for October (see above plots) [20091013]

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 [20091005]
  • system job enacted within moab so that no new jobs to be run on remaining WNs until system update first run

• adler32 checksum error in user analysis jobs; 132 instances [20091006] and 10 instances [20091007]
  • http://panda.cern.ch:25980/server/pandamon/query?job=1023470996
  • http://panda.cern.ch:25980/server/pandamon/query?job=1023551372

• production issues related to SE activity [20091008]
  • communication error / timeout during lfc_setfsizeg and LFC python module import
  • communication error on send
  • staging input file failure ; dccp error accessing DBRelease on hotdisk
  • see 20091008 production summary email notes

• T2 squid cache deployment [20091013]
  • likely to deploy on current bugaboo CE which has under-utlized hardware and is separate from torque/moab for site

• bugaboo (SL5) commissioning within atlas
  • see matrix of base and cache release validation and installation at http://spreadsheets.google.com/pub?key=th4DG22nRYxcdltZ0Upy5kQ&output=html

Outstanding Issues

• bugaboo (SL5) commissioning within atlas (see above link for status)

• snowpatch - fairshare status wrt different pilots not running [20090918]
  • all are members of the overall atlas group - sub-group fairshare (?)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52195 DATADISK full and permissions (ticket awaiting verification before closing)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52062 DATADISK full (ticket still to be closed)

• ggus ticket 52099 resolved during unscheduled downtime [20091008]
  • panda queues for site set offline during downtime
  • cancelled scheduled downtime originally booked for 20091009

• SE space issues
  • DATADISK https://gus.fzk.de/ws/ticket_info.php?ticket=52062
  • DDM Ops reduced CR and reprocessed data shares to site by 5% at request of CA cloud [20091006]
  • all dCache pools reported full [20091007]
    • central cleaning of proddisk started (see online link for progress information)
    • fts transfer channels set to inactive pending liberation of needed space
    • suggested running cleanup script to search for any files on pools which do not have a valid pnfsID and can therefore be removed
    • possible 1 to 2 TB of dark data on site disk basedon LFC information (see http://gridinfo.triumf.ca/lfc/)
    • DDM requested CA cloud DDM team to run script to detect dark data [20091008]
      • still to be done with action for Bryan with results to be sent to DDM Ops

• production job errors
  • staging input file failed ; missing entry for site SE in lcg-bdii [20091007]

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://savannah.cern.ch/bugs/?56816 PRODDISK
  • https://savannah.cern.ch/bugs/?56710 PRODDISK
  • https://savannah.cern.ch/bugs/?56644 DATADISK
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51844 Node 217 lfc_getreplicas failed - security plugin not loaded (bad node 018 removed and resolved)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51912 HOTDISK permissions (awaiting site response)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51945 DATADISK full (site responded but still open)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52153 lcg_cp: connection timed out (awaiting site response)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52187 dBRelease file access issue (site responded; should now be resolved but still open)

• DATADISK reported full through multiple DDM savannah bug tickets and ggus tickets
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51945 and https://savannah.cern.ch/bugs/?56644
  • site share of CR and reprocessed temporarily set to 0% to compensate until scinet comes online

• production job failures [20091006] and [20091007]
  • possible WNs with too little space left on local disk (nodes 196, 199, 201)
  • SE access errors:
    • lcg_cp: communication error on send
    • failure accessing specific DBRelease file (same as at SFU, BNL) in HOTDISK; test access to file is ok
    • too little space left on proddisk area : see https://savannah.cern.ch/bugs/?56710 and https://savannah.cern.ch/bugs/?56816
  • missing input files reported from september migration from defunct storage pool server
    • cleanup of pnfs database and dq2 catalogue required to remove unavailable entries (to be requested to DDM ops savannah)

• updated space token allocations [20091007]
  • ATLASDATADISK -1 TB (now at 15 TB)
  • ATLASPRODDISK +1 TB (now at 2.5 TB)

• CA-SCINET-T2 installation:
  • dCache 1.9.4 now on pools but finding some configuration issues
  • dCache admin node being updated to CentOS 53
  • DDN firmware update debugging from vendor during downtime
  • designated node for squid cache installation

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51863 dccp failed with output (resolved but ticket still needs to be closed)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51882 no pool candidates (resolved but ticket still needs to be closed)

• checksum mismatch for file [20091009]
  • https://savannah.cern.ch/bugs/?56811

• ce3 (SL5) commissioning within atlas
  • see matrix of base and cache release validation and installation at http://spreadsheets.google.com/pub?key=th4DG22nRYxcdltZ0Upy5kQ&output=html

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	14484	604	299019720	1883602	96%	99.4%
SFU-LCG2	3044	58	61868494	515882	98.1%	99.2%
TORONTO-LCG2	1064	1977	32570074	47881838	35%	40.5%
VICTORIA-LCG2	853	237	20151420	6471200	78.3%	75.7%
ALBERTA-LCG2	906	35	30934426	588115	96.3%	98.1%
total	20351	2911	444544134	57340637	87.5%	88.6%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,561
EXEPANDA_DQ2PUT_FILECOPYERROR,2,16
TRFERROR,3,12
TRF_UNKNOWN ,4,6
TRF_SEGFAULT ,5,5
EXEPANDA_JOBKILL_BYPILOT,6,3
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,1
EXEPANDA_RUNJOBEXCEPTION,8,1

Alberta:

name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,17
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,10
TRFERROR,3,6
TRF_UNKNOWN ,4,1
TRF_SVRINIT ,5,1

SFU:

name,rank,value
TRFERROR,1,29
EXEPANDA_DQ2_STAGEIN,2,9
EXEPANDA_GET_REPLICANOTFOUND,3,8
EXEPANDA_PUT_LFCMODULEIMPORT,4,5
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,3
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,6,2
EXEPANDA_PUT_GLOBUSSYSTEMERROR,7,1
TRF_UNKNOWN ,8,1

Toronto:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,955
EXEPANDA_DQ2PUT_FILECOPYERROR,2,955
EXEPANDA_JOBKILL_BYPILOT,3,22
EXEPANDA_GET_NOSUCHFILE,4,16
EXEPANDA_DQ2GET_INFILE,5,14
EXEPANDA_GET_REPLICANOTFOUND,6,5
TRF_SEGFAULT ,7,5
TRFERROR,8,4
EXEPANDA_JOBKILL_SIGTERM,9,1

Victoria:

name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,227
EXEPANDA_DQ2_STAGEIN,2,4
EXEPANDA_JOBKILL_SIGTERM,3,3
EXEPANDA_DQ2GET_INFILE,4,1
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,1
EXEPANDA_GET_GLOBUSSYSTEMERROR,6,1

You can find the meanings of the transform error codes on this link.

Notes

• How we fared to other ATLAS sites ...

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
FZK	204064	6998	1823687031	82814113	96.7%	95.7%
BNL	125214	16864	3731941987	284958832	88.1%	92.9%
NDGF	46904	3152	991534200	26362575	93.7%	97.4%
None	45606	3246	570026688	14868800	93.4%	97.5%
LYON	30713	4253	713353724	95822531	87.8%	88.2%
TRIUMF	20351	2911	444544134	57340637	87.5%	88.6%
ASGC	20132	1745	155455023	1063711	92%	99.3%
SARA	13426	695	200676366	18850899	95.1%	91.4%
CNAF	7430	2194	121771413	19141354	77.2%	86.4%
PIC	719	6581	24732239	364293	9.8%	98.5%
RAL	3870	2000	77307719	16502432	65.9%	82.4%
CERN	0	1	0	153	0%	0%
total	518429	50640	8.855030524e+09	618090330	91.1%	93.5%

• Issues:
  • Victoria and Toronto had issues with SE (space got filled up) so errors above are understood.

-- BryanCaron - 07 Oct 2009