Tier1 and Tier-2 Weekly Report 2009-10-20

General News

• ADC News [20091020]
  • UAT exercise postponed to backup dates of 28 - 30 October
  • ATLAS Tier-1-2-3 Jamboree [20091013] See agenda http://indico.cern.ch/conferenceDisplay.py?confId=66012
    • Summary: http://indico.cern.ch/materialDisplay.py?contribId=11&sessionId=3&materialId=slides&confId=66012

• HammerCloud tests to CA sites last week. See https://twiki.atlas-canada.ca/bin/view/AtlasCanada/DAChallenge#October_2009

• end-of-life announced for dCache versions < 1.9.4 (see dCache user-forum announcement)

• announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

• REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
  • see dcache user-forum archives or forwarded email from Reda
  • all production versions affected and require advertised patches [2009-08-04]
    • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
  • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
    • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
    • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
    • WestGrid-Alberta: all nodes at 1.9.3-3
    • TRIUMF: all nodes at 1.9.3-3

• REMINDER #2: STEP09 Site Follow-up Homework [since 2009-07-14]
  • Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

• REMINDER #3: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults

• REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20091020.png:
  

• ops-reliability-t2-20091020.png:
  

ATLAS VO

• atlas-availability-t2-20091020.png:
  

• atlas-reliability-t2-20091020.png:
  

APEL Accounting

APEL Publishing Status

• apel-status-20091020.png:
  

APEL Accounting Results

• apel-statistics-20091020.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25980/server/pandamon/query?mode=ddm_req&filter=on&dpat=*&ver=ANY&status=ANY&cloud=CA&site=ALL&period=30&req_id=&action=List
  • This past week:

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51713 CA-ALBERTA-WESTGRID-T2 commissioning

• production job issues

• analysis job issues

• T2 squid cache installation [20091020]
  • assigned ip address to service node: scache01.atlas.ualberta.ca (142.244.105.82)
  • test installation started thus far with base OS now through VM on ATLAS admin node; squid rpms installed and successful with fnget tests to BNL and FZK Frontier

• installation status: WLCG on WestGrid-Alberta cluster [20091006]
  • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
  • 3-day cycle of SAM tests spawned by CERN ROC at the site request
  • dCache configuration and information provider tested independently with ATLAS VO credentials - successful [20091005]
    • looking to modify dCache configuration to have multihomed pools to separate internal and external traffic
    • debugging OPS VO group access errors during SAM tests of CE-sft-lcg-rm-rep
  • scheduled downtime booked into GOCDB for computing centre power maintenance on 2009-10-08 in the morning

Outstanding Issues

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52165 bugaboo CE and SAM tests (solved/closed)

• apel publishing information missing for October (see above plots) [20091013]

• notification from CERN ROC security contact in regards to CVE-2009-2692/8 [20091005]
  • system job enacted within moab so that no new jobs to be run on remaining WNs until system update first run - status?

• production job issues
  • adler32 errors during SE activity
    • http://condorg.triumf.ca/productionPilots/2009-10-14/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas/14654588.6.out
  • communication error / timeout during lfc_statg and LFC python module import
    • http://condorg.triumf.ca/productionPilots/2009-10-16/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas/14715093.5.out

• analysis job issues
  • adler32 checksum error in user analysis jobs; 2 instances [20091015], 15 instances [20091016], 18 instances [20091019]
    • http://panda.cern.ch:25980/server/pandamon/query?job=1024549256
    • http://condorg.triumf.ca/analysisPilots/2009-10-16/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-GEN/14702802.9.out
    • http://condorg.triumf.ca/analysisPilots/2009-10-18/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-GEN/14746575.4.out

• T2 squid cache deployment [20091016]
  • likely to deploy on current bugaboo CE which has under-utlized hardware and is separate from torque/moab for site
  • first installed squid rpms on test instance with unexpected result of deleting contents of root filesystem due to needed environment variable not being set by install scripts
    • feedback given to rpm provider at CERN

• bugaboo (SL5) commissioning within atlas
  • see matrix of base and cache release validation and installation at http://spreadsheets.google.com/pub?key=th4DG22nRYxcdltZ0Upy5kQ&output=html

Outstanding Issues

• bugaboo (SL5) commissioning within atlas (see above link for status)

• snowpatch - fairshare status wrt different pilots not running [20090918]
  • all are members of the overall atlas group - sub-group fairshare (?)

• apel publishing [20090825]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

• recurrent problems only seen at SFU SE:
  • dccp failure with "exception caughtascii codec can't decode byte ..." (example: http://condorg.triumf.ca/analysisPilotsCA/2009-08-21/snowpatch-hep.westgrid.ca_2119_jobmanager-lcgpbs-atlas-CA/13191271.0.out)
  • adler32 error (example: http://panda.cern.ch:25980/server/pandamon/query?job=1019391988)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09)
  • list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
    • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
  • list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
  • Sergey creating list of files that exist in pnfs but are not on any pools
  • Sergey additionally creating list of files that exist in pnfs but are not on any pools
    • Original reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52195 DATADISK full and permissions (ticket awaiting site reply)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52062 DATADISK full (solved/closed)

• production job issues
  • staging DB release file failed ; SE viewed as down in lcg-bdii [20091013]
  • low production activity during period of limited space availability on PRODDISK; increased by 500 GB [20091015]
  • communication error on send and timeout [20091016]

• analysis job issues
  • 2 instances of communication error on send and timeout [20091015] and 74 instances [20091016]
    • http://panda.cern.ch:25980/server/pandamon/query?job=1024549040
  • 1 instance of CGSI-gSOAP error reading token data header [20091015] and [20091018]
    • http://condorg.triumf.ca/analysisPilots/2009-10-15/lcg-ce.rcf.uvic.ca_2119_jobmanager-lcgpbs-general-GEN/14676115.5.out
  • no space left on device globus_xio: /scr out of space [20091016]

Outstanding Issues

• SE space issues
  • DATADISK https://gus.fzk.de/ws/ticket_info.php?ticket=52062
  • DDM Ops reduced CR and reprocessed data shares to site by 5% at request of CA cloud [20091006]
  • all dCache pools reported full [20091007]
    • central cleaning of proddisk completed last week
    • suggested running cleanup script to search for any files on pools which do not have a valid pnfsID and can therefore be removed - has this been done by the site?
    • possible 1 to 2 TB of dark data on site disk basedon LFC information (see http://gridinfo.triumf.ca/lfc/)
    • DDM requested CA cloud DDM team to run script to detect dark data [20091008]
      • ran script after central cleanup of PRODDISK (not to be done concurrently) identified only few files; so no results needed to be passed along to DDM Ops

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://savannah.cern.ch/bugs/?56816 PRODDISK (closed)
  • https://savannah.cern.ch/bugs/?56710 PRODDISK (open)
  • https://savannah.cern.ch/bugs/?56644 DATADISK (open)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51912 HOTDISK permissions (awaiting site response)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51945 DATADISK full (on hold)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52153 lcg_cp: connection timed out (awaiting site response)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=52187 dBRelease file access issue (on hold but awaiting further site response before ROC will close)

• production job issues
  • put error copying output from wn to localSE [20091016]
    • gplasma timeout and communication error on send
    • http://condorg.triumf.ca/productionPilots/2009-10-14/bigmac-lcg-ce2.physics.utoronto.ca_2119_jobmanager-pbs-atlas/14655046.3.out
  • globus_xio errors: communication error on send [20091017]
    • http://condorg.triumf.ca/productionPilots/2009-10-15/bigmac-lcg-ce2.physics.utoronto.ca_2119_jobmanager-pbs-atlas/14686373.9.out

• analysis job issues
  • lcg_cp communication error on send, 5 instances [20091016]
  • lcg_cp connection timeout, 174 instances [20091016]
  • communication error on send and globus_xio error, 14 instances [20091016]
    • http://condorg.triumf.ca/analysisPilots/2009-10-15/bigmac-lcg-ce2.physics.utoronto.ca_2119_jobmanager-pbs-atlas-GEN/14682560.8.out
  • restarted srm and door services, as well as pool node which encountered spike in load. Being monitored [20091019]

• Toronto fairshare settings [20091015]
  • large number (300) of long running production jobs appear to be blocking execution of analysis jobs; maxjobs for production reduced to 200 (100 each generic atlas and atlasprd)

• SAM test failures [20091019]
  • reporting no space left on device; under investigation

• CA-SCINET-T2 installation:
  • new site certification by CERN ROC: https://gus.fzk.de/ws/ticket_info.php?ticket=49823
  • dCache 1.9.4 now on pools but finding some configuration issues
  • dCache admin node being updated to CentOS 53
  • DDN firmware update debugging from vendor during downtime
  • designated node for squid cache installation

Outstanding Issues

• Still have two (previously) open GGUS tickets to resolve GIP information for the dCache 1.9
  installation that require follow-up.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48236 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.
  • https://gus.fzk.de/ws/ticket_info.php?ticket=48235 - ticket closed - requires upgrade from dCache 1.9.0-8 to 1.9.1-7. Is this done? From SiteAdmin: No but workaround in place.

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51863 dccp failed with output (closed 20091020)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=51882 no pool candidates (closed 20091020)

• dpool13 node down twice [20091016]

• production job issues
  • dccp failed ; alarm clock [20091013]
  • put error copying file from wn to localSE; CGSI_SOAP error reading token data header [20091013]
    • http://condorg.triumf.ca/productionPilots/2009-10-11/ce2.triumf.ca_2119_jobmanager-lcgpbs-atlas/14595504.23.out
  • lcg_cr communication error on send [20091013]
    • http://condorg.triumf.ca/productionPilots/2009-10-09/ce1.triumf.ca_2119_jobmanager-lcgpbs-atlas/14537703.17.out

• ce3 (SL5) commissioning within atlas
  • see matrix of base and cache release validation and installation at http://spreadsheets.google.com/pub?key=th4DG22nRYxcdltZ0Upy5kQ&output=html

Outstanding Issues

• checksum mismatch for file [20091009]
  • https://savannah.cern.ch/bugs/?56811 (open)

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	10604	75	172484842	9030795	99.3%	95%
SFU-LCG2	1193	117	28411680	3230553	91.1%	89.8%
ALBERTA-LCG2	914	41	27294736	2684843	95.7%	91%
VICTORIA-LCG2	820	91	32203890	10297920	90%	75.8%
TORONTO-LCG2	314	119	36549859	17959129	72.5%	67.1%
total	13845	443	296945007	43203240	96.9%	87.3%

DashboardSummary.pdf

Failed Jobs

FailedJobs.pdf

From Dashboard:

TRIUMF:

name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,26
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,19
EXEPANDA_JOBKILL_BYPILOT,3,13
TRF_OUTFILE ,4,11
TRF_SEGFAULT ,5,6

Alberta:

name,rank,value
EXEPANDA_GET_NOSUCHFILE,1,25
EXEPANDA_JOBKILL_BYPILOT,2,13
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,3,2
TRF_OUTFILE ,4,1

SFU:

name,rank,value
EXEPANDA_GET_NOSUCHDBRELEASEFILE,1,80
EXEPANDA_JOBKILL_SIGTERM,2,16
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,3,8
EXEPANDA_DQ2_STAGEIN,4,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,4
TRF_SEGFAULT ,6,2
EXEPANDA_PUT_LFCMODULEIMPORT,7,1

Toronto:

name,rank,value
EXEPANDA_PUT_GLOBUSSYSTEMERROR,1,51
EXEPANDA_JOBKILL_BYPILOT,2,45
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,8
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,4,6
TRF_OUTFILE ,5,3
EXEPANDA_DQ2PUT_FILECOPYERROR,6,3
EXEPANDA_GET_GLOBUSSYSTEMERROR,7,2
TRF_SEGFAULT ,8,1

Victoria:

name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,44
EXEPANDA_DQ2_STAGEIN,2,19
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,3,12
TRF_OUTFILE ,4,5
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,5,3
EXEPANDA_DQ2GET_INFILE,6,3
EXEPANDA_JOBKILL_SIGTERM,7,2
TRF_SEGFAULT ,8,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,9,1

You can find the meanings of the transform error codes on this link.

Notes

• How we fared to other ATLAS sites ...

  cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
  BNL	135426	4411	3884629108	33559666	96.8%	99.1%
  None	55774	12548	943495702	90540589	81.6%	91.2%
  FZK	43251	1120	252950662	14488264	97.5%	94.6%
  CNAF	24503	5486	518594509	44749387	81.7%	92.1%
  LYON	21051	6123	396704716	78859684	77.5%	83.4%
  ASGC	17659	2147	487247496	23383433	89.2%	95.4%
  TRIUMF	13776	443	296294210	43203240	96.9%	87.3%
  SARA	9577	673	186556008	32827621	93.4%	85%
  RAL	3197	599	89964363	24788543	84.2%	78.4%
  NDGF	3395	367	82104480	15745176	90.2%	83.9%
  PIC	2905	114	56350772	2451319	96.2%	95.8%
  CERN	215	1154	3590844	454856	15.7%	88.8%
  total	330729	35185	7.19848287e+09	405051778	90.4%	94.7%

• From ADCos (Youri's summary):
  • - active tasks from PandaDB: 232
  • -- by cloud:
  • CA:13 DE:5 ES:8 FR:26 IT:8 ND:16 NL:25 TW:3 UK:95 US:33

• Issues:
  • Toronto : SE issues ( 16 - 19 Oct)
  • Victoria : SE issues (16 -19 Oct)

-- BryanCaron - 19 Oct 2009