Tier1 and Tier-2 Weekly Report 2010-02-02

General News

• ADC News [20100202]
  • Past Week
    • no ADC Weekly Meeting of 20100127
    • 2009 Software and Computing Postmortem Workshop @ CERN : January 18-19 and 26 2010
      • http://indico.cern.ch/conferenceDisplay.py?confId=76898 (Part 1)
      • http://indico.cern.ch/conferenceDisplay.py?confId=80197 (Part 2)
    • ATLAS Tier-3 Workshop @ CERN : January 25-26, 2010
      • http://indico.cern.ch/conferenceDisplay.py?confId=77057
  • Next Week: ATLAS Tier-1/2/3 Jamboree @ CERN: February 9, 2010
    • http://indico.cern.ch/conferenceDisplay.py?confId=76900

• site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100202]
  • each site currently running with SL4 WNs should please provide an updated timeline for the transition of their WNs to SL5

• information system reminders [20100202]
  • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

• CA Cloud Squad Actions [20100202]
  • none this week
  • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • issues with regards to blocking of jobs to ce1 or ce2 when one CE set to test
      • see http://savannah.cern.ch/bugs/?61918
  • To Do: follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
    • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

• Cloud Operations Reminder [20100126]
  • sites should please distribute all requests for panda queue modification to CA cloud contacts (Bryan, Asoka, Di, Reda) via usual operations list
  • aim is have a consistent process by which site status changes and readiness is confirmed prior to being brought back into full production mode
    • cloud contacts will then:
      • for the case of ANALY queues directly set queue status as needed to test/offline/online with feedback or request for SiteTests to be run by the site contact
      • request to ADCoS shifter for the site production queue status to be changed to 'test' mode and have test jobs submitted and validated
        • if 'test' job succeeds then the ADCoS shifter should change status to 'online'.

• Security News [20100126]
  • SL5 kernel update for security fixes - see http://listserv.fnal.gov/scripts/wa.exe?A2=ind1001&L=scientific-linux-errata&T=0&P=2192
    • where are the SL5 sites with regards to this? Please indicate during site report go-around.

• ROC_Canada news [20100126]
  • ROC node offline for a few days to complete the installation of new hardware [20100126]

• Squid deployment [20100202]
  • tested in SAM for ATLAS https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning (WARN is ok)
  • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
  • CA-SCINET-T2: Error (Frontier setup not yet defined as per standard ATLAS software installation procedures)
  • SFU-LCG2 OK (for both CEs but test for SL4 last ran 20100201; local is OK; backup squid at Victoria ok)
  • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
  • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
  • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql (old interface)
  • new sites CA-ALBERTA-WESTGRID-T2 and CA-SCINET-T2 will not be added https://savannah.cern.ch/bugs/?60346

*

• • new info to be displayed via programmatic XML interface (coming soon)

APEL Accounting

• general APEL server issues where results are not being propagated to SAM nor the accounting portal [20100128]
  • https://cic.gridops.org/index.php?section=cod&page=broadcast_archive&step=2&typeb=C&idbroadcast=44967

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=54707 CA-ALBERTA-WESTGRID-T2 apel publishing (20100115) [in progress]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53828 ALBERTA-LCG2 lcg-CE out-of-date (20091203) [in progress]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• ALBERTA-LCG2 site
  • site offline to ATLAS since early December 2009
  • currently in state of scheduled downtime in GOCDB until 20100222

• CA-ALBERTA-WESTGRID-T2 site installation status [20100202]
  • open items to be followed-up:
    • APEL accounting information publishing (open ticket for site follow-up) ggus 54707
    • SL5 WNs and additional CE
    • need to update to dCache golden release cycle from currently running 1.9.3-3

Outstanding Issues

• squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
  • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• upgrade to Moab on snowpatch cluster (SL4) to fix a few bugs (20100202?)
  • Moab dying periodically with segfaults ; being monitored and will rollback if needed

Outstanding Issues

• identified one chassis of nodes where IB performance was degraded [20100105]
  • nodes in chassis marked offline for repair by vendor - status?

• apel publishing [20090825]
  • apel publishing information missing for October (see above plots) [20091013]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
  • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported
  • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

• discrepancy between SRM and DQ2 views of MCDISK occupancy (observed during STEP09) [closed but a reminder]
  • SFU-LCG2_MCDISK cleaned by DDM Ops and helped resolve the discrepancy between DQ2 and SRM usage statistics (20091217)
    • also occurred at some other sites and not just SFU
      • site assistance to debug and resolve important, otherwise drastic DDM involvment needed (ie. cleanup, blacklisting)
      • important for the future that sites provide requested assistance to investigate SE consistency issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• apel accounting / publishing issues - /var/lib on local server (shared with SE head node) became full (20100128)

Outstanding Issues

• SCRATCHDISK became full [20091112]
  • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=54764 CA-SCINET-T2 APEL publishing problem (20100118) [in progress]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53830 TORONTO-LCG2 update to lcg-CE (20091203) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53605 TORONTO-LCG2 lcg_cp timeout (20091124) [in progress]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• TORONTO-LCG2 site
  • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

• CA-SCINET-T2
  • SiteTests package updated for CA-SCINET-T2 (Asoka) [20100201]
  • undergoing ATLAS configuration and validation (software release installation, gcc installation on WN, ...)
  • software releases (See https://atlas-install.roma1.infn.it/atlas_install/list.php?sitename=CA-SCINET-T2)
    • releases installed:
      • 14.2.25, 14.2.25.4, 14.2.25.6, 14.2.25.7, 14.2.25.8
      • 15.3.1, 15.3.1.8, 15.3.1.9, 15.3.1.10, 15.3.1.11, 15.3.1.12, 15.3.1.13
      • 15.5.0, 15.5.1, 15.5.2, 15.5.4, 15.5.4.10-t0, 15.5.5
      • 15.6.0, 15.6.0.3, 15.6.1, 15.6.1.5, 15.6.1.6, 15.6.1.7
      • DQ2clients 0.1.28, 0.1.29, 0.1.31, 0.1.32
      • gcc-x86_64:4.3.2
    • problem releases:
      • failed:
        • 14.2.25.1, 14.2.25.2, 14.2.25.5
        • 15.3.1.2, 15.3.1.20, 15.3.1.4, 15.3.1.5, 15.3.1.6, 15.3.1.7
        • 15.6.0.1, 15.6.0.2, 15.6.1.2, 15.6.1.3, 15.6.3
      • un-tagged:
        • 14.2.25.3, 15.3.1.1, 15.3.1.3, 15.6.1.4
    • Missing: PoolCondPFC
    • slow execution (> 2 hours) of build jobs compared to other sites (approx. 10 mins)
      • software installation area still under GPFS.
      • GPFS/NFS server node being built but requires testing.
      • also to test NFS server using DDN LUNs directly.

Outstanding Issues

• Squid/Frontier access tests via ReadReal [20091028]
  • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
  • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55089 TRIUMF LFC does not accept voms proxy from vo.racf.bnl.gov (20100128) [solved]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55026 Transfer failures between TRIUMF and TW - FTS agent restart (20100127) [solved]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=54994 job CMTCONFIG error (non-site issue) (20100126) [solved]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55188 unreachable biomed SE (20100202) [assigned]

• site production queue (Australia-ATLAS-agh2-atlas-lcgpbs) set to offline (20100128) and online (20100129)
  • site migration to PBS Version 2.3.9

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

• PATH clobbering issue preventing ANALY_AUSTRALIA from being marked online (2009-12-16)

McGill

Recent Issues

• status?

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	17578	588	493390438	785466	96.8%	99.8%
SFU-LCG2	5216	12	161980107	100338	99.8%	99.9%
CA-ALBERTA-WESTGRID-T2	5202	19	183463502	452595	99.6%	99.8%
Australia-ATLAS	2687	102	85913084	4431692	96.3%	95.1%
VICTORIA-LCG2	1410	274	109468197	817218	83.7%	99.3%
total	32093	995	1034215328	6587309	97%	99.4%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:
name,rank,value
EXEPANDA_JOBKILL_BYEXTIF,1,144
EXEPANDA_JOBEXPIRED_SIXDAYS,2,128
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,3,124
EXEPANDA_CMTCONFIGMISMATCH,4,61
TRF_UNKNOWN ,5,50
TRF_OUTFILE_CORRUP ,6,19
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,17
EXEPANDA_DQ2_STAGEIN,8,15
EXEPANDA_JOBKILL_BYSERVER,9,9
EXEPANDA_DQ2GET_INFILE,10,6
TRFERROR,11,6
EXEPANDA_DQ2PUT_FILECOPYERROR,12,5
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,13,3
EXEPANDA_RUNJOBEXCEPTION,14,1

Australia:
name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,66
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,17
EXEPANDA_GET_GLOBUSSYSTEMERROR,3,10
EXEPANDA_JOBKILL_BYPILOT,4,8
TRF_UNKNOWN ,5,1

Alberta:
name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,16
EXEPANDA_GET_SIZEMISMATCH,2,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,3,1

SFU:
name,rank,value
EXEPANDA_GET_REPLICANOTFOUND,1,5
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,2,3
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,2
TRFERROR,4,1
EXEPANDA_PUT_GLOBUSSYSTEMERROR,5,1

Victoria:
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,261
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,5
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,4,1
EXEPANDA_DQ2GET_INFILE,5,1

Notes

• How we fared to other ATLAS sites ...

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
LYON	145854	39396	1968587277	100079786	78.7%	95.2%
FZK	68620	4713	1410268510	51372417	93.6%	96.5%
NDGF	70249	1540	1576242841	30081248	97.9%	98.1%
SARA	53710	5362	1656208556	38203063	90.9%	97.7%
PIC	40888	4764	625373489	47479337	89.6%	92.9%
BNL	36165	5132	126768795	79484704	87.6%	61.5%
None	27177	6564	313359709	140352252	80.5%	69.1%
TRIUMF	32093	995	1034215328	6587309	97%	99.4%
CNAF	20271	1284	699891913	13821932	94%	98.1%
RAL	3516	11431	165200161	520490288	23.5%	24.1%
ASGC	13009	816	400009676	28574871	94.1%	93.3%
CERN	592	745	4017139	1771909	44.3%	69.4%
total	512144	82742	9.980143394e+09	1058299116	86.1%	90.4%

• My Notes:
  • Low efficiency at Victoria because of expired jobs.
• ADCoS (Yuri's notes):
  • active tasks by Panda-monitor: 90
    • by cloud: CA:4 ES:1 FR:9 IT:2 ND:6 NL:3 TW:1 UK:8 US:56
  • Tue Jan 26.
    • + CA/TRIUMF >100 job f./Trf couldn't be installed.Savannah 61918:se2 inst. of 15.6.3.4 done.Problems with dynamic inst.fixed in the new pilot.
• Panda Stats:
  • PandaStats.pdf
  • See Key for what the headers mean

-- BryanCaron - 02 Feb 2010