Tier1 and Tier-2 Weekly Report 2010-02-16

General News

• ADC News [20100216]
  • Past Week
    • ADC Weekly Meeting of 20100210 - cancelled
    • ATLAS Tier-1/2/3 Jamboree @ CERN: February 9, 2010
      • http://indico.cern.ch/conferenceDisplay.py?confId=76900
        • Highlights:
          • CREAM CE Status report - not ready ready for production and sites need to at least have an lcg-CE for now (both is possible so as to not rely on debugging of CREAM CE integration)
          • ATLAS Site Validation, blacklisting and site status dashboard update - ATLAS Grid Information System, Site Status Dashboard
          • Panda Server and Pilot Identity switching update
            • risks exist if not all sites adopt the use of identity switching as one site that doesn't can be a weak-link

• WLCG GDB (Grid Deployment Board) [20100210]: agenda and minutes

• site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100216]
  • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status

• CA Cloud Squad Actions [20100216]
  • submitted request to ADCoS shifter to test and mark CA-SCINET-T2 production queue online [20100211]
    • marked online [20100212]
  • requested Ganga remove manual site blacklist for CA-SCINET-T2 [20100211]
  • deletion of 4 files from user dataset on TRIUMF_LOCALGROUPDISK [20100211] http://savannah.cern.ch/bugs/?62750
  • update to FTS settings for CA-SCINET-T2 channel from TRIUMF to help recover from backlog of subscriptions to the site [20100212]
    • current link from CA-SCINET-T2 to TRIUMF is via regular routed academic network and not dedicated 1 Gbps lightpath
  • preparing updated maxinputsize parameter settings for CA cloud sites in Panda schedconfig [20100213]
    • increase from default 14 GB size for sites with sufficiently large WN local disk space that can be properly identified within Panda schedconfig
  • Ongoing Discussion:
    • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
      • issues with regards to blocking of jobs to ce1 or ce2 when one CE set to test
        • see open ticket http://savannah.cern.ch/bugs/?61918
  • Actions to Do:
    • update T2 site shares for MCDISK once it is confirmed that the additional 22 TB of space is online at VICTORIA-LCG2
    • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
      • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
    • site information sanity checks
      • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

• ROC_Canada news [20100216]
  • shift schedule drafted last week: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ROCShifts
  • brief report from current week ROC Canada shifter (A. Agarwal)

• Squid deployment [20100216]
  • tested in SAM for ATLAS https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning (WARN is ok)
  • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
  • CA-SCINET-T2: Error (Frontier setup now installed but error connecting to either ALBERTA or SFU Squid cache)
    • ACTION ITEM: SFU and ALBERTA to confirm configuration of Squid ACLs to accept connections from CA-SCINET-T2
  • SFU-LCG2 OK (for both CEs but test for SL4 last ran 20100201; local is OK; backup squid at Victoria ok)
  • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
  • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
  • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql (old interface)
  • new sites CA-ALBERTA-WESTGRID-T2 and CA-SCINET-T2 will not be added https://savannah.cern.ch/bugs/?60346
  • new info to be displayed via programmatic XML interface (coming soon)

APEL Accounting

• general APEL server issues where results are not being propagated to SAM nor the accounting portal [20100128]
  • https://cic.gridops.org/index.php?section=cod&page=broadcast_archive&step=2&typeb=C&idbroadcast=44967

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=54707 CA-ALBERTA-WESTGRID-T2 apel publishing (20100115) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53828 ALBERTA-LCG2 lcg-CE out-of-date (20091203) [in progress]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• ALBERTA-LCG2 site
  • site offline to ATLAS since early December 2009
  • currently in state of scheduled downtime in GOCDB until 20100222

• CA-ALBERTA-WESTGRID-T2 site status [20100216]
  • open items to be followed-up:
    • APEL accounting information publishing (open ticket on hold for site follow-up) ggus 54707
    • SL5 WNs and additional CE [20100216]
      • test system installed for ATLAS verification. Needed set of RPMs are installed and software release being tested with kit validation
    • need to update to dCache golden release cycle from currently running 1.9.3-3
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

• squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
  • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
  • Action Item: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

• pending release installations for 15.6.3.x on bugaboo (20100202)
  • installation files not replicated to SE
  • pending jobs killed and resubmitted by Joe - status?

• identified one chassis of nodes where IB performance was degraded [20100105]
  • nodes in chassis marked offline for repair by vendor - status?

• apel publishing [20090825] (on hold for confirmation pending resolution of central APEL publishing problems)
  • apel publishing information missing for October (see above plots) [20091013]
  • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
  • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported
  • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• added 22 TB of additional storage to VICTORIA-LCG2 site dCache [20100209]
  • once it is confirmed to have been added to MCDISK token area then Bryan will request DDM Ops to update share to the site

Outstanding Issues

• apel accounting / publishing issues - /var/lib on local server (shared with SE head node) became full (20100128)
  • update 20100209 - cleaned APEL databases

• SCRATCHDISK became full [20091112]
  • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=54764 CA-SCINET-T2 APEL publishing problem (20100118) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53830 TORONTO-LCG2 update to lcg-CE (20091203) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53605 TORONTO-LCG2 lcg_cp timeout (20091124) [on hold]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• TORONTO-LCG2 site
  • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

• CA-SCINET-T2
  • undergoing ATLAS configuration and validation (software release installation, gcc installation on WN, ...)
  • complete set of required software releases installed including PoolCondPFC (See https://atlas-install.roma1.infn.it/atlas_install/list.php?sitename=CA-SCINET-T2)
    • list of required releases https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ProductionSoftwareReleases
    • SAM Frontier Tests currently failing (see above)
  • site production queue marked for test by ADCoS shifter [20100211] and brought online [20100212]
  • removed from manual Ganga blacklist [20100211]

• unscheduled downtime registered in GOCDB for 20100209
  • reconfiguration of hostnames and grid-user home space on CE

Outstanding Issues

• slow execution (> 2 hours) of build jobs compared to other sites (approx. 10 mins)
  • software installation area still under GPFS.
  • GPFS/NFS server node being built but requires testing.
  • also to test NFS server using DDN LUNs directly.

• Squid/Frontier access tests via ReadReal [20091028]
  • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
  • added secondary backup of SFU squid into ToA
  • Action Item to SFU and ALBERTA: Confirm Squid ACLs to accept CA-SCINET-T2 network

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55528 TRIUMF FTS server cannot be contacted. [20100214] (verified)
    • FTS process hung as Oracle RAC encountered disk errors requiring restart and memory usage adjustment

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• scheduled downtime underway from 20100215 to 20100217
  • migration of dCache from pnfs to chimera

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55337 request for installation of nightly releases at site (20100208) [assigned]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=55188 unreachable biomed SE (20100202) [in progress]

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• at risk scheduled downtime (6 hours) for upstream network provider upgrade of connection to 10 Gbps [20100213]

• identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault
  • causing issues with regards to release 15.5.1 testing. A modified setup for pilot users added to profile.d to get around this in the interim
  • bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=563759

Outstanding Issues

McGill

Recent Issues

• status?

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	29036	1238	510990424	10675985	95.9%	98%
CA-ALBERTA-WESTGRID-T2	10575	42	233375508	80817	99.6%	100%
SFU-LCG2	7303	93	160352789	260562	98.7%	99.8%
VICTORIA-LCG2	2579	59	116154922	208541	97.8%	99.8%
Australia-ATLAS	2166	105	44653186	8526884	95.4%	84%
total	51659	1537	1065526829	19752789	97.1%	98.2%

DashboardSummary.pdf

(SCINET seems to be missing in the above ATLAS Dashboard - will contact developers about this.)

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:
name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,1056
EXEPANDA_DQ2GET_INFILE,2,58
TRFERROR,3,54
EXEPANDA_GET_NOSUCHFILE,4,17
EXEPANDA_DQ2_STAGEIN,5,17
EXEPANDA_JOBKILL_SIGTERM,6,12
TRF_UNKNOWN ,7,9
EXEPANDA_JOBKILL_NOLOCALSPACE,8,5
TRF_PYT_SYNTAX ,9,3
EXEPANDA_DQ2PUT_FILECOPYERROR,10,3
TRF_SEGFAULT ,11,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,12,1
EXEPANDA_PILOTEXCEPTION,13,1

Alberta:
name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,19
TRFERROR,2,17
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,3
EXEPANDA_GET_SIZEMISMATCH,4,2
EXEPANDA_UNKNOWNERROR_JOBWRAPPERCRASH,5,1

Australia:
name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,62
EXEPANDA_DQ2PUT_FILECOPYERROR,2,19
TRFERROR,3,13
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,6
EXEPANDA_GET_GLOBUSSYSTEMERROR,5,4
EXEPANDA_JOBKILL_SIGTERM,6,1

SFU:
name,rank,value
TRFERROR,1,65
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,2,9
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,7
EXEPANDA_GET_SIZEMISMATCH,4,4
EXEPANDA_PUT_LFCMODULEIMPORT,5,3
EXEPANDA_JOBKILL_SIGTERM,6,3
EXEPANDA_PUT_GLOBUSSYSTEMERROR,7,1
EXEPANDA_DQ2PUT_FILECOPYERROR,8,1

Toronto:

Victoria:
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,43
TRFERROR,2,11
EXEPANDA_JOBKILL_NOLOCALSPACE,3,3
EXEPANDA_DQ2GET_INFILE,4,2

• How we fared to other ATLAS sites ...

cloud	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
BNL	303171	31866	3775640500	188565449	90.5%	95.2%
LYON	236575	36771	4368524744	151154651	86.5%	96.7%
FZK	212714	10809	3229005087	63981341	95.2%	98.1%
SARA	137699	20921	1649334908	161911395	86.8%	91.1%
RAL	121771	19294	2148081971	194331654	86.3%	91.7%
None	99671	12494	1742633072	169481811	88.9%	91.1%
NDGF	106826	1054	2312842920	5988057	99%	99.7%
PIC	57906	3345	817560860	22833864	94.5%	97.3%
TRIUMF	51659	1537	1065526829	19752789	97.1%	98.2%
CNAF	29102	3673	711282964	48813119	88.8%	93.6%
ASGC	13102	3054	430939984	17652160	81.1%	96.1%
CERN	78	155	245992	125771	33.5%	66.2%
total	1370274	144973	2.2251619831e+10	1044592061	90.4%	95.5%

• My Notes:
  • TRIUMF - scheduled outage on Mon (15 Feb)
  • SCINET: 12 Feb: ANALY queue online; prod queue online later that day after testing by ADCoS.
  • Australia: 15 Feb: Analy queue online after fix.
  • SFU bugaboo : what is the release status for 15.6.3.x ?

• Yuri's notes:
  • active tasks from PandaDB: 198
  • distribution by clouds:
    • CA:6 CERN:1 DE:3 ES:1 FR:26 IT:7 ND:5 NL:17 TW:2 UK:17 US:113
  • Sun Feb.14
    • + CA/TRIUMF FTS down/Failed polling FTS request.GGUS 55528 verified:3D Oracle RAC was shutdown due to disk err.,memory issues(swapping) resulting FTS'hung'.Lowered OracleDB mem.7->6GB,restarted FTS DB.

• Stats:
  • PandaStats.pdf
  • See Key for what the headers mean

-- BryanCaron - 15 Feb 2010