TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100302 (2010-03-02, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-03-02

General News

  • ADC News [20100223]
    • Past Week: ADC Weekly Meeting [20100224] http://indico.cern.ch/conferenceDisplay.py?confId=86137
      • ADC Monitoring to revive various dashboards (prodsys, task/task monitoring)
      • February reprocessing summary - follow-up discussion about job share distribution to be done in separate meeting
      • pilot changed (done and planned) - pilot space requirements for WNs, dataset DQ2 tracing, Tier-3 support

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100302]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2
      • SFU-LCG2 (snowpatch)
      • VICTORIA-LCG2

  • CA Cloud Squad Actions [20100302]
    • marked CA-SCINET-T2 analysis and production queues offline following report of storage access problems [20100227]
      • restored analysis queues to online (cloud contact) [20100302] and production queue (by ADCoS shifter) [20100302]
    • marked TRIUMF-LCG2 analysis and production queues offline [20100301] and restored to online less than 3 hours later
    • Ongoing Discussion:
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • Squid deployment [20100302]
    • tested in SAM for ATLAS https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: Error (Frontier setup now installed but error connecting to either ALBERTA or SFU Squid cache)
      • ACTION ITEM: cloud contacts to request ToA update for site info to reference local Squid cache as primary once local client tests pass
      • ACTION ITEM: SFU and ALBERTA to confirm configuration of Squid ACLs to accept backup connections from CA-SCINET-T2
    • SFU-LCG2 OK (for both CEs but test for SL4 last ran 20100201; local is OK; backup squid at Victoria ok)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local Asquid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

APEL Publishing Status

  • apel-status-aus-20100302.png:
    apel-status-aus-20100302.png

  • apel-status-ca-20100302.png:
    apel-status-ca-20100302.png

APEL Accounting Results

  • apel-statistics-aus-20100302.png:
    apel-statistics-aus-20100302.png

  • apel-statistics-ca-20100302.png:
    apel-statistics-ca-20100302.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100315

  • CA-ALBERTA-WESTGRID-T2 site status [20100302]
    • scheduled downtime booked into GOCDB for main campus computing centre power upgrades [20100319]
      • network cabling maintenance planned for lighpath connection during this time
    • open items being followed-up:
      • APEL accounting information publishing (open ticket in progress with new email to central apel support) ggus 54707
        • APEL server updated to accept site information; now debugging local mysql access errors for APEL service
      • SL5 WNs and additional CE [20100216]
        • still verifying test configuration of SL54 x86_64 with ATLAS kits which are showing athena crash errors (investigating)
      • torque qsub wrapper to be activated this week to remove email notification errors associated with grid jobs coming from CE to torque/moab
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • ACTION ITEM: Please confirm ACL setup on site squid to accept backup connections from CA-SCINET-T2

Outstanding Issues

  • APEL accounting information publishing (see above)
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • Action Item: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ACTION ITEM: Please confirm ACL setup on site squid to accept backup connections from CA-SCINET-T2

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • CA-SCINET-T2
    • DDN storage problems [20100226]
      • associated with failed drawers and various pools running in degraded mode
      • atlas software area unavailable to nodes so cloud contact marked production and analysis queues offline in panda and downtime booked in GRIDOPS
      • data transfers via FTS unaffected and so left active within DDM
      • restored access sufficient for running without tempoary software area (?)
        • queues marked online following validation [20100302]
    • installed local Squid server which is undergoing commissioning and validation through the usual client tests [20100302]

Outstanding Issues

  • slow execution (> 2 hours) of build jobs compared to other sites (approx. 10 mins) - Is this still an issue?? Needs confirmation
    • software installation area was originally under GPFS.
    • GPFS/NFS server node built and moved into production.

  • SAM Frontier Tests currently failing (see above and next item)

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA
    • ACTION ITEM to SFU and ALBERTA: Confirm Squid ACLs to accept CA-SCINET-T2 network

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 38137 4602 625310464 21378278 89.2% 96.7%
CA-ALBERTA-WESTGRID-T2 13604 26 279965733 1553971 99.8% 99.4%
SFU-LCG2 2876 3 54826104 125100 99.9% 99.8%
Australia-ATLAS 2413 104 44328467 1132723 95.9% 97.5%
VICTORIA-LCG2 762 22 37256517 133854 97.2% 99.6%
total 57792 4757 1041687285 24323926 92.4% 97.7%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:
name,rank,value
TRFERROR,1,3622
EXEPANDA_DQ2_STAGEIN,2,642
EXEPANDA_JOBKILL_BYPILOT,3,99
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,4,59
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,54
EXEPANDA_JOBKILL_SIGTERM,6,43
EXEPANDA_Release_MISSING,7,39
EXEPANDA_JOBEXPIRED_SIXDAYS,8,21
TRF_UNKNOWN ,9,7
EXEPANDA_DQ2_FILEADD,10,6
EXEPANDA_DQ2PUT_FILECOPYERROR,11,5
EXEPANDA_DQ2GET_INFILE,12,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,13,2
EXEPANDA_JOBKILL_BYSERVER,14,1

Alberta:
name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,7
TRF_UNKNOWN ,2,5
TRFERROR,3,5
EXEPANDA_GET_SIZEMISMATCH,4,4
EXEPANDA_DQ2_FILEADD,5,4
EXEPANDA_DQ2PUT_FILECOPYERROR,6,1

Australia:
name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,39
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,35
EXEPANDA_DQ2PUT_FILECOPYERROR,3,11
TRFERROR,4,7
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,5,6
TRF_UNKNOWN ,6,3
EXEPANDA_DQ2_FILEADD,7,2
EXEPANDA_JOBKILL_BYPILOT,8,1

SCINET:
Missing

SFU:
name,rank,value
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,1,2
EXEPANDA_JOBKILL_BYPILOT,2,1

Victoria:
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,21
EXEPANDA_JOBKILL_BYPILOT,2,1

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
SARA 89800 14154 2558042873 130672642 86.4% 95.1%
NDGF 89363 6742 2203380649 47338180 93% 97.9%
BNL 60236 5742 727824376 90731668 91.3% 88.9%
TRIUMF 57792 4757 1041687285 24323926 92.4% 97.7%
CNAF 21409 6111 762991017 60225027 77.8% 92.7%
None 17136 3327 334502211 24402137 83.7% 93.2%
RAL 15551 2860 142414714 18748316 84.5% 88.4%
ASGC 13730 2522 425380452 63677911 84.5% 87%
FZK 6033 879 116130360 11159421 87.3% 91.2%
CERN 4161 548 76075983 2374956 88.4% 97%
LYON 659 858 8810742 21847063 43.4% 28.7%
PIC 361 59 1622871 22621 86% 98.6%
total 376231 48559 8.398863533e+09 495523868 88.6% 94.4%

  • Yuri's notes:
    • active tasks from PandaDB: 49
    • distribution by cloud:
      • CA:2 CERN:3 DE:6 ES:1 FR:3 IT:3 ND:5 NL:6 TW:2 UK:9 US:9
    • Wed Feb.24 * + CA/TRIUMF repro t. f./missing 15.6.4 inst.GGUS 55896 solved:15.6.4 was not deployed in the grid until Feb.25, because it was not requested. Installed in T1s,except for Taiwan(still installing)..
    • Thu Feb.25
      • - CA/TRIUMF 3K job f./15.6.3.9 setup missing.GGUS 55938 in progress: fails only on ce1 (and works on ce2)

-- BryanCaron - 02 Mar 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 103.9 K 2010-03-02 - 20:00 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 113.7 K 2010-03-02 - 20:00 AsokaDeSilva  
PDFpdf PandaStats.pdf r1 manage 267.6 K 2010-03-02 - 20:01 AsokaDeSilva  
PNGpng apel-statistics-aus-20100302.png r1 manage 851.3 K 2010-03-02 - 20:46 BryanCaron  
PNGpng apel-statistics-ca-20100302.png r1 manage 1156.5 K 2010-03-02 - 20:46 BryanCaron  
PNGpng apel-status-aus-20100302.png r1 manage 15.2 K 2010-03-02 - 20:46 BryanCaron  
PNGpng apel-status-ca-20100302.png r1 manage 22.2 K 2010-03-02 - 20:47 BryanCaron  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2010-03-02 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback