TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100119 (2010-01-19, AsokaDeSilva) EditAttach

Tier1 and Tier-2 Weekly Report 2010-01-19 [In Progress]

General News

* ADC News [20100112]

* information system reminders [20100112]

  • sites will be contacted in the next week if their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

* CA Cloud Squad Actions [20100112]

  • none in the past week

* ROC_Canada news [20100112]

* Squid deployment [20100105]

  • tested in SAM for ATLAS https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
  • currently status as of this morning (WARN is ok)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: Error (Frontier setup not yet defined as per standard ATLAS software installation procedures)
    • SFU-LCG2 WARN (for both CEs; local is OK; Victoria squid is the backup but not online)
    • TORONTO-LCG2 WARN (no local squid; primary backup points to ALBERTA squid but fails due to network routing issue + squid offline; secondary backup to SFU OK)
    • TRIUMF-LCG2 WARN (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

OPS VO

  • Tier-2 Summary (All Sites)

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009 and currently in state of unscheduled downtime in GOCDB
      • site storage migrated to CA-ALBERTA-WESTGRID-T2 site (except for 5.5 TB which are hosted on a server showing memory errors)
      • site will be transitioned to Tier-3 functionality, upgrade to SL5 and return of SRM storage functions

  • CA-ALBERTA-WESTGRID-T2 site installation status [20100111]
    • network fibre cut between Calgary and Edmonton affecting WestGrid network access to cluster as well as end-to-end lightpath to TRIUMF at approx. 09:30 (20100111)
      • brief period of 2 hours where internal network misconfiguration prevented proper communication between CE and local Torque-Moab server
      • automatic failover of lightpath to research network WAN observed with return to lightpath just after midnight (see attached)
        • UAlberta_lcgce01_last_108000-20100112.png:
          UAlberta_lcgce01_last_108000-20100112.png

    • update to Moab fairshare values for ATLAS activities based on NRAC allocations made [20100111]
      • prior to this [20100108] various groups of jobs were not running due to the incorrect early-cluster commissioning fairshare settings for ATLAS being picked-up
      • open items to be followed-up:
        • APEL accounting information publishing
        • SL5 WNs and additional CE
        • need to update to dCache golden release cycle from currently running 1.9.3-3

  • following up on some ATLAS software installation problems following network hostname resolution changes for internal network prior to Christmas
    • first noticed due to zero level of production activity in the previous reporting week with many jobs assigned but not running, waiting for input files [20100105]

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • apel publishing results for September and October 2009 [20091110]
      • low values published but not expected when comparing with ATLAS production dashboard for the same period
      • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • will likely switch to using the TRIUMF top-level BDII from the CERN-based BDII in order to help resolve observed timeouts [20100105]

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently on unscheduled downtime in GOCDB

Outstanding Issues

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 18892 623 366235332 14900768 96.8% 96.1%
SFU-LCG2 7808 17 262798615 454320 99.8% 99.8%
CA-ALBERTA-WESTGRID-T2 5550 164 193665843 30302 97.1% 100%
VICTORIA-LCG2 1620 190 112642424 187962 89.5% 99.8%
total 33870 994 935342214 15573352 97.1% 98.4%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:
name,rank,value
TRF_UNKNOWN ,1,311
EXEPANDA_JOBEXPIRED_SIXDAYS,2,187
EXEPANDA_JOBKILL_BYPILOT,3,80
EXEPANDA_DQ2_STAGEIN,4,16
EXEPANDA_JOBKILL_SIGTERM,5,10
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,7
EXEPANDA_JOBKILL_NOLOCALSPACE,7,4
EXEPANDA_RUNJOBEXCEPTION,8,3
TRFERROR,9,2
TRF_SEGFAULT ,10,1
EXEPANDA_DQ2PUT_FILECOPYERROR,11,1
EXEPANDA_DQ2_FILEADD,12,1

Alberta:
name,rank,value
EXEPANDA_DQ2_STAGEIN,1,164

SFU:
name,rank,value
EXEPANDA_PUT_GLOBUSSYSTEMERROR,1,7
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,6
EXEPANDA_DQ2PUT_FILECOPYERROR,3,4

Victoria:
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,158
EXEPANDA_DQ2_STAGEIN,2,28
TRF_SEGFAULT ,3,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,2

Notes

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
BNL 168666 15566 4831978444 129305494 91.6% 97.4%
RAL 87283 5631 2992706255 40618781 93.9% 98.7%
NDGF 87060 5528 1060847089 39392846 94% 96.4%
LYON 56512 5866 2020488256 214949396 90.6% 90.4%
None 46948 13693 1478638000 48130960 77.4% 96.8%
PIC 35437 1447 745315918 33311767 96.1% 95.7%
TRIUMF 33870 994 935342214 15573352 97.1% 98.4%
ASGC 16427 6573 502107324 81862903 71.4% 86%
CNAF 17778 3721 636664719 62904493 82.7% 91%
FZK 8007 119 266432372 3496501 98.5% 98.7%
SARA 3408 2582 122907395 15127895 56.9% 89%
CERN 550 180 1895314 183729 75.3% 91.2%
total 561946 61900 1.55953233e+10 684858117 90.1% 95.8%
  • My Notes:
    • Australia offline.

  • ADCoS (Yuri's notes):
    • active tasks from PandaDB: 80
      • by cloud:
      • CA:5 DE:2 ES:3 FR:3 IT:4 ND:43 NL:2 TW:3 UK:4 US:11
    • Tue Jan.12
      • + CA/ALBERTA all jobs f./stage-in/lcg-cp timeout.Set offline.GGUS 5460 solved:via SRM restart.Test jobs succedeed,set online on Wed.
    • Thu Jan.14
      • + CA/ALBERTA still job f./stage-in.Notified site admin.Elog 8622.Fixed.
  • Panda Stats:

-- BryanCaron - 19 Jan 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 102.8 K 2010-01-19 - 18:32 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 86.9 K 2010-01-19 - 18:32 AsokaDeSilva  
PDFpdf PandaStats.pdf r1 manage 223.6 K 2010-01-19 - 18:32 AsokaDeSilva  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2010-01-19 - AsokaDeSilva
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback