TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100105 (2010-01-05, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-01-05

General News

* ADC News [20100105]

* information system reminders [20100105]

  • sites will be contacted in the next week if their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

* CA Cloud Squad Actions [20100105]

* ROC_Canada news [20100105]

  • shifters
    • each site should provide the names of at least one volunteer to perform shifts so as to share the load
    • shifts are relatively light, during business hours, and involves checking a dashboard and making sure tickets are addressed in a timely way by the sites involved
    • please contact Di, Reda and Mike regarding shifters volunteering from your site
  • sites are reminded to submit to Di the SLA between their site and ROC_Canada before the start of next week

* Squid deployment [20100105]

  • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
  • currently status as of this morning (WARN is ok)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • SFU-LCG2 OK (for both CEs; Victoria squid is the backup but not online)
    • TORONTO-LCG2 WARN (no local squid; primary backup points to ALBERTA squid but fails due to network routing issue + squid offline; secondary backup to SFU OK)
    • TRIUMF-LCG2 WARN (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

  • apel-status-ca-20100105.png:
    apel-status-ca-20100105.png

  • apel-status-aus-20100105.png:
    apel-status-aus-20100105.png

APEL Accounting Results

  • apel-statistics-ca-20100105.png:
    apel-statistics-ca-20100105.png

  • apel-statistics-aus-20100105.png:
    apel-statistics-aus-20100105.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009 and currently in state of unscheduled downtime in GOCDB
      • site storage migrated to CA-ALBERTA-WESTGRID-T2 site (except for 5.5 TB which are hosted on a server showing memory errors)
      • site will be transitioned to Tier-3 functionality, upgrade to SL5 and return of SRM storage functions

  • CA-ALBERTA-WESTGRID-T2 site installation status [20091215]
      • updated setup to multi-homed pools to separate internal and external traffic (instructions from Sergey)
      • site validated for production and analysis activity 20091222 and marked online in Panda 20091223
      • savannah bug filed requesting site be added to Tier-2 availability and reliability display pages
      • items to be followed-up:
        • APEL accounting information publishing
        • SL5 WNs and additional CE
        • need to update to dCache golden release cycle from currently running 1.9.3-3

  • site currently quiet for production with jobs listed as assigned but not running [20100105]
    • initial investigation appears to be a combination of the job requirements for a release not yet installed at the site, along with failing installation of this particular release
      • to be followed-up

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • identified one chassis of nodes where IB performance was degraded
    • nodes in chassis marked offline for repair by vendor
  • new allocations for ATLAS fairshare between the bugaboo and snowpatch clusters implemented and checked prior to christmas break

Outstanding Issues

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • apel publishing results for September and October 2009 [20091110]
      • low values published but not expected when comparing with ATLAS production dashboard for the same period
      • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • will likely switch to using the TRIUMF top-level BDII from the CERN-based BDII in order to help resolve observed timeouts

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently on unscheduled downtime in GOCDB

Outstanding Issues

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • additional hardware arrived from vendor for T1
  • pnfs database installed and cached locally on each WN to help resolve SRM overload issues previously observed
    • significantly reduces load on pnfs server
  • need to investigate in cooperation with panda the implementation of queues for short jobs (such as merging) in order to minimize the number of concurrently running jobs of this particular type

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

Production

Note: This is for a 3 week period (Dec 16 2009 - Jan 5 2010) since the last production report.

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 118352 4515 1410216149 40419324 96.3% 97.2%
SFU-LCG2 41555 1297 717934776 29833960 97% 96%
Australia-ATLAS 22964 244 393800856 1039141 98.9% 99.7%
VICTORIA-LCG2 9579 1134 354032285 5317337 89.4% 98.5%
CA-ALBERTA-WESTGRID-T2 8881 44 87280818 56942 99.5% 99.9%
total 201331 7234 2.963264884e+09 76666704 96.5% 97.5%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

(work in progress to retrieve this information - will post when available.)

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,2319
EXEPANDA_DQ2_STAGEIN,2,815
EXEPANDA_DQ2GET_INFILE,3,730
EXEPANDA_DQ2PUT_FILECOPYERROR,4,401
EXEPANDA_JOBKILL_BYPILOT,5,110
EXEPANDA_GET_ADLER32MISMATCH,6,82
TRF_UNKNOWN ,7,21
EXEPANDA_UNKNOWN_CATCHALL,8,12
TRFERROR,9,10
EXEPANDA_JOBDISPATCHER_HEARTBEAT,10,6
EXEPANDA_JOBKILL_BYSERVER,11,5
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,12,3
TRF_SEGFAULT ,13,1


Alberta:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,34
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,2,6
TRFERROR,3,3
EXEPANDA_UNKNOWN_CATCHALL,4,1


Australia:

name,rank,value
EXEPANDA_GET_REPLICANOTFOUND,1,209
TRFERROR,2,10
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,8
EXEPANDA_JOBKILL_BYPILOT,4,5
EXEPANDA_PUT_LFCMODULEIMPORT,5,4
EXEPANDA_DQ2PUT_FILECOPYERROR,6,3
TRF_UNKNOWN ,7,2
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,8,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,9,1


SFU:

name,rank,value
EXEPANDA_GET_REPLICANOTFOUND,1,378
TRF_SVRINIT ,2,375
EXEPANDA_DQ2PUT_FILECOPYERROR,3,283
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,117
EXEPANDA_JOBKILL_BYPILOT,5,69
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,25
TRFERROR,7,15
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,8,10
EXEPANDA_JOBKILL_NOLOCALSPACE,9,8
EXEPANDA_DQ2_STAGEIN,10,4
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,11,4
TRF_UNKNOWN ,12,3
EXEPANDA_GET_SIZEMISMATCH,13,2
EXEPANDA_PUT_GLOBUSSYSTEMERROR,14,2
EXEPANDA_PUT_LFCMODULEIMPORT,15,1
EXEPANDA_DQ2GET_CANTCALL,16,1


Victoria:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,588
EXEPANDA_JOBEXPIRED_SIXDAYS,2,270
EXEPANDA_DQ2PUT_FILECOPYERROR,3,179
EXEPANDA_GET_REPLICANOTFOUND,4,81
EXEPANDA_DQ2GET_INFILE,5,5
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,2
EXEPANDA_JOBKILL_BYPILOT,7,2
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,8,2
TRF_SEGFAULT ,9,2
EXEPANDA_JOBKILL_SIGTERM,10,1
TRFERROR,11,1
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,12,1

Notes

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
BNL 772537 39096 14181815501 1793970888 95.2% 88.8%
RAL 498881 25640 12489455187 727315655 95.1% 94.5%
PIC 264828 48987 3940695504 357026142 84.4% 91.7%
FZK 260676 37816 6075083595 855426522 87.3% 87.7%
SARA 243474 35137 4515476051 448085540 87.4% 91%
NDGF 260763 5144 4515923567 56000284 98.1% 98.8%
LYON 211038 9242 4090354002 234471879 95.8% 94.6%
TRIUMF 201331 7234 2963264884 76666704 96.5% 97.5%
None 148177 51077 2874106938 218121292 74.4% 92.9%
CNAF 134552 11012 3640307167 321161290 92.4% 91.9%
ASGC 46461 1638 1482813255 42516091 96.6% 97.2%
CERN 7762 1521 37459091 40804194 83.6% 47.9%
total 3050480 273544 6.0806754742e+10 5.171566481e+09 91.8% 92.2%

  • ADCoS (Yuri's notes): (unavailable)

-- BryanCaron - 05 Jan 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 117.1 K 2010-01-05 - 19:28 AsokaDeSilva  
PDFpdf PandaUsageStats.pdf r1 manage 356.4 K 2010-01-05 - 19:28 AsokaDeSilva  
PNGpng apel-statistics-aus-20100105.png r1 manage 791.0 K 2010-01-05 - 22:33 BryanCaron  
PNGpng apel-statistics-ca-20100105.png r2 r1 manage 1074.6 K 2010-01-05 - 22:32 BryanCaron  
PNGpng apel-status-aus-20100105.png r1 manage 15.1 K 2010-01-05 - 22:34 BryanCaron  
PNGpng apel-status-ca-20100105.png r1 manage 21.9 K 2010-01-05 - 22:34 BryanCaron  
PDFpdf ops-t2-alberta-20100105.pdf r1 manage 102.3 K 2010-01-05 - 22:08 BryanCaron  
PDFpdf ops-t2-australia-20100105.pdf r1 manage 91.7 K 2010-01-05 - 22:08 BryanCaron  
PDFpdf ops-t2-scinet-20100105.pdf r1 manage 97.6 K 2010-01-05 - 22:09 BryanCaron  
PDFpdf ops-t2-sfu-20100105.pdf r1 manage 89.5 K 2010-01-05 - 22:09 BryanCaron  
PDFpdf ops-t2-summary-20100105.pdf r1 manage 52.4 K 2010-01-05 - 20:24 BryanCaron  
PDFpdf ops-t2-victoria-20100105.pdf r1 manage 96.3 K 2010-01-05 - 22:09 BryanCaron  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2010-01-05 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback