TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091215 (2009-12-15, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2009-12-15

General News

* ADC News [20091215]

* procedural reminder for sites in regards to scheduled/unscheduled downtimes and panda queues [20091215]

  • all requests for site downtime and restart with regards to panda queues (analysis and production) should go through cloud contacts (Asoka, Bryan, Di)
  • cloud contacts with then contact or notify ADC operations in regards to actions taken for panda queues
  • more detailed procedural info to be circulated

* availability of ADC Cloud support during Christmas period (Dec 18 - Jan 3)

  • in general email contact for response to ADC issues over holiday period
  • Bryan, Di, Reda, Joe (Asoka away December 18 to January 3)
  • Tier-2 sites should clarify today their availability and coverage plans
    • Toronto/SciNet
    • Alberta/WestGrid
    • SFU
    • Victoria
    • TRIUMF
    • Australia

* security announcement reminders [20091215]

  • SL5 acpid vulnerability announced to sites via (limited) EGEE broadcast [20091208]
  • critical and important java security errata announced for SL [20091116]

* information system reminders [20091215]

  • sites will be contacted in the next week if their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

* CA Cloud Squad Actions [20091208]

  • corrupt file on TRIUMF-LCG2_MCDISK [20091201]
  • mc08*merge.AOD* dataset subscription for input to GangaRobot submissions to CA-ALBERTA-LCG2 [20091214]

* Squid deployment [20091208]

  • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
  • currently status as of this morning (WARN is ok)
    • ALBERTA-LCG2 WARN (local squid server offline since 20091108)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • SFU-LCG2 OK (for both CEs; Victoria squid is the backup but not online)
    • TORONTO-LCG2 WARN (no local squid; primary backup points to ALBERTA squid but fails due to network routing issue + squid offline; secondary backup to SFU OK)
    • TRIUMF-LCG2 WARN (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

* update to golden release of dCache announced last week (1.9.5-10) (see dCache user-forum announcement)

  • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement from 20091011)
  • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)
    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • ALBERTA-LCG2: 1.8.0.12p6
        • CA-ALBERTA-WESTGRID-T2: all nodes at 1.9.3-3
        • CA-SCINET-T2: mixture of nodes at 1.9.5-4 and 1.9.5-9
        • SFU-LCG2: all nodes at 1.9.2-8 and client 1.9.2-4
        • TORONTO-LCG2: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • TRIUMF-LCG2: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
      • VICTORIA-LCG2: 1.8.0.12p5

SAM Results

OPS VO

  • ca-ops-availability-t2s-20091215.png:
    ca-ops-availability-t2s-20091215.png

  • ca-ops-reliability-t2s-20091215.png:
    ca-ops-reliability-t2s-20091215.png

ATLAS VO

  • ca-atlas-availability-t2s-20091215.png:
    ca-atlas-availability-t2s-20091215.png

  • ca-atlas-reliability-t2s-20091215.png:
    ca-atlas-reliability-t2s-20091215.png

APEL Accounting

APEL Publishing Status

  • apel-ca-status-20091215.png:
    apel-ca-status-20091215.png

  • apel-aus-status-20091215.png:
    apel-aus-status-20091215.png

APEL Accounting Results

  • apel-statistics-ca-20091215.png:
    apel-statistics-ca-20091215.png

  • apel-statistics-aus-20091215.png:
    apel-statistics-aus-20091215.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS
    • downsizing of site dCache completed [20091214]
      • single occurrence of SAM errors for SE actions prior to removal of ATLAS space tokens [20091214]

  • CA-ALBERTA-WESTGRID-T2 site installation status [20091215]
    • infrastructure setting and debugging
      • grid certificate CRLs correctly updating; CRL update info missing from .r0 files is normal for latest release of fetch-crl
      • intermittent job submission (JS) errors from SAM tests found to be due to misconfigured lcg-voms.cern.ch.lsc coming from yaim configuration step (20091203)
      • savannah ticket open in regards to migrating current ALBERTA-LCG2 site storage to CA-ALBERTA-WESTGRID-T2 site
        • migration complete with all storage now part of CA-ALBERTA-WESTGRID-T2 and LFC cleaned of ALBERTA-LCG2 entries
      • update to host resolution to dCache pools from WNs to correct failing gridftp copies observed during ATLAS software installations [20091211]
      • added 3 of the 4 x 5.5 TB pools from ALBERTA-LCG2 site [20091213]
      • actions on hold until LHC winter technical stop
        • update to dCache 1.9.5-10 (golden release)
        • update setup to multi-homed pools to separate internal and external traffic (instructions from Sergey)
      • savannah bug filed requesting site be added to Tier-2 availability and reliability display pages
    • ATLAS grid commissioning
      • DDM integration complete
        • missing entry in data distribution dashboard (action on DDM ops)
      • ATLAS software installation proceeding and almost complete following gridftp and host name resolution fixes
      • site production and analysis queues created in panda to be tested when release installs confirmed

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • few minor issues pending correction on existing T2 facility (ALBERTA-LCG2):
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • dCache downtime of approx. 10 mins for restart (20091209)
    • correct dCache headnode configuration which resulted in all transfers going through 2 NAT servers

  • status of bugaboo (SL5) commissioning status within atlas
    • ANALY_bugaboo and production queues set online (20091211)

  • issues seen with atlas software installation (14.2.25) area reference
    • checked on all WNs; under further investigation

  • SFU-LCG2_MCDISK being cleaned by DDM Ops to help resolve discrepancy between DQ2 and SRM usage statistics (20091215)

Outstanding Issues

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • scheduled downtime for CE maintenance and upgrade [20091209]
    • downtime entered into GOCDB but not published via site downtime subscriptions (either atlas operations or atlas canada tier2 list)
    • site panda queues (analy_ and production) set to online 20091210

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently on unscheduled downtime in GOCDB until 20091216

Outstanding Issues

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • dccp get timeout errors reported for analysis jobs (20091209)

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • unscheduled downtime due to disk server issues (20091209)
    • resulted in transfer errors arising at TRIUMF dCache as FTS may have cancelled transfers when errors occurred
    • resolved and transfers confirmed successful (20091214)

  • ANALY_ queue set to test status (20091210)
    • test jobs fails to build; base release 15.5.1 reinstalled but error still present ; under investigation
    • GangaRobot already running successfully at site but still blacklisted (why?)

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 20269 4735 566201092 47376363 81.1% 92.3%
SFU-LCG2 6108 119 197776106 233263 98.1% 99.9%
Australia-ATLAS 3727 317 123783308 8690500 92.2% 93.4%
VICTORIA-LCG2 572 67 38545526 25574 89.5% 99.9%
total 30676 5238 926306032 56325700 85.4% 94.3%
DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,1805
EXEPANDA_DQ2PUT_FILECOPYERROR,2,1194
EXEPANDA_JOBKILL_BYSERVER,3,813
EXEPANDA_JOBKILL,4,617
EXEPANDA_DQ2GET_INFILE,5,269
EXEPANDA_DQ2_STAGEIN,6,15
TRF_SEGFAULT ,7,6
TRFERROR,8,5
EXEPANDA_PUT_GLOBUSSYSTEMERROR,9,4
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,10,3
EXEPANDA_RUNJOBEXCEPTION,11,2
EXEPANDA_UNKNOWN_CATCHALL,12,2
EXEPANDA_JOBKILL_BYPILOT,13,1


Australia:

name,rank,value
EXEPANDA_JOBKILL_BYSERVER,1,147
EXEPANDA_DQ2PUT_FILECOPYERROR,2,120
EXEPANDA_JOBKILL_BYPILOT,3,19
EXEPANDA_DQ2_STAGEIN,4,12
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,9
EXEPANDA_GET_GLOBUSSYSTEMERROR,6,8
EXEPANDA_JOBKILL_SIGTERM,7,1
TRFERROR,8,1


SFU:

name,rank,value
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,1,74
EXEPANDA_JOBKILL_BYSERVER,2,15
EXEPANDA_DQ2_STAGEIN,3,10
TRF_SEGFAULT ,4,8
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,6
TRF_OUTFILE ,6,4
EXEPANDA_DQ2PUT_FILECOPYERROR,7,1
TRFERROR,8,1


Victoria:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,63
TRF_SEGFAULT ,2,4

Notes

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
LYON 246035 7542 5621814943 140424695 97% 97.6%
BNL 205254 14167 4839671983 179706409 93.5% 96.4%
FZK 182516 2947 4418315830 57439761 98.4% 98.7%
RAL 159384 5250 4500906475 130196312 96.8% 97.2%
None 77272 17599 1851535561 43058175 81.4% 97.7%
SARA 81804 5991 2235510763 153844738 93.2% 93.6%
PIC 85336 1543 1670301590 26395862 98.2% 98.4%
NDGF 70549 1862 1806925297 32447762 97.4% 98.2%
TRIUMF 30676 5238 926306032 56325700 85.4% 94.3%
CNAF 26388 1372 995966847 27792502 95.1% 97.3%
ASGC 24285 469 771257752 17569894 98.1% 97.8%
CERN 2075 1484 22432246 4737959 58.3% 82.6%
total 1191574 65464 2.9660945319e+10 869939769 94.8% 97.2%

  • My Notes:
    • Victoria - online Wed
    • bugaboo - online Sat
    • TRIUMF: SE issues (dcache timeouts etc, known.)
    • Australia - SE issues (Wed, Sun)
  • ADCoS (Yuri's notes):
    • active tasks from PandaDB: 117
      • by cloud: CA:6 CERN:2 DE:6 ES:5 FR:5 IT:5 ND:7 NL:3 TW:2 UK:7 US:25
    • Wed Dec.9.
      • + CA/TRIUMF/VICTORIA-LCG2 >600 job f./expired after 7 days.Looks like these jobs were lost due to CERN power cut a week ago.Elog 7866,7867
    • Thu Dec.10.
      • - CA/TRIUMF 700 job f./lcg-cr failed.GGUS 53997 in progress:perhaps due to update of CRL on WNs(with new fetch-crl no update history).
      • + CA/VICTORIA-LCG2/ANALY_VICTORIA set online.Elog 7890.
    • Fri Dec.11.
      • + CA/SFU test jobs succeeded, site set online for prod.Elog 7929,7955.

-- BryanCaron - 15 Dec 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 103.3 K 2009-12-15 - 19:45 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 74.8 K 2009-12-15 - 19:45 AsokaDeSilva  
PDFpdf PandaStats.pdf r1 manage 208.2 K 2009-12-15 - 19:45 AsokaDeSilva  
PNGpng apel-aus-status-20091215.png r1 manage 15.0 K 2009-12-15 - 21:54 BryanCaron  
PNGpng apel-ca-status-20091215.png r1 manage 21.6 K 2009-12-15 - 21:54 BryanCaron  
PNGpng apel-statistics-aus-20091215.png r1 manage 80.9 K 2009-12-15 - 21:53 BryanCaron  
PNGpng apel-statistics-ca-20091215.png r1 manage 132.7 K 2009-12-15 - 21:53 BryanCaron  
PNGpng ca-atlas-availability-t2s-20091215.png r1 manage 153.1 K 2009-12-15 - 21:52 BryanCaron  
PNGpng ca-atlas-reliability-t2s-20091215.png r1 manage 98.0 K 2009-12-15 - 21:52 BryanCaron  
PNGpng ca-ops-availability-t2s-20091215.png r1 manage 152.8 K 2009-12-15 - 21:51 BryanCaron  
PNGpng ca-ops-reliability-t2s-20091215.png r1 manage 99.5 K 2009-12-15 - 21:51 BryanCaron  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2009-12-15 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback