TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091110 (2009-11-10, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2009-11-10

General News

  • new MC AOD dataset distribution policy discussed within ADC and ICB so as to take into account operational constraints (among others) [20091110]
    • 10 copies of latest datasets available across Grid
      • as new dataset versions are available, numbers of custodial copies reduced to 2 (for version N-1) and 0 (for version N-2)
      • when a site is > 80% full:
        • select least used and old data
        • mark and publish non-custodial data to be deleted
        • delete 1 week later until site is < 60% full
    • T2s are for user analysis but should be considered if:
      • at least 100 TBs
      • pass HammerCloud validation criteria for efficiency and throughput: 90% efficiency, 150M events per day throughput at the site
      • should meet reliability criteria
    • removing sites not meeting above criteria reduces overall space availability within ATLAS by approx. 10%
    • ability will be there for clouds to decide within their respective federation in general what the endpoint locations will be

  • Squid deployment [20091110]
    • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning: all sites good (WARN is ok).
      • ALBERTA-LCG2 WARN (local squid server offline until this AM)
      • SFU-LCG2 WARN (for both CEs; Victoria squid is the backup but not online)
      • TORONTO-LCG2 WARN (local squid not online; primary backup points to ALBERTA squid but fails due to network routing issue; secondary backup to SFU OK)
      • TRIUMF-LCG2 WARN (for both CEs; local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 WARN (local squid not online; backup points to SFU squid)
    • ReadReal test passes at all sites except SFU and TRIUMF which at the time of the problem the only sites labelled with dcache_std in the ToA
    • PoolCondPFC file structure (problematic extra strings) issues for bugaboo CE and TRIUMF ce2 being followed-up with Asoka, Alessandro, Rod, D Q2 developers, etc. for the implementation of short-term and long-term needed fixes.

  • central DDM Ops dataset deletion [20091110]
    • approx. 25% of all tid datasets from spring mc08 re-reconstruction (r6*_t53) lost in DDM accident
    • redoing reconstruction of samples to be performed only upon specific requests by groups
    • https://savannah.cern.ch/bugs/index.php?58355

  • announced vulnerability to sites via EGEE broadcast * please take indicated steps as soon as possible

  • dCache REMINDERS

    • golden release of dCache announced: version 1.9.5 (see dCache user-forum announcement)
      • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement)
      • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
        • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • WestGrid-Alberta: all nodes at 1.9.3-3
        • TRIUMF: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
        • VICTORIA: 1.8.0.12p5
        • ALBERTA: 1.8.0.12p6

  • REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

  • production job issues
    • 20091108
      • dccp timeout errors - coincident with unscheduled downtime of site SE services

  • analysis job issues
    • 20091109
      • dccp timeout errors - concident with end of unscheduled downtime of site SE services

  • scheduled downtime for CA-ALBERTA-WESTGRID-T2 on 20091108 for several hours during computing centre power maintenance
    • added unscheduled downtime to ALBERTA-LCG2 during same period as some dCache pools and Squid server are hosted on WestGrid
    • services restored late in the evening of 20091108

  • installation status: WLCG on WestGrid-Alberta cluster [20091110]
    • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
      • some job submission errors found with no results returned; "cannot read job wrapper output" and "abort request expired" being investigated
        • CERN ROC indicated these errors likely to their listing of the site in the GOCDB as 'candidate' instead of 'uncertified'.
        • CERN ROC now [20091110] new 3 day period of SAM tests.
        • need to modify dCache configuration to have multihomed pools to separate internal and external traffic; instructions kindly provided by Sergey

Outstanding Issues

  • few minor issues pending correction on existing T2 facility:
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • production job issues
    • to be updated

  • analysis job issues
    • to be updated

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site

  • SL5 (bugaboo) queue marked in 'test' within Panda pending follow-up post scheduled downtime
    • additional separate analysis queue (ANALY_SFU_bugaboo) requested of Panda team and created

Outstanding Issues

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • production job issues
    • to be entered

  • analysis job issues
    • to be entered

  • site MCDISK area temporarily increased in size to accomodate a single small UAT dataset [20091027]
    • dataset transfer of UAT DS101-medcut requested of DDM Ops
    • slow rate of transfer observed (10 MBps) with one parallel transfer and 1 stream configured within FTS
      • increased to 3 parallel files and 3 streams each, with improved throughput and no SE problems observed [20091101]

Outstanding Issues

Toronto

Recent Issues

  • production job issues
    • to be entered

  • analysis job issues
    • to be entered

  • CA-SCINET-T2 installation:
    • new site certification by CERN ROC: https://gus.fzk.de/ws/ticket_info.php?ticket=49823
    • dCache 1.9.4 now on pools but finding some configuration issues
    • dCache admin node being updated to CentOS 53
    • DDN firmware update debugging from vendor during downtime
    • designated node for squid cache installation

Outstanding Issues

  • cleanup of suspended transfers from 20091031 still pending on SE

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • to be entered

  • analysis job issues
    • to be entered

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

  • attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
    • pilots (without payload) successfully running
    • next target is successful SAM tests
    • SE functionality still to be tested on current dCache instance with two pools

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 8492 522 216945595 6736723 94.2% 97%
VICTORIA-LCG2 1233 208 56272315 3912282 85.6% 93.5%
SFU-LCG2 1222 182 39349103 3556924 87% 91.7%
ALBERTA-LCG2 927 240 27785552 3367981 79.4% 89.2%
TORONTO-LCG2 815 101 52714162 3360524 89% 94%
total 12689 1253 393066727 20934434 91% 94.9%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
TRFERROR,1,157
TRF_SEGFAULT ,2,89
TRF_UNKNOWN ,3,73
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,4,62
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,5,50
EXEPANDA_JOBKILL_SIGTERM,6,32
EXEPANDA_DQ2PUT_FILECOPYERROR,7,30
EXEPANDA_JOBDISPATCHER_HEARTBEAT,8,17
EXEPANDA_JOBKILL_BYPILOT,9,12


Alberta:

name,rank,value
EXEPANDA_DQ2GET_INFILE,1,52
TRF_UNKNOWN ,2,51
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,3,41
TRFERROR,4,39
TRF_SEGFAULT ,5,30
EXEPANDA_GET_REPLICANOTFOUND,6,18
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,8
EXEPANDA_JOBKILL_BYPILOT,8,1


SFU:

name,rank,value
TRFERROR,1,61
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,55
TRF_UNKNOWN ,3,27
TRF_SEGFAULT ,4,20
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,5,9
EXEPANDA_JOBKILL_BYPILOT,6,4
EXEPANDA_DQ2PUT_FILECOPYERROR,7,2
EXEPANDA_JOBKILL_SIGTERM,8,1
TRF_SVRINIT ,9,1
TRF_OUTFILE ,10,1
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,11,1


Toronto:

name,rank,value
TRF_SEGFAULT ,1,22
TRF_UNKNOWN ,2,20
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,19
TRFERROR,4,18
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,5,10
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,6,10
EXEPANDA_JOBKILL_BYPILOT,7,2


Victoria:

name,rank,value
TRFERROR,1,61
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,29
EXEPANDA_GET_REPLICANOTFOUND,3,28
TRF_UNKNOWN ,4,23
TRF_SEGFAULT ,5,21
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,6,16
EXEPANDA_DQ2_STAGEIN,7,14
EXEPANDA_DQ2PUT_FILECOPYERROR,8,13
EXEPANDA_JOBKILL_BYPILOT,9,1
EXEPANDA_DQ2GET_INFILE,10,1
EXEPANDA_PUT_GLOBUSSYSTEMERROR,11,1

Notes

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
BNL 171173 12111 2795269316 96372163 93.4% 96.7%
FZK 148578 5808 3475097925 73761294 96.2% 97.9%
None 114737 9256 2589255896 131146066 92.5% 95.2%
LYON 87966 5194 1935370827 91338527 94.4% 95.5%
RAL 62377 6268 2049156016 104414311 90.9% 95.2%
NDGF 56495 6168 1607111533 182143494 90.2% 89.8%
SARA 39329 9762 804435343 102644696 80.1% 88.7%
CNAF 22664 1932 870631816 30036354 92.1% 96.7%
TRIUMF 12449 1253 381368143 20934434 90.9% 94.8%
PIC 6432 1714 179095553 40182148 79% 81.7%
CERN 2079 1874 47312918 35806060 52.6% 56.9%
ASGC 434 266 8659365 2024252 62% 81.1%
total 724713 61606 1.6742764651e+10 910803799 92.2% 94.8%

  • Yuri's notes:
    • - active tasks from PandaDB: 113
      • -- by cloud: CA:8 DE:7 ES:9 FR:15 IT:6 ND:9 NL:10 TW:9 UK:27 US:13
    • Thu Nov.5.
      • + CA/VICTORIA PRODDISK FT f./REQUEST_TIMEOUT.GGUS 53003 solved:dCache server/pools restarted.No errors in past 12 hours.
    • Fri Nov.6.
      • + CA/SFU job f./lfc_getreplicas f./could not load a sec.plugin.GGUS 53045 in progress:a new pilot factory issue.Reverted to the old one and developer notified.Works again.
    • Mon Nov.9.
      • + CA/ALBERTA 64 job f./DBrel. dccp timeout.GGUS 53097 in progress: UD, 4 of 12 dCache pools were unavailable,now dccp on the file works.
  • Notes:
    • TRIUMF drained queues (8-9 Nov)
    • SFU bugaboo: new pilot factory problem (6 Nov). Still using old factory for this CE.
    • SFU bugaboo - missing releases 15.3.1.7 and 15.3.1.8 (fixed)
    • jobs still going to T2 where data is not there.

-- BryanCaron - 03 Nov 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 95.4 K 2009-11-10 - 18:30 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 100.9 K 2009-11-10 - 18:30 AsokaDeSilva  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2009-11-10 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback