TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130305 (2013-03-05, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-03-05

General News

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Feb.26-Mar.4,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Feb.26-Mar.4,2013) Panda production service
      - completed successfully 1,448,446 managed group, MC production,
        validation and reprocessing jobs
      - average ~206,921 jobs per day
      - failed 116,373 jobs
      - average efficiency:
         -- jobs ~92.6%
      - active tasks: 924
         -- distribution by cloud:
CA:84 CERN:24 DE:170 ES:86 FR:164 IT:46 ND:66 NL:58 RU:4 TW:63 UK:250
US:486

B)   Major Downtimes:

     - IT/INFN-T1 DT (ce01 OUTAGE:Upgrade to EMI2,SL6), Feb.21-28.
     - IT/INFN-T1 DT (ce04 OUTAGE:Upgrade to EMI2,SL6), Mar.1-7.
     - IT/INFN-T1 DT (SE,SRM OUTAGE:space token merging), Mar.4-5.
     - NL/SARA-MATRIX DT(CE,SE,SRM OUTAGE:network+WN upgrade),Feb.25-27.
     - TW/TAIWAN-LCG2 DT(CE,SE,SRM OUTAGE:castor,dpm upgrade),Feb.25-26.
     - UK/RAL-LCG2 DT(CE,SE,SRM,LFC warning:2 short con.breaks),Feb.26.
     - NL/NIKHEF DT(SE,SRM warning:disk server defect),Feb.28-Mar.1.
     - CA/TRIUMF-LCG2 DT (CE,SE,SRM,BDII OUTAGE:s.power outage),Mar.2-3.

     - US/BNL LFC T2 DB transparent intervention (patches), Feb.28.
     - CERN two rolling interventions on INTR,INT8R DB, Feb.28.
     - CERN/ATLAS PanDA shedconfig->AGIS migration, Feb.28.
     - CA/TRIUMF DB:Cond.,TAG,FTS,Frontier,ELSSI DT:power outage,Mar.2-3

     - New ATLAS DDM Dashboard 2.0 M4.5 upgrade released, Feb.28.
     - New ATLAS release TRF caches were distributed:
       -- AtlasOffline    -
       -- AtlasProduction 16.6.70.4 (64b), 17.2.8.11 (32+64b)
       -- AtlasPhysics    17.2.7.5.8 (32+64b)
       -- AtlasMCProd     17.2.1.4.4 (32+64b)
       -- AtlasProd1      -
       -- AtlasTrigMC     -

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Feb.26
      -> Cloud/site PROD efficiency ever<50% : UK/UKI-LT2-BRUNEL
      -> Cloud/site DDM  efficiency <50% : TW
        ........................................
+ IT/INFN-BOLOGNA-T3 450|4h FT f./DEST.:HTTP_TIMEOUT:SRM|REQUEST_TIMEOUT
  GGUS 97779 solved:restarted all services on SE the same day,fixed.
+ FR/IN2P3-LPC many job f./Get errors.GGUS 91781 solved:many timeouts on
  rfio transfers job log file(rfio get),but nothing in DPM logfiles.
  Maybe a temporary network problem,no new f.,Mar.1.
- NL/RRC-KI many FT f./AGENT:CONFIG._ER.:Channel closed for your VO or
  VO not authorized for transferring between RRC-KI and CERN-PROD.GGUS
  91786 on hold:errors come from RRC-KI-T1?Due to the absence of the
  relevant FTS channel in CERN's FTS service:GGUS 91755.
+ CA/TRIUMF Frontier down in SLS.GGUS 91791 solved: there was a
  network outage,the network is back.Efficiency 100%,Feb.27.
+ CA/TRIUMF-LCG2 still FT f./ SOURCE:CONN.:SRM.GGUS 91766 solved:
  network outage,fixed,the network is back the same day.
+ CA/CA-VICTORIA-WESTGRID-T2 job f./staging input err.GGUS 91748 solved:
  :also FT f./DEST.:CONN.:SRM - perhaps related to GGUS 91748 for TRIUMF
  fixed.No recent stag-in errors,Feb.27.
+ UK/UKI-SCOTGRID-GLASGOW<-FZK still 150|2h FT f./GRIDFTP_ER.:server
  err.426.GGUS 91439 solved:there was a problem in the routing at FZK
  that affected many UK T2.Fixed,transfers good,no recent failures on
  Feb.28.
- UK/UKI-LT2-RHUL still ~900 FT f./DEST.:SRM_ABORTED.GGUS 91711 in
  progress updated:~150 job f./transfer timeout.Set PRODDISK offline for
  fwu because of srm problems.And ANALY_RHUL is offline.Savannah 136186:
  other queues offlined due to evgen job f.,Mar.4.

2) Wed Feb.27
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : RU
        ................................................................
+ CERN/ATLAS_DDM_VOBOXES degraded,experts informed.Fixed Elog.43115,7,8.
+ CERN/ATLAS-T0-Frontier degraded.GGUS 91793 solved:hyperv machines on
  CVICLR04FC h/w disk failure,machines went into read-only.Fixed.
+ CERN/CERN-PROD t.1208763 1400 job f./Staging input file failed.
  Savannah 136167.Task ABORTED.
+ FR/IN2P3-CC FRONTIER-LDAP were offline for the update ~2h,completed,
  back online.Elog.41128,41131.
+ FR/IN2P3-CC-T2 ~70% job f./Put err.:put_data destination path in SE
  not defined.GGUS 91829 solved:multicloud for UK issue,ATLASPRODISK
  ddm endpoint added the same day.
+ UK/UKI-SOUTHGRID-RALPP 90% job f./lost heartbeat.GGUS 91827 solved:
  problem with Argus fixed.
- US/AGLT2 ~70% FT f./SOURCE:Locality is unavailable.GGUS 91835 assigned
  :also ~50% prod.job.f./staging.Network issues,fix didn't work.Still
  5331 f.,Feb.28.Blacklisted as source:Savannah 136180:restarted the
  problematic pools on dCache,whitelisted,Mar.2.GGUS 91896:continue to
  have host instabilities,watch how things behave with a different
  kernel,Mar.4.
+ ND/SE-SNIC-T2 job f./I|O error.GGUS 91831 verified: one WN affected;
  ti24.uppmax.uu.se,fixed,Feb.28.
+ CA/CA-SCINET-T2 3K FT f./SOURCE+DEST.err.Savannah 136176:all systems
  were taken down by a power glitch.Blacklisted in DDM.GGUS 91850
  solved:external Cisco WAN switch seems to have been damaged in the
  power glitch.No more errors,Mar.4.

3) Thu Feb.28
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50% : RU
        ............................................
+ CA/TRIUMF-LCG2 again 25% job f./killed by batch system.GGUS 91766
  reopened,solved:new f./pilot:No such file or dir.The issue resolved,
  production quite clean in last 24h.,Feb.26.
- US/BU_ATLAS_Tier2 still 930|4h deletion errors.GGUS 89339 on hold.
+ US/SWT2_CPB Xrootd system problems,offline.Blacklisted,DT announced:
  Savannah 136087:new errors after the DT and auto-whitelisting:DEST.:
  /xrd/datadisk...has trouble with canonical path. can't access it.
  Another restart of the xrootdfs process on the SRM host helped,Mar.1.
- NL/RRC-KI<->CERN still 400 FT f./SOURCE file doesn't exist:maybe
  something has been done about the FTS channel.GGUS 91786 on hold:460+
  f./AGENT er:Channel closed for your VO,Mar.3.Due to the absence of the
  relevant FTS channel in CERN's FTS service.GGUS 91755 in progress.
+ UK/UKI-NORTHGRID-LIV-HEP many job f./transfer timeout.GGUS 91865
  solved:no more such f.,Mar.1.
- UK/UKI-LT2-IC-HEP ~100 job f./cvmfs access on bad WNs.GGUS 91866
  waiting for reply:cleared the NFS caches for the CVMFS volume the same
  day.
+ DE/TUDREZDEN-ZIH still job f./Put:Err.reading token data header.GGUS
  91578 updated,solved:jobs succeeded(?).A slightly increasing load on
  the SRM node the last weeks,but this should not be critical.
+ DE/DEZY-HH ~800 job f./computingSite DESY-HH is unknown in schedconfig
  Savannah 136215:it was related to the AGIS switch,fixed,Mar.1.

4) Fri Mar.1
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : RU,CA/CA-SCINET-T2
        ............................................
- DE/CYFRONET-LCG2 FT f./SOURCE:GRIDFTP_ER.:end-of-file was reached.
  GGUS 91875 reopened:had some site-independent networking problems
  which are gone.Still all FT to/from CYFRONET f./SOURCE:Server side
  credential failure,Mar.4.
+ TW/Taiwan-LCG2 many job f./no such file or dir.Disk server f-dpmp30
  unavailable due to h/w failure, offlined and data can't be accessed
  from it.Rebuilding.Elog 43168-69,75,98.Fixed,Mar.3.
- US/AGLT2 many job f./Get err.:Input file staging timed out.
  Elog 43170.
+ US/NET2 many job f./SFN not set in LFC for guid.GGUS 91874 solved:
  also FT f./SOURCE:file doesn't exist/DEST.:CONN.:SRM.Network outage.
  Still the same f.,Mar.2.New power outage.Power and the route to the
  outside world was restored,Mar.3.DDM and transfer works.

5) Sat Mar.2
      -> Cloud/site PROD efficiency ever<50% : RU(brokeroff)
      -> Cloud/site DDM  efficiency <50% : RU
        ..............................
+ CA/TRIUMF-LCG2|CA cloud 1000s FT f./SOURCE:available CRL has expired.
  GGUS 91893 in progress:problems for nodes to connect the CA web site
  used by SARA,copied the CRL by hand.But still ~2K|4h same f.,Mar.3.
  Recopied those CRL files manually and restarted all services Mar.4,
  works again.
+ NL/SARA-MATRIX 200+ FT f./SOURCE:available CRL has expired.GGUS 91891
  solved:caused by TRIUMF,see above GGUS 91893.After CA cloud copied CRL
  files manually and restarted all services, things started to work
  again,Mar.4.
- NK/RRC-KI-T1->CERN_PROD 100+ FT f./AGENT err.:No Channel found.GGUS
  91889,91775,91786 in progress:a request to create a new channel in FTS
  since Feb.25.Sill 460+|4h same f.,Mar.3.Due to the absence of the
  relevant FTS channel in CERN's FTS service.
+ DE/PRAGUELCG2 80+ FT f./SOURCE:available CRL has expired.GGUS 91890
  verified:CRLs on the disk nodes look fine.The problem is not probably
  on the praguelcg2 side,Mar.3.

6) Sun Mar.3
      -> Cloud/site PROD efficiency ever<50% : RU (Brokeroff), ND, CA
      -> Cloud/site DDM  efficiency < 50% : RU, TW
        ........................................
- US/UPENN ~1K FT f./DEST.:CONN.:SRM.GGUS 91122 reopened,updated:
  bestman2 started failing authentications again,restarted.
- US/AGLT2 ~100 FT f./GRIDFTP_ER.:server err.451:Missing routing entry
  for msufs16_4.GGUS 91896 assigned:had some issues with particular
  dCache pools blocking on trying to CANCEL specific transfers that are
  not progressing.This causes issues with access to files on the
  affected pool.Had 6 large pools in this state.Testing a new dCache
  configuration to prevent.Still 1K+|4h FT f./SOURCE:locality is
  UNAVAILABLE,Mar.4.Continue to have host instabilities,watch how things
  behave with a different kernel,Mar.4.
- IT/INFN-MILANO_ATLASC many FT f./SOURCE:server err.550:Stale NFS file
  handle.GGUS 91378 in progress updated: almost all FT f./GRIDFTP_ER.:
  Unable to connect to gridftp-storm.mi.infn.it,Mar.4.Blacklisted in
  DDM:Savannah 136293.Put in place a new gridftp server to face transfer
  errors.
- NL/RU-PROTVINO-IHEP ATLASGROUPDISK 50TB full of dark data?Savannah
  100712:cloud squad informed.

7) Mon Mar.4
      -> Cloud/site PROD efficiency ever<50% : RU(brokeroff)
      -> Cloud/site DDM  efficiency <50%: -
        .......................................................
- FR/BEIGING-LCG2 70 FT f./SOURCE:GRIDFTP_ER.:server err.550:I/O err.
  GGUS 91906 assigned:have a disk array failed.Some of the data on this
  disk could not copy out.
- US/NET2 almost all FT f./DEST.:CONN.:SRM.GGUS 91874 reopened:problem
  on the BU side of NET2.SRM is off,the site team clearing things up.
  Problem resolved.FTS channels,SRM back up the same day.
- UK/RHUL evgen job f.Savannah 136186:Panda queues set offline.
- UK/UKI-NORTHGRID-LANCS-HEP many FT f./SOURCE:GRIDFTP_ER.:Unable to
  connect to stor015.hec.lancs.ac.uk/host credential...has expired.GGUS
  92067 assigned.
- DE/GOEGRID still same FT f./SOURCE:AsyncWait.GGUS 91995 reopened,
  updated.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.7.5.7 data12_8TeV Embedding reco (urgent Higgs group) task
 1213123 failures:ERROR LArRawChannelGetter unusable.Savannah 136251:
 group production Tau Embedding.All jobs have failed several attempts
 at least at 3 different sites running for NL cloud.Other affected tasks
 :1213183,1213124,1213101,1213115,1213113 failing in NL,ND,CA,US clouds.
 Tasks are still "submitting".Issue with empty/no events files is under
 investigation.

-- 17.2.7.5.7 data12_8TeV Embedding reco (urgent Higgs group) tasks
 1213187,213172,1213174,1213166,1213152 failures:TRF_SEGVIO -
 Segmentation violation.Savannah 136256:group production Tau Embedding.
 Many jobs have crashed at FZK/DE,input file issue?Tasks are still in
 "running" state.No comments published so far.

++ 17.2.8.8 mc12_8TeV evgen task 1208125-6 failures:Syntax error:EOL while scanning string literal.Savannah 136152:MC12 production - SUSY.
All jobs have failed 9 attempts at different sites in DE cloud.Task ABORTED.
The expert updated the jobOptions to (hopefully) fix the problem.

-+ 17.2.6.5 valid1 express_express Reco reprocessing task 1207941:all jobs failing with "Problems preLoading proxies".Savannah 100653:RU T1 
reprocessing test.All jobs have failed 11 attempts each at RU/RRC-KI-T1
Task ABORTED.No comments added.

++ 17.2.7.5.4 data12_8TeV Reco merge task 1209451 failures:object has no
 attribute 'BJetBuilderAntiKt4LCTopo'.Savannah 136179:Jobs 52-84 failed
 multiple attempts at different sites in US cloud.Expected.Run 204442 is
 special in that the ~second half is with the solenoid off. These jobs
 fail in the b-tagging,need to be canceled.Task set to "finished".

-+ 17.2.1.4 valid1 DigiMReco task 1211422-446,448-461 failures:object
 has no attribute 'AddressRemappingSvc'.Savannah 100665: ISF AFII
 VALIDATION 2012 SampleA s1639 r4378.All jobs have failed multiple
 attempts in various clouds.Tasks ABORTED,no comments added.

++  17.3.4.7 valid1 DigiMReco tasks 1212996-7 failures: AttributeError:
 '<class 'GaudiSvc.GaudiSvcConf.ToolSvc'>' object has no attribute
 'PixelClusterOnTrackTool'.Savannah 100677:T1_McAtNlo_Jimmy.All jobs
 have failed repeatedly at various sites running for CERN,UK clouds due
 to the errors on the ESDtoAOD step.Tasks ABORTED,new tag submitted.

-+ 17.2.7.5.2 mc12_8TeV Reco merge tasks 1212341-1212368 failures:Can't
 find object in event store for JetCollection(1162448536)/AntiKt4Topo
 EMJetsReTagged.Savannah 136204:group production EXOTICS.All jobs have
 failed multiple attempts in various clouds due to the FATAL error
 coming from JetVertexFractionAlg_AntiKt4TopoEMJetsReTagged on AODtoDPD
 step.Tasks ABORTED.Seems this NTUP_SMWZ doesnt' work for current
 dataset.

-+ 17.2.1.4.4 valid1 DigiMReco tasks 1215161-1215198 failures:Attribute
 Error:'<class 'AthenaCommon.AlgSequence.AlgSequence'>' object has no
 attribute 'InTimeAntiKt4TruthJetsFixIt'.Savannah 100709:VALIDATION 2012
 SampleA.All jobs have failed repeatedly at various clouds due to the
 error on ESDtoAOD step.Tasks ABORTED:config broken.

-- 17.2.8.10 mc12_8TeV evgen task 1215242-3 failures: Pythia  FATAL
 ../src/GenModule.cxx:204 (virtual StatusCode...Savannah 136274:MC12
 production - PHYSICS.Many jobs have failed up to 5-6 attempts each at
 various sites in US and DE clouds due to FATAL pythia error.Tasks are
 still in "running" or "submitting" state.No comments published so far.

-- LeslieGroer - 2013-03-05

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2013-03-05 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback