TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130226 (2013-02-26, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-02-26

General News

  • ADC and ATLAS Computing News [20130226]
    • ADC Weekly meeting
    • GDP
      • Rogue user filling up many slots on grid (10k's). Wasted >33k cpu-days due to walltime limits!
      • KillUser tool takes a while - UVic was affected
      • Lessons
        • Better monitoring to see wasteful usage
        • Default maxtime on submissions
        • maxtime set for all queues(then pilot will kill)
        • maybe enforce scout mechanism for large jobsets and only release bulk after some tests complete
      • SL6 WN deployment
        • SARA deploying today
        • Potential for missing system libs – is the sl6 meta-rpm for atlas dependencies existing, up to date?
        • RPMCompatSLC6 Twiki page
        • Production should be OK. Analy OK for patched releases(subset) and all new releases
        • Group production still needs to be signed off on - physics groups need to understand the urgency
        • Long discussion on release and cache mechanism for patches
        • Ad-hoc validation at the moment - assume SL6 + CVMFS is sufficient (could be missing libs, configs?)
      • Group production logs not being kept long enough - will be addressed
    • Group DPD Production
      • High priority - Top group production
        • Validation done Feb 9 (split jobs to reduce runtime per job)
        • period A-H submitted, I-M now. About 200k jobs running+activated
        • bulk of MC production (1.4 M events) will be requested after data is done
        • Top needs about 6.5M CPU-hours (4B events, 6 seconds/event)
      • Tau embedding production
        • Still validating. Want 3 outputs per long-running job
        • Estimate 1M cpu-hours
      • SL6 validation - some datasets available for comparison
    • MC Production Status
      • Most large requests done. Evgen going on but only targeted at some clouds and rely on MCP to distribute - may not be the best strategy
    • Data reprocessing
      • Preliminary plans:
        • 2012 physics delayed streams (bphysics 388M events, SUSY 418M events)
        • 2011 2.76 TeV pp reprocessing - need conditions
        • Possibly, 2012 pp reprocessing for period A - fix luminosity bug
        • Targeted reprocessing of 2010/2011 pp data
    • Interim report on FAX FDR activities - Rob Gardner

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Feb.19-25,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Feb.12-18,2013) Panda production service
      - completed successfully 1,500,077 managed group, MC production,
        validation and reprocessing jobs
      - average ~214,297 jobs per day
      - failed 124,444 jobs
      - average efficiency:
         -- jobs ~93.3%
      - active tasks: 924
         -- distribution by cloud:
 CA:57 CERN:20 DE:117 ES:63 FR:135 IT:82 ND:69 NL:60 RU:3 TW:40 UK:107
 US:231

B)   Major Downtimes:
     - ES/PIC DT (wms01 OUTAGE:migration EMI-1 to EMI-2), Feb.18-20.
     - NL/SARA-MATRIX DT(CE,SE,SRM,BDII OUTAGE:data centre issue),Feb.21
     - IT/INFN-T1 DT (ce01 OUTAGE:Upgrade to EMI2,SL6), Feb.21-28.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:head nodes security update), Feb.21.
     - ES/PIC DT (ce10 OUTAGE:Migration EMI-1 to EMI-2), Feb.22-25.
     - NL/SARA-MATRIX DT(CE,SE,SRM OUTAGE:network+WN upgrade),Feb.25-26.
     - TW/TAIWAN-LCG2 DT(CE,SE,SRM,BDII OUTAGE:fire disaster), Feb.25.

     - New LCG 64d,61d configurations have been released, Feb.19,25.
     - New DB Release 22.1.1,22.1.2 subscribed to sites, Feb.19,22.
     - New ATLAS DDM Dashboard 2.0 M4.4 upgrade released, Feb.21.
     - New ATLAS release TRF caches were distributed:
       -- AtlasOffline 17.0.7 (32b)
       -- AtlasProduction 17.2.8.10,17.0.7.1(32+64b),16.6.70.3(64b)
       -- AtlasMCProd 17.2.1.4.3 (32+64b)
       -- AtlasProd1 17.2.6.2.3 (64b)
       -- AtlasTrigMC 17.2.8.7.2 (32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Feb.19
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : NL/RRC-KI, US/BU-ATLAS-T2
        ........................................
- FR/IN2P3-CPPM job f./cannot access release 17.2.2 on WNs marwn95,
  marwn111.in2p3.fr.GGUS 91615 waiting for reply:upgrade cvmfs after DT
  yesterday,probably a misconfiguration.HC tests,cvmfs issue?Jobs still
  failing,Feb.22.Under investigation.
+ DE/DESY-HH job f./cannot access release 17.2.2 on WN's grid-wn0002,
  grid-wn0150.desy.de.GGUS 91616 solved:run cvmfs over nfs,see the
  effect then many jobs come at the same time to worker node,then the
  setup time for s/w increased significantly.Closed these two nodes for
  atlas jobs for the further inspection.
+ ES/IFAE_PHYS-TOP 1K+ FT F./NO_SPACE_LEFT.Savannah 135991:whitelisted
  the same day.
- NL/IL-TAU-HEP_PRODDISK FT f./NO_SPACE_LEFT.Savannah 135992:auto-
  blacklisted in DDM.
+ NL/RRC-KI DATADISK 660+ FT f./NO_SPACE_LEFT.Savannah 136011:auto-
  blacklisted in DDM.Whitelisted,solved,Feb.22.
+ US/BU_ATLAS_Tier2 2K+ FT f./DEST.:CONN.:SRM.GGUS 91641 solved:SRM
  was offline for a few minutes to fix central deletion issue.Other f.
  appeared/TRANSFER:NO_PROGRESS/SOURCE file doesn't exist,Feb.19-20.
  Caused by tuning up DDM performance with FTS.
+ US/ANALY_MWT2 3K assigned,no activated jobs.Brokerage warnings:skip
  site,notmaxweight.Experts informed,seems fixed,Feb.20.Elog.43002,009.

2) Wed Feb.20
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ UK/UKI-LT2-UCL-HEP 1.1K+ FT f./DEST.:CONN.:SRM.GGUS 91644 assigned:
  more f./SOURCE:CONN.:SRM the same day.Fixed:forgot to restart
  services at the end of downtime.
+ DE/PRAGUELCG2 FT f./GENERAL_F.:dpmpool13.farm.particle.cz:No route to
  host.GGUS 91645 verified:dpmpool13.farm.particle.cz had HW problem
  with RAID controller.75% job f./Get:Globus system error.dpmpool13 is
  repaired and running.
- DE/FZK-LCG2->GLASGOW 1340 FT f./GRIDFTP_ER.:server err.500.GGUS 87958
  in progress updated:also 30 FT f. to LIV.Still ~1K|4h f.,Nov.22,25.
+ DE/HEPHY-UIBK FT f./SOURCE+DEST.:CONN.:SRM.GGUS 91672 verified:started
  dpm daemon,Feb.21 .
- RU/RRC-KI-T1_DATADISK<-PIC 730|4h FT f./server err.426:conn.timed out.
  GGUS 91648 reopened:in a nutshell, GridFTP doors return internal
  IPs of the pools for external clients,so they can't connect to the
  data channel endpoint.More f./end-of-file was reached,Feb.22.Fixed.
  GridFTP doors were hitting connection limit.Increased it by 10 times.
  Still many f./SOURCE:invalid SRM version [] for endpoint,Feb.25.
+ CA/AUSTRALIA-ATLAS still 24 FT f./DEST.:checksum mismatch.GGUS 91589
  updated,solved:the errors were due to the no connectivity from TRIUMF
  SRM to the non-Canadian internet.
+ IT/INFN-NAPOLI ~270 FT f./DEST.:FILE_EXISTS.GGUS 91675 solved:the
  corrupted f/s systems are recovered,but some files may be lost.
  Transfer efficiency is 100%,Feb.21.

3) Thu Feb.21
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50% : NL
        ............................................
+ CERN/VCR Jabbim server was down for 2 hours,OK after.Elog 43023-24.
+ CERN/PandaMon DB release replication error.The problem from prodsys
  system.Savannah 100548,100568,100567 fixed:new RU cloud name added
  into the prodsys configuration.
+ CERN/SSB error/can't load tables.Fixed.Elog.43037-38.
- CERN/CERN-PROD SLS storage space degraded:ATLASMCTAPE 77% availab.,
  ATLAS CALIBDISK storage space degraded in SLS.Elog 43050.
+ CERN/VOMS proxy problems affecting prod.+analy jobs.GGUS:91704,91706,
  91710 verified:recreated the certificates with the correct subject,
  Feb.22.
+ NL/RU-PROTVINO-IHEP ~3K FT f./DEST.:SRM Authentication failed.GGUS
  91677 solved:FT efficiency of NL cloud at ~7%.Problem with cooling
  system at the site fixed.
+ US/SWT2_CPB_USERDISK ~370 deletion errors.GGUS 90644 solved:38 DSs
  were manually cleaned at the site and removed from LFC the same day.
+ US/BNL_ATLAS_RCF 91% (~3K) job f./cmtside timeout+lost heartbeat.GGUS
  91695 verified:2.4 jobs evicted by PHENIX,400+ cmt timeouts due to
  high load on the NFS server upon the startup of a large number of
  jobs,fixed.
+ IT/INFN-ROMA1 job f./too little space left on local disk.GGUS 91708
  solved:WN 411 affected,has been excluded,cleanup to be be done.No f.
  anymore,Feb.25.
- UK/UKI-LT2-RHUL ~100 FT f./DEST.:SRM_ABORTED.GGUS 91711 in progress:
  still ~400|4h same f.,Feb.25.

4) Fri Feb.22
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ ES/LIP-LISBON_LOCALGROUPDISK<-TRIUMF FT f./GRIDFTP_ER.:globus_gass
  _copy_register_url_to_url transfer timed out.GGUS 91715 solved:not a
  site problem. One VOMS at CERN had a wrong certificate,and depending
  on the precise VOMS used to issue proxies,transfers/job submissions
  might fail.Fixed,see GGUS 91706.
+ ES/PIC_MCTAPE 6K+|4h FT f./DEST.:server err.451.No write pools
  configured.GGUS 91739 verified:an error creating tape file families,
  fixed,Feb.23 at~10am.
+ IT/INFN-COSENZA Frontier/squid stopped.GGUS 91716 solved:caused by
  a separation from network GARR begun around 22 during the night
  until 9:30.Fixed.

5) Sat Feb.23
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : US/SWT2
        ..............................
- US/SWT2_CPB 100% FT f./DEST.:/xrd...has trouble with canonical path.
  can't access it.GGUS 91741 assigned.Blacklisted:Savannah 136087.One of
  the storage hosts went off-line due to a problem with the NIC in the
  machine,this was fixed.Whitelisted.Still some f./DEST.:INVALID_
  PATH:cannot create dir at HOTDISK+PHYS-TOP tokens,Feb.24.
- CA/CA-ALBERTA-WESTGRID-T2 is being decomissioned.Savannah 136086:all
  primary replicas removed.ATLASDATADISK to be reduced to 20 TB.
+ UK/RAL-LCG2 ~10K|4h FT f./SOURCE:SRM_ABORTED.GGUS 91743 solved:large
  number of pending requests were stuck in the Atlas Castor instance.
  These were cleared out (the appropriate daemon(s) restarted)..

6) Sun Feb.24
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
- CERN/Savannah: problem submitting a savannah bug by the shifter to
  ADCO-support.Elog.43077.
- UK/UKI-SCOTGRID-GLASGOW still FT f./GRIDFTP_ER.:server er.500:conn.
  reset by peer.GGUS 91439 waiting for reply updated.
- UK/UKI-LT2-RHUL still ~400|4h FT f./DEST.:SRM_ABORTED.GGUS 91711 in
  progress updated.

7) Mon Feb.25
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50%:CA/AUSTRALIA-NECTAR,US/BU,TW,RU
        .......................................................
- CA/TRIUMF-LCG2 many job f./no such file or dir(group gener input for
  shepra).GGUS 91766 in progress:some local implementations to reduce
  the load on storage system.Unfortunately if the same files are created
  again soon after they were deleted,the implementations still give old
  path for them thus jobs would fail to find the files.This time the
  massive failures are caused by this,under investigation to avoid this.
+ ND/ARC+ARC-t2 many job f./exceeded the memory limit.GGUS 91757 solved:
  Not a site problem.The corresponding group production jobs have task
  memory set too low at 2GB.The failed jobs will be automatically
  increased to 4GB and resubmitted.
- RU/RRC-KI-T1 Frontier down.Elog 43089,43090.Need to check/update
  monitoring for the stie.
- UK/UKI-SOUTHGRID-SUSX many FT f./SOURCE:Invalid SRM version [] for
  endpoint.GGUS 91773 assigned.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+ 17.2.7.5.7 data12_8TeV Reco merge task 1205147 failures:TRF_UNKNOWN |
 "Unable to build inputFileSummary from any of the specified input
 files".Savannah 136081:urgent group production TOP.Many (33 so far)
 jobs have failed at ND-ARC running for UK cloud due to the error while
 opening input AOD files.Task is still in "running",1 remaining job.

-- 17.2.7.5.7 data12_8TeV MergeNTUP task 1205820 failures:TRF_OUTFILE_
 TOOFEW (64131) Transform output file contains too few events.Savannah
 136082:group production TOP.1 remaining job has failed 6 attempts in
 a row at DE/FZK due to the same reproducible error.Tasks is still
 "running".

-- 17.2.7.5.7 data12_8TeV Reco merge task 1205059 failures/waiting:
 special brokerage for useT2 - zero share at CYFRONET-LCG2.Savannah
 136099:urgent group production Top.All jobs have failed/cancelled 2
 attempts each.All 341 jobs for some reason can't be reassigned by Panda
 to other site in DE cloud.The site is allowed to run only evgen or
 simul while the job is reco,the DS is missing at T1,subscription is
 still incomplete.

-- 17.2.8.4 valid12 BSFw+AtlasG4+overlay_bs task 1206117 failures:
 Invalid index specified for accessing a variable in the AttributeList
 ( CORAL : "AttributeList" from "CoralBase" ).Savannah 100575:MC12
 Overlay VALIDATION.All jobs have have failed up to 5attemps in a row
 each at various sites running for US cloud at AtlasG4 step.Task is
 in "failed" state.No comments published.

 -+ 17.2.8.7.1 mc12_8TeV DigiMReco task 1206204  failures: Attribute
 Error:'<class 'AthenaCommon.AlgSequence.AlgSequence'>'object has no
 attribute 'InTimeAntiKt4TruthJetsFixIt'.Savannah 136069:MC12  production - HIGGS.All jobs have failed repeatedly at various sites
 in CA cloud due to the error on ESDtoAOD step.Task ABORTED.No comments
 added.

 -+ 17.6.0.3 valid1 Sim_tf task 1206203 failures: Payload core dump
 Savannah 100592:Manual Sim_tf test.All jobs have failed 5 attempts
 in a row each at lest at 2 different queues/sites in FR cloud due to
 the error which sounds as the wrong definition of job/task input
 parameters (unrecognized arguments).Task ABORTED.No comments added.

-+ 17.2.1.4.3 mc12_valid DigiMReco tasks 1207341-42 failures:TRF_UNKNOWN
 :Could not find the data object PixelRDOs.Savannah:136089:MC12
 production - EGAMMA.All jobs failed repeatedly so far at various sites
 running for US cloud due to the same reproducible error on the RAWtoESD
 step.Tasks ABORTED.JO configuration problem.The tracker is switched off
 while running the trigger.Will be followed up with the experts.

-+ 17.2.1.4.2 mc12_valid DigiMReco task 1207405 failures: TRF_UNKNOWN:
 commitOutput failed.Savannah 136091:MC12 production - JETETMISS.All
 jobs have failed so far at different sites running for IT cloud due to
 the same reproducible error on RAWtoESD step.Task "failed",need to pin
 down for the experts.

++ 17.2.5.2 mc12_8TeV evgen task 1206757-1206772 failures:RuntimeError:
 inputGenConfFile...is incompatible with inputconfch.Savannah 136092:
 MC12 production - SM.All jobs have failed multiple attempts at various
 sites running for CERN cloud due to incompatibility in JobOptions.Tasks
 are in "failed" and ABORTED state.The JOs needed a minor modification.
 Issue is fixed and the (resubmitted) tasks are now running.

-+ 17.2.1.4.2 mc12_valid DigiMReco tasks 1207825-28 failures: FATAL
 Please don't call this tool if usePixel and useSCT are...Savannah
 136101:MC12 production - JETETMISS.All jobs have failed repeatedly at
 various sites running for IT cloud due to the error on the RAWtoESD
 coming from InDetTrigSiDetElementsRoadMaker.Tasks ABORTED.JobOptions
 configuration problem.The tracker is switched off while running the
 trigger.Will be followed up with the experts.

-- LeslieGroer - 2013-02-26

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2013-02-26 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback