TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121218 (2012-12-18, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-12-18

General News

  • UMD 2.3.1 has been released - https://operations-portal.egi.eu/broadcast/archive/id/850
  • ADC and ATLAS Computing News [20121218]
    • ADC Weekly meeting
    • LHC total delivered: 28.8 fb^-1 (27.0 recorded by ATLAS). pp run terminated yesterday. Heavy Ions in mid-January.
    • 7 minute power cut at CERN tomorrow morning - not supposed to affect grid operations as on UPS.
    • Issues with priority creep
      • Historic way to finish old tasks - add 10 to priority every day, up to max 990
      • Problems
        • prio > 950 forces job to T1 only (avoid assigned and transferring delays)
        • if cpucount > T1 maxtime the stuck
          • now we increase cpucount in 2 ways: scouts and on walltime failure
        • reduces resources for hi prio jobs!
          • large lower prio old tasks swamp newer higher prio ones
        • requester sets intended prio in steps of 100
          • after 10 days, this is washed out
      • Proposal
        • short term: set cap at 930
        • in new year, reduce daily creep to 1? - needs to be examined
        • medium term: replace creep with intelligent task finisher
        • boost nearly finished tasks and small tasks, etc.
    • Reprocessing
      • Periods A-H done, including reconstruction/merging
      • Still some cleanup and unmerged DESD for physics streams, and unmerged ESD/AOD for JetCalibDelayed stream (~0.13 PB)
      • Expect some of this to run over the Xmas break, but not sure who will be watching
    • MC Production
      • Planning over the Christmas Break (until end-Jan)
        • Full sim hits go to disk, merged full sim go to TAPE; fast sim hits go to TAPE
        • AOD and merged AOD go to DISK
        • Few hundred million events to be produced - still deciding to double the fast sim production. Digi+reco is a bottleneck.
        • Some concern about disk space needed plus monitoring at sites and competition with group production
    • Group Production
      • Physics groups are asked for prioritized list of DPD production by end of this week. Also asked what datasets can be deleted.
      • Already have 1,700 cpu-years for data and 561 cpu-years for MC, 2.8PB of disk
      • Are expecting to use 20% of T1 and 50% of T2 resources - no fudge factors
      • Resource estimation: http://nozturk.web.cern.ch/nozturk/DPDProduction/DPD_Resource.numbers
    • AMOD Report
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Dec.11-17,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Dec.11-17,2012) Panda production service
      - completed successfully 1,697,234 managed group, MC production,
        validation and reprocessing jobs
      - average ~239,891 jobs per day
      - failed 83,501 jobs
      - average efficiency:
         -- jobs ~95.3%
      - active tasks: 1493
         -- distribution by cloud:
  CA:154 CERN:19 DE:172 ES:159 FR:197 IT:114 ND:75 NL:180 TW:70 UK:211
  US:142

B)  Downtimes:
     - NL/SARA-MATRIX DT (SE,SRM OUTAGE:dCache upgrades), Dec.10-11.
     - NL/SARA-MATRIX DT (creamce2 OUTAGE:upgrade to UMD2), Dec.11-18.
     - FR/IN2P3-CC DT (cccreamceli05,6,9,10 OUTAGE:electric m.),Dec.9-13
     - FR/IN2P3-CC DT(SE,SRM OUTAGE:installation power supply),Dec.10-12
     - FR/IN2P3-CC DT (FTS warning:electric maintenance), Dec.10-12.
     - UK/RAL-LCG2 DT (all services warning:at risk for UPS test),Dec.11
     - DE/FZK-LCG2 DT (Frontier OUTAGE:Tomcat service upgrade), Dec.11.
     - DE/FZK-LCG2 DT (cream-2,6,8 OUTAGE:service retirement), Dec.16-31
     - DE/FZK-LCG2 DT (CE,SE,SRM,BDII OUTAGE:network outage), Dec.17.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:OS,firmware,dCache upgrade), Dec.13.
     - ND/NDGF-T1 DT (CE-arc.bccs.uib.no OUTAGE:arc update), Dec.14-18.
     - IT/INFN-T1 DT (SE,SRM warning:b/e server substitution), Dec.13.

    Other news:
     - CERN:ATLAS Distributed Computing Tier-1/2/3 Jamboree, Dec.10-11.
     - New OSG Software version 3.1.12 released, Dec.13.
     - New ATLAS releases and TRF caches were distributed:
       -- AtlasOffline    - 17.2.8 (32+64b)
       -- AtlasProduction - 17.2.7.9(32+64b)
       -- AtlasPhysics    - 17.2.7.5.2|3(32+64b),17.0.6.9.3(32b)
       -- MuonAlignment   - 17.2.7.7.1(32b).
       -- TagProd         - 17.2.7.9.1(32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Dec.11
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ IT/INFN-BOLOGNA-T3_SCRATCHDISK FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 89346
  solved:after restart the service Storm b/e & f/e are OK.
+ IT/INFN-NAPOLI-ATLAS FT f./DEST.:CONN.:SRM.GGUS 89528 solved:all FT
  (~800) TAIWAN-LCG2->NAPOLI f./DEST.:Parsing document'/usr/etc/glite-
  data-srm-util-cpp.patterns.xml' failed.The cron job "fetch-crl"
  problem fixed,transfers succeed,Dec.12.
+ UK/UKI-SOUTHGRID-BHAM-HEP_PRODDISK FT f.(eff.~5%)/DEST.:SRM_ABORTED/
  SRM-FAILURE/CONN.:SRM/Couldn't get user mapping/AGENT error.GGUS 89436
  solved:due to ongoing issues with upgrading from Glite to EMI2 head
  node,reverted back to Glite and it should be back to normal.

2) Wed Dec.12
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ DE/FZK Frontier down.Elog.41639-40,48,51.After DT for Tomcat the
  problem was identified as a soft link to the incorrect java version,
  fixed.
- DE/GoeGrid stll 162 job f./stage-out errors.GGUS 89114 in progress
  since Dec.2 updated.
+ FR/IN2P3-LPSC 300+ FT f./SOURCE:GRIDTP_ER.:globus_ftp_client:the
  operation was aborted.GGUS 89533 solved:local copy (lcg-cp) at LPSC
  of the file seems to be working.But ~600 job f./stage in|out err.
  Restarted service srmv2 on DPM server,Dec.13.
- FR/RO-07-NIPNE ~1.2K FT f./SOURCE:CONN.:SRM.GGUS 88797 reopened:many
  FT+job f.,Dec.13.Problem with DPM-disk down.2K+ FT f./DEST.:CONN.:SRM,
  Dec.15.DT on DPM((tbit00.nipne.ro) announced.
+ ND/ARC job f./too little space left on local disk to run job/DB access
  failed.GGUS  89617 solved:2 WNs affected at ce01.titan.uio.no.1-Fixed.
  2-Most likely a cvmfs cache problem.Files are there and report being
  of size about hundred of MB but any operation on these reports file
  empty.Node is being drained so that cvmfs cache can be cleared,Dec.13.


3) Thu Dec.13
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: US:NET2,FR:LPSC,RO-07,RO-16
        .....................................................
+ US/NET2 ~1K FT f./DEST.:CONN.:SRM.GGUS 89621 solved:some f/s issue
  fixed.
+ IT/INFN-T1 100 job f./staging input file failed.Savannah 134530:site
  problem,most likely storage overloaded.No errors of this type anymore.
- UK/UKI-SOUTHGRID-BHAM-HEP_PRODDISK still FT f./DEST.:CONN.:SRM.GGUS
  89436 reopened,on hold:~3K f.,Dec.15.Some issues with EMI2 upgrade.

4) Fri Dec.14
      -> Cloud/site PROD efficiency ever<50% : CERN-RELEASE(t.1099259)
      -> Cloud/site DDM efficiency  <50% : DE/MPPMU
        .....................................................
+ US/UPENN_LOCALGROUPDISK 400 FT f./DEST.:CONN.:SRM.GGUS 89656 solved:
  the issue was fixed the same day.
- UK/UKI-NORTHGID-MAN-HEP FT f./err.on the turl request:StatusOfPut
  Request:SRM_NO_FREE_SPACE.GGUS 89668 in progress:due to FTS3 tests.FT
  ongoing,but with the very high failure rate >1300|4h.,Dec.15.
  Investigating with FTS3 developers.Solution might not be found before
  Christmas.FTS3 tests will be interrupted for the holiday period anyway
+ CA/VICTORIA-LCG2 ~150/4h deletion errors.GGUS 89676 solved:the storage
  lcg-se.rcf.uvic.ca doesn't exist anymore.Savannah 99435:removed the
  replicas from LFC with lcg-uf,Dec.17.
- DE/MPPMU 1000s FT f./SOURCE:Defective credential detected:CRL... has
  expired.GGUS 89688 waiting for reply:had a problem with the IGTF
  bundle and it's updates.Fixed,Dec.15.
+ FR/GRIF-LAL_DATADISK FT f./NO_SPACE_LEFT.Savannah 134576:incorrect
  behavior of DaTRI fixed.
- FR+ND/PHYS-HIGGS:IN2P3-CC+NDGF-T1 DaTri requests awaiting subscription
  QUOTA-EXCEEDED.Savannah 99463.

5) Sat Dec.15
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : ND(due to MPPMU?),TW
        ..............................
- FR/TR-10-ULAKBIM 330 FT f./AGENT err.:No status updates received since
  more than 360sec.GGUS 89689 assigned:2K+ FT f. the same day.
+ FR/IN2P3-CC_SCRATCHDISK FT f./SOURCE file doesn't exist.GGUS 89691
  solved:file has been deleted from SE 10 days ago,now copied.
+ CA/TRIUMF-LCG2 SCRATCHDISK FT f./SOURCE file doesn't exist.GGUS 89690
  solved:Files have eventually been copied.
- DE/FZK-LCG2 ~486 FT f./GRIDFTP_ER.:server err.426:conn.timeout.GGUS
  89110 updated,solved:UK sites probems with FZK are followed in GGUS
  87958 in progress,updated.
- DE/MPPMU_CALIBDISK FT f./SOURCE:Defective credential detected:CRL...
  has expired.GGUS 89695,89688 assigned:ticket cross-linked,seems fixed?
- IT/INFN-MILANO-ATLASC FT f./SOURCE:HTTP_TIMEOUT:SRM.GGUS 89692
  assigned:in addition FT f./DEST.:GRIDFTP_ER.:CKSM (checksum) operation
  aborted.
- UK/UKI-LT2-RHUL 1700 FT f./DEST.:NO_SPACE_LEFT.GGUS 89698 solved:
  DATADISK full,hence it was blacklisted for writing to stop failing
  transfers:Savannah 134591.

6) Sun Dec.16
      -> Cloud/site PROD efficiency ever<50% : TW(known issue)
      -> Cloud/site DDM  efficiency < 50% : TW
        ........................................
+ CERN/CERN-PROD FT f./SOURCE:HTTP_TIMEOUT:SRM.GGUS 89701 solved:SRM-
  EOSATLAS was stuck, service restarted at 10:30 local time, seems to be
  OK now.
+ NL/TECHNION-HEP FT f ./SOURCE file doesn't exist/GRIDFTP_ER.:operation
  was aborted/CONN.:SRM.GGUS 89699 solved:Lustre f/s issue,disconnecting
  clients randomly.A complete clients shut down and writeconf to MDS and
  OST's,seems to solve the problem,Dec.17.
- NL/WEIZMANN-LCG2 GROUPDISK full.Savannah 134594:blacklisted.Site has
  increased the space and suddenly bourricot has had a dropped in the
  occupied space reported by SRM.It is critical that the site corrects
  the srm accounting.
+ IT/INFN-NAPOLI-ATLAS 6800+ FT f./DEST.:Parsing document'/usr/etc/glite
  -data-srm-util-cpp.patterns.xml' failed.GGUS 89700 solved:CALIBDISK
  not available/DEST.:CONN.:SRM.Fixed.Transfers are on going although at
  a low level there haven't been errors and CALIBDISK is back to normal.
- ES/UAM-LCG2-PHYS-HIGGS(GROUPDISK) NO_SPACE_LEFT.Savannah 134592:
  blacklisted.
+ DE/WUPPERTALPROD 850 FT f./SOURCE:AsyncWait.GGUS 89704 solved:dCache
  was resetted,no errors seen anymore.
- US/BNL-ATLAS 800 FT f./SOURCE file doesn't exist.GGUS 89708 assigned:
  the source files seems to be removed by ATLAS DM in the morning.

7) Mon Dec.17
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ CERN/PandaMonitor,voatlas140 not available.Elog 41786,8,9:fixed.
- US/SWT2_CPB all FT f./DEST.:CONN.:SRM.GGUS 89713 in progress:a problem
  with the system drives in the cluster admin node affects most services
  Unscheduled DT declared.
+ US/MWT2 ~2K job f./Get err.:Failed to get LFC replicas.GGUS 89752
  solved:due to a brief network interruption at IU this morning.
+ TW/TAIWAN-LCG2 disk partition was fixed and recovered.Elog.41764.
- ES/UAM-LCG2 24% job f./Required CMTCONFIG incompatible with WN.GGUS
  89716 in progress.
- DE/GoeGrid 300 job f./Missing installation,cvmfs.GGUS 89754 assigned:
  2 WNs: compute-4-48.local and compute-5-7.local affected.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+ 17.2.5.2.4 data12_8TeV Reco merge tasks 1038894,1038903,1038922,
 1038999,1039307,1039334,1039376,1039395 f./Reached maximum batch system
 time limit.Savannah 134492:group production TOP.All jobs have failed
 multiple attempts at various sites/clouds.5 jobs got maxtime
 automatically reset to 96hrs.Some failed to broker because the prio was
 too high(restricts to T1).Reduced prio and reassigned.Tasks 1038894,
 1038903,1039344,1039376,1039395 ABORTED,but 1038999 is still in
 "submitted",1039307 in "running" state.

-+ 17.2.7.7 mc12_8TeV evgen tasks 1091610-1091747 failures:RuntimeError:
 No 'madgraph...evts' file found.Savannah 134524:MC12 production - SUSY.
 All jobs have failed multiple attempts at various clouds.Tasks are
 ABORTED and the fix is in work.

-+ 17.2.4.1 mc12_8TeV. DigiMReco task 1090997-1091024 failures:Attribute
 Error:'<class 'AthenaCommon.AlgSequence.AlgSequence'>' object has no
 attribute 'InTimeAntiKt4TruthJetsFixIt'.Savannah 134234:MC12 production
 - SUSY.All jobs have failed multiple attempts in various clouds.Tasks
 ABORTED,no comments added.

++ 17.2.7.5.1 data12_8TeV Reco merge task 1092208 failures:Too many/too
 large input files.Savannah 134528:group production Higgs.All jobs have
 failed multiple attempts at UK/UKI-LT2-RHUL.Task ABORTED and
 resubmitted with lumiblock=yes,diskcount=15000.

-- 17.2.7.5 mc12_8TeV evgen tasks 1095966-1095977 failures:No param card
 found at susy_lecompte_martin_compsusy_600_0p8.slha.No such file or
 directory: 'PROC_mssm_0/Events/Test/unweighted_events.lhe'.Savannah
 134554:MC12 production - SUSY.All jobs have failed multiple attempts in
 various clouds.Tasks are in "failed" state.No comments added so far.

-+ 17.1.6.1.2 data12_hip Reco BatchHLTTrf task 1099259 failures:Expected
 output file NTUP_TRIGRATE.01099259._002126.root.1 does not exist.
 Savannah 134577:group production TRIGGER HLT.All jobs have failed
 multiple attempts at CERN-RELEASE.Task ABORTED.

-+ 17.2.7.5.1 data12_8TeV Reco merge tasks 1099495,1099521,1099573,
 1099867,1099885-7,1099893,1099945,1100239,1100257-9,1100265,1100317,
 1102208 failures:AttributeError: '<class 'AthenaCommon.AlgSequence.Alg
 Sequence'>' object has no attribute 'Kt5TopoJets'.Savannah 134601:group
 production Higgs,TRIGGER L1CALO.All jobs have failed multiple attempts
 in various clouds.The problem is due to the number events of the input
 DAOD file is zero.Some tasks are in "failed",others are still
 "submitting" and need to be finished or ABORTED.

-- 16.6.8.3 mc11_7TeV evgen tasks 1102507-10 failures:MC11JobOptions/
 MC11.*.TTbar_MT*_PowHeg_Pythia_P2011C.py can not be found.Savannah
 134602:MC11 production - TOP.All jobs have failed multiple attempts at
 different sites running for US+UK clouds.Tasks are still in
 "submitting" state.No comments published so far.

-- 17.2.2.2 mc12_8TeV Reco merge tasks 1090389,1090391-2 failures:Too
 many/too large input files.Savannah 134603:MC12 production - JETETMISS.
 All jobs have failed multiple attempts at various sites running for CA+
 IT clouds.Tasks are still in "submitted" state.No comments added so far

-- LeslieGroer - 2012-12-17

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2012-12-18 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback