TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130521 (2013-05-21, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-05-21

General News

  • ADC and ATLAS Computing News [20130521]
    • ADC Weekly meeting
    • ATLAS Installation System 2
      • Some fixes in the next version of the pilot to solve multicore and CERN specific issues
      • SL6 bug in latest kernel ( openfs() call returns EROFS for a file on a read-only mount point that is opened with O_CREAT|O_RDONLY flags)
        • Temporary workaround - mount CVMFS r/w
      • Sites will need revalidation once SL6 installed - should be under separate PanDA resource with separate BDII tags for now
    • DQ2 client 2.4.0
      • Support for older client versions going away explicitly in June (meaningful error message to be sent back to client)
      • Still some services that need to be updates (Hammercloud, consistency and deletion services, lxplus nodes)
      • Users should switch to /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/DQ2Client/current/setup.sh
    • DDM Ops
      • Some discussion about Beijing T2 disks out of warranty and pledged resources
      • LRZ-LMU changed hostname of SE which took a fair amount of work and several days to reregister 2.7M entries in LFC
    • MC Report
      • Some discussion of the different characteristics and patterns for MC production this year versus last
        • Higgs discovery, key dates for results, new datasets, etc...
      • Now: pp run over, MC12 production nice and steady, no major redos necessary, conferences ~month apart
        • Still some activity generated by groups
        • More upgrade samples expected (14 TeV simulation, higher mu digitization and reconstruction)
    • AMOD Report
    • Guidelines for Space Reservations for T2 Space Tokens

  • Security News [20120801]

  • ROC_Canada News [20130423]
    • Current shift - Sergey Chelsky April 22-26, 2012
    • Next Week - Erming Pei

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

      World-wide weekly ATLAS ADCoS summary (May 14-20,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (May 14-20,2013) Panda production service
      - completed successfully 1,373,532 managed group, MC production,
        validation and reprocessing jobs
      - average ~196,219 jobs per day
      - failed 97,526 jobs
      - average efficiency:
         -- jobs ~93.3%
      - active tasks: 729
         -- distribution by cloud:
 CA:39 CERN:8 DE:137 ES:34 FR:123 IT:51 ND:39 NL:53 TW:19 UK:91 US:135

B)  Major Downtimes:
     - CERN/CERN-PROD DT (ce206 OUTAGE:upgrade to EMI2), May 8-14.
     - RU/RRC-KI-T1 DT(CE,SE,SRM,BDII warning:routers maint.), May 15.
     - CA/TRIUMF-LCG2 DT (SE,SRM,BDII warning:host cert.update), May 15.
     - IT/INFN-T1 DT (ce05 OUTAGE:upgrade to EMI3,SL6), May 9-16.
     - IT/INFN-T1 DT (ce06 OUTAGE:upgrade to EMI3,SL6), May 20-28.
     - ND/NDGF-T1 DT (SE,SRM warning:some data unavailable), May 15-16.
     - ND/NDGF-T1 DT (SE,SRM warning:security patch reboot), May 17.
     - FR/IN2P3-CC DT (cccreamceli09 OUTAGE:maintenance), May 17.
     - NL/SARA-MATRIX DT (SE,SRM warning:h/w problem with tape), May 17.
     - DE/FZK-LCG2 DT (SE,SRM OUTAGE:dCache upgrade to v.2.2), May 20.

C)  Other news:
   - New LCG 65 configuration has been released, May 17.
   - New ATLAS release TRF caches were distributed:
    -- AtlasOffline    -
    -- AtlasProduction -
    -- AtlasPhysics    17.2.7.5.15(32+64b)
    -- AtlasMCProd     -
    -- AtlasIBLProd    -
    -- AtlasProd1      17.2.7.2.6 (64b)
    -- AtlasTrigMC     -
    -- EvgenTP4MC11    -

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue May 14
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
- DE/PSNC 274 FT f./SOURCE:CONN.:SRM.GGUS 9400 assigned,updated:90% FT
  f./DEST.:CONN.:SRM,May 16.
+ DE/MPPMU PERF_MUONS DaTRI requests 534542,51,60,6978  are in
  AWAITING_SUBSCRIPTION (QUOTA_EXCEEDED).Savannah 137606:subscriptions
  were stopped,May 20.
- DE/UNI-BONN->PRAGUE 180 FT f./DEST.:Checksum mismatch.Savannah 101483:
  100% f. for week for channel BONN-PRAGUES,ADLER32 doesn't work.
- DE/CYFRONET-LCG2 FT f./SOURCE:GRIDFTP_ERR:the operation was aborted.
  GGUS 94052 assigned:blacklisted in DDM:Savannah 137626.H/w on one pool
  got broken and is out of warranty,have to wait until repaired,May 18.
+ US/NET2 FT f./DEST.:HTTP_TIMEOUT:SRM/Checksum mismatch.GGUS 94050
  solved:file system problem fixed the same day.

2) Wed May 15
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : RU/RRC-KI-T1
        ................................................................
+ US/BU_ATLAS_Tier2 frontier squid down.GGUS 94054 solved:AGIS and
  monitoring change was done for moving the host,fixed May 16.
+ FR/GRIF-LAL 91% job f./No space left on device.GGUS 94076 solved:user
  (not atlas) took all space available on /tmp, not cleaned after job
  execution.Did clean /tmp repository on WNs manually the same day.
+ CA/CA-MCGILL-CLUMEQ-T2 85% FT f./SOURCE:SRM returned an error, but
  with an invalid code/DEST.:SRM_ABORTED.GGUS 94085 solved: issue with
  storm b/e and Bdii service fixed the same day.
+ RU/RRC-KI-T1 many FT f. after DT/could not open connection to
  sdrm.t1.grid.kiae.ru.GGUS 94098 solved:was non-symmetric routing after
  IOS upgrade,fixed the same day.
+ IT/INFN-ROMA2 frontier squid down.GGUS 94105 verified:works again,May
  18.

3) Thu May 16
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50% : -
        ............................................
+ IT/INFN-MILANO-ATLASC ~1K FT f./SOURCE+DEST.err.GGUS 94106 solved:f/s
  daemon mmfsd crashed on al the servers overnight,restarted,May 17.
- DE/LRZ-LMU->US 100% FT f./SOURCE:Invalid SRM version [] for endpoint.
  GGUS 94110 reopened:blacklisted,Savannah 137652.Still all (20K+) FT f.
  from SCRATCHDISK,May 18.Some (possibly) DB related errors in the SRM,
  opened a bug against dCache... in the meantime restarted the SRM.No
  err.|12h to US (but only 98 FT),May 20.
+ DE/UNI-FREIBURG FT f. after DT/SOURCE:SECURITY_ER.:SRM Authentication
  failed.GGUS 94136 solved:set amount of allowed dcache user processes
  to unlimited after upgrade to dCache 2.2 SE,May 17.
+ CA/TRIUMF frontier unavailable.GGUS 94127 solved:network issues fixed.
+ US/BU_ATLAS_Tier2 606 job f./installPyJobTransforms failed: Can't find
  [AtlasPhysics_17_2_7_5_15_i686_slc5_gcc43_opt].GGUS 94129 solved:
  caused by moving squid from atlas.bu.edu to another host,AGIS fixed.
- FR/TW job f./Failed to get pacman-3.18.3.tar.gz from any source url.
  Savannah 137656.

4) Fri May 17
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ DE/PRAGUELCG2_PPSLOCALGROUPDISK 180+ FT f./DEST.:CONN.:SRM.GGUS 94182
  solved:Related to https://rt4.cesnet.cz/rt/Ticket/Display.html?id=
  73343.Fixed,May 18.
- DE/MaiGRID<-AGLT2 1200 FT f./GRIDFTP_ER.:server err.426.GGUS 94183 on
  hold:informed the user about the connectivity problem to AGLT2 and
  gave him the advice to transfer the data via fzk. In order trace the
  (routing?) problem have to talk to the network admins of the
  University, which will be done next week.

5) Sat May 18
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
+ IT/INFN-NAPOLI-ATLAS all FT f./failed to contact on remote SRM.GGUS
  94184 solved:SRM crached,restarted May 19.Savannah 137684.
+ IT/INFN-ROMA3 400+ FT f./SOURCE:the request token is empty.GGUS 94186
  solved:problem on local storage controllers,fixed,May 20.Savannah
  137692.
+ DE/UNI-FREIBURG_SCRATCHDISK 1K+ FT f./locality is unavailable.GGUS
  94187 fixed:several pools had problems after the upgrade and had to be
  restarted.HC tests and FT is OK,May 20.Savannah 137697.
- DE/HEPHY-UIBK->FZK 700+ FT f./GRIDFTP_ER.:File does not have expected
  length.GGUS 94188 assigned.
+ US/BNL-OSG2_DATADISK 160+ FT f./SOURCE file doesn't exist.GGUS 94189
  solved:problem is not on the site, the replica of dataset was deleted
  from BNL-OSG2_DATADISK on 18.05.2013,so FTS should not ask copy from
  there,it's problem somewhere on Atlas DDM side.Will be tracked on
  Savannah.

6) Sun May 19
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
+ CA/TRIUMF-LCG2 FT f./SOURCE:CONN.:SRM.GGUS 94190 solved:files were
  transfered in subsequent attempt, therefore assuming it was only
  temporary overload of the SRM.
+ DE/GoeGRID all FT f./failed to contact on remote SRM.GGUS 94191 solved
  : SRM service is restarted and running finely,a script to detect the
  dead SRM service, gridftp services and to restart them is installed.
  Savannah 137696.

7) Mon May 20
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50%: -
        .......................................................
+ DE/UNI-FREIBURG FT f./locality is UNAVAILABLE.GGUS 94187 solved:
  Several pools had problems after the upgrade and had to be restarted.
  Fixed:HC tests,FT are OK.Savannah 137697.
- FR/GRIF-LAL job f./No space left on device.GGUS 94208 assigned:some
  WN affected,100% job f.List of those WNs provided.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

++ 17.2.6.2 mc12_14TeV evgen tasks 1252938-42 failures:ATH_JOP_NOTFOUND
 Savannah 137601:MC12 production - JETETMISS.Jobs have failed repetedly
 in NL,US,ND,IT clouds.Tasks ABORTED,added the JOs to the tarball.

-- 17.2.2.2 mc12_8TeV Reco merge tasks 1251226,27,34 failures:"Too many/
 too large input files".Savannah 137610:MC12 production - JETETMISS.All
 jobs have failed up to 25(!) attempts each at various sites running for
 ES,TW clouds.Task are still submitting without success,no comments
 published so far.

-+ 17.2.10.3 mc12_8TeV evgen tasks 1252734-1252773 failures: "Could not
 find 1000006 1 in _table. In HadronSelector::lightestHadron().Savannah
 137633:MC12 production - SUSY.All jobs have failed multiple attempts in
 different clouds.Tasks ABORTED,no comments published.

-+ 17.2.7.5.15 data12_8TeV Reco merge tasks 1255060-64,029,30-32,34,38-
 40 failures:out of memory.Savannah 137643:group production SUSY.Many
 jobs have failed repeatedly in various clouds.Tasks ABORTED:s/w problem

-+ 17.3.10.1 valid1 DigiMReco task 1251332 failures:TRT_ID ERROR TRT_ID
 is NOT valid for this layout.Savannah 101500:MC12 VALIDATION 14TeV
 SampleA s1636 r4548.Many jobs have failed repeatedly at different sites
 running for US cloud.Task set to "finished",no comments published.

-- 17.3.10.1 valid1 DigiMReco task 1251336 failures:histogram MV2/AntiKt
 4Truth/MV2Calib does not exist - you are probably using an old database
 tag.Savannah 101494:MC12 VALIDATION 14TeV SampleA s1636 r4548. Jobs
 failed 10 attempts at various sites running for US cloud.Tasks is still
 "submitting",no comments added so far.

-+ 17.2.10.3 mc12_8TeV evgen tasks 1255330-1255334 failures:Unwanted
 ATOOLS::exception caught.Savannah 137659:MC12 production - HIGGS.All
 jobs have failed multiple attempts at various sites running for ND,DE
 clouds.Tasks ABORTED,no comments added.

-+ 17.2.10.3 mc12_8TeV evgen tasks 1255783-90 failures:Pythia8 ERROR
 Unable to read requested LHE file: events.lhe!! Savannah 137685:MC12
 production - EXOTICS.All jobs have failed multiple attempts at various
 sites/clouds.Tasks ABORTED,no comments added.

++ 16.6.8.3  mc11_7TeV evgen task 1255602 failures:GSL error: endpoints
 do not straddle y=0 in file: brent.c , line: 74, error code 4.Savannah
 136687:MC11 production - SUSY.All jobs failed 11 attempts each at
 different sites in IT cloud.Task ABORTED,it needed a later version of
 Herwig++.

-- 17.2.7.5.3 data12_8TeV MergeNTUP tasks 1255090,1255127,37,39 failures
 :'std::bad_alloc'(out of memory).Savannah 137688:group production SMWZ.
 All jobs have failed multiple attempts at various sites running for
 UK,US,ES clouds.Tasks are still "submitting",no comments published so
 far.

-- 17.3.10.1 valid1 DigiMReco tasl 1244610 failures:special brokerage
 for manyInput - not online at CERN_8CORE.CPU core mismatch at CERN-PROD
 Savannah 101533:McAtNloJimmy_CT10_ttbar_LeptonFilter.All jobs have
 failed 21(!) attempts in a row at CERN.Task is still "submitting"
 without success,no comments published so far.

-- 17.2.10.1 data12_8TeV merging task 1254297 failures:lost heartbeat.
   Savannah 137713:DESD-merging for physics_HadDelayed periodL.All jobs
   have failed up to 8 attempts in a row each at different sites in FR
   cloud.Task is still "submitting".

-- LeslieGroer - 2013-05-21

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2013-05-21 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback