TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121030 (2012-10-30, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-10-30

General News

  • ADC and ATLAS Computing News [20121030]
    • ADC Weekly meeting
    • LHC 0.89 fb^-1 delivered last week.
    • MC Production
      • Some tape problems in DE cloud caused some backlog in pileup production (59 days). Also some backup in UK and NL clouds (48 days in NL!)
      • Changing priorities to speed up production
      • T2's should be utilized better for this - put merged HITS automatically at T2 sites?
      • http://atladcops.cern.ch:8000/j_info/mc12_pile.html
    • Reprocessing
    • T1 disks at INFN, Taiwan running low on space - being watched. T1-T1 replication halted. No T0 data export for INFN.
    • AMOD Report
      • FTS proxy expiration problem thought to be fixed and patch deployed and being watched (GGUS:81844).
        • New bug in reading VOMS attributes also thought tp be corrected and patch to be deployed next week.
        • Bamboo now reports (in pandamon) if too many transferring jobs at a site causes job brokerage to stop
        • FZK and RAL had transparent migration to EMI CREAM
        • New Frontier server alias for TRIUMF went in smoothly
        • FZK had some serious outages on their storage (dCache pools unavailable) which they are still recovering from
        • BNL "at risk" due to hurricane, but seems to have ridden out the storm ok
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Leslie Groer October 29-Nov 2 2012
    • Next Week - Neil Knecht

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Oct.23-29,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Oct.23-29,2012) Panda production service
      - completed successfully 1,348,814 managed group, MC production,
        validation and reprocessing jobs
      - average ~192,687 jobs per day
      - failed 81,777 jobs
      - average efficiency:
         -- jobs ~94.3%
      - active tasks: 1573
         -- distribution by cloud:
CA:126 CERN:161 DE:277 ES:90 FR:129 IT:107 ND:156 NL:162 TW:44 UK:130
US:191

B)   - CERN/CERN-PROD DT (ce206 OUTAGE:migration to ARGUS), Oct.15-22.
     - CERN/CERN-PROD DT (WMS OUTAGE:upgrade to EMI1 Update 19),Oct.23-24
     - IT/INFN-T1 DT (ce09-lcg OUTAGE:closed to submissions), Oct.18-29.
     - DE/FZK-LCG2 DT(cream1 OUTAGE maintenance,EMI2 update),Oct.22-Nov.5
     - DE/FZK-LCG2 DT (SE,SRM warning:330TB on failing storage),Oct.22-25
     - ND/NDGF-T1 DT (ce01.titan.uio.no OUTAGE:some issues), Oct.22-23.
     - ND/NDGF-T1 DT (SE,SRM warning:dcache upgrade on DCSC), Oct.24.
     - UK/RAL-LCG2 DT(lcgwms02 OUTAGE:EMI WMS upgrade,v3.3.8), Oct.21-25.
     - UK/RAL-LCG2 DT (lcgce03 OUTAGE:repl.EMI-2 CREAM nodes), Oct.23-30.
     - ES/PIC DT (SE,SRM warning:backup tests on LHCOPN), Oct.23.
     - NL/SARA-MATRIX DT (SE,SRM warning:tape b/e restart), Oct.24.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:SRM headnode patching), Oct.29.
     - FR/IN2P3-CC DT (CE warning:GE batch system upgrade), Oct.29.

     - Express stream reprocessing finished, Oct.24.
     - Bulk reprocessing expected to start this week.
     - New DB Release 20.7.1 subscribed to sites, Oct.24.
     - New ATLAS releases and TRF caches were distributed:
         -- AtlasProduction - 16.6.7.35(32+64b), 17.2.7.3(32+64b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------
1) Tue Oct.23
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
- IT/INFN-MILANO-ATLASC_LOCALGROUPDISK still 0% eff.FT f./DEST.:Checksum
  mismatch.GGUS 86129 reopened,in progress:still 11787 bad files with
  missing checksum found,to be removed and declared lost.
+ DE/HEPHY-UIBK FT f./DEST.:CONN.:SRM.GGUS 87677 reopened,solved:had
  another crashed daemon,but it should be stable since then,Oct.29.
+ ND/ARC still 2K+ job f./transformation not installed in CE.GGUS 83778
  reopened,solved:1500 t.937944 job f./LRMS er.:(15)Job probably
  exceeded memory limit.original issue fixed(?) New ticket for memory
  failures requested.
+ NL/IL-TAU-HEP still FT f./GRIDFTP_ER.:server err.500:I/O error.GGUS
  86486 reopened,solved(?):the problem is not from the TAU site(?)

2) Wed Oct.24
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
- FR/ROMANIA07 again job f./Get err.:adler32 mismatch on input file.GGUS
  86394 reopened:asked to delete/rewrite the files to solve the problem,
  still some err.,Oct.29.
- DE/PSNC FT f./failed to conntact on remote SRM.GGUS 86480 waiting for
  reply:also deletion errors.Problem with srmv2.2 solved the same day.
  Seems "fixed" the problem with ChecksumGetter timeouts in Atlas
  transfers to PSNC by changing the number of parallel file transfers to
  1.This was needed to avoid timeouts in trans.of files of size>0.5GB,
  which we have in Atlas many.This reduces significantly thruput to site
+ DE/WUPPERTALPROD_SCRATCHDISK ~420 FT f./SOURCE:locality is UNAVAILABLE.
  GGUS 87788 verified:several pools had problems,restarted the same day.
+ US/BNL 1000s FT f./AGENT er.creating file for memmap /var/tmp...No
  space left on device.Fixed the same day.Elog.40369.
+ US/UPenn:INVALID_PATHserver er.500:no such file or dir.GGUS 87760
  solved:this was a real "full filesystem" problem,not otherwise caught
  because of a bad configuration.Both issues have been resolved,Oct.25.
- US/BU_ATLAS_TIER2 ~200|h deletion errors/Er.reading token data header.
  GGUS 84189 on hold updated.
+ US/BU_ATLAS_TIER2 ~1700 FT f./DEST.:CONN.:SRM.GGUS 87784 assigned.

3) Thu Oct.25
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: -
        ............................................
- FR/IN2P3-LAPP Frontier|squid still down in monitoring.GGUS 87227
  waiting for reply:new Lapp-squid02.in2p3.fr is fully functional,but
  test continues with old LAPP squid.Experts informed,no answer?
- ND/NDGF-T1 ~100 FT f./DEST.:Checksum mismatch.GGUS 87791 in progress:
  All the checksum mismatch errors are from INFN-MILANO-ATLASC,see GGUS
  86129,cross-linked.
- DE/FZK-LCG2 3K+|4h FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE.GGUS 87790,87826,87526 in progress:still working on assuring a
  baseline activity and gaining stability.1 tape got stuck in a tape
  drive.Have to wait for a technician to fix this Oct.30.Most of the
  files ATLAS wants to stage currently are located on this specific tape,
  therefore a lot of failures occure.
+ TW/TAIWAN-LCG2 DATATAPE  ~40 FT f./SRMV2STAGER:SRM_FILE_UNAVAILABLE:
  File has no copy on tape and no diskcopies are accessible.Savannah
  98437,GGUS 87796 solved:fixed the stuck tape,files accessible,Oct.29.
+ US/MWT2 2K+ job f./Get er:Failed to get LFC replicas:Bad credentials.
  GGUS 87805 solved:the nodes failing these jobs had an old version of
  CVMFS.Set offline for upgrade.
- US/SLACXRD ~220 FT f./DEST.:Request timeout due to internal err.GGUS
  87815 assigned.
- NL/SARA-MATRIX_LONG job f./getSiteroot:Missing installation:No such
  file or dir:/cvmfs/atlas.cern.ch...GGUS 87812 in progress: WN v31-20,
  mount point has been remounted files available again,Oct.26.
- IT/INFN-ROMA1 DATADISK+PRODDISK ~300|h FT f./DEST.:FILE_EXISTS.Savannah
  98459.

4) Fri Oct.26
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ CERN/CERN-PROD 10K FT f./GRIDFTP_ER.:server er.500:open/create error.
  GGUS 87850 solved:EOSATLAS namespace crashed.The service was
  transferred to the standby machine.
- US/BNL_CVMFS_1 some job f./IOErr.:(2,'No such file or dir','Generate.
  log').GGUS 87818 in progress.
- DE/DE-FZK_LCG2 FT f./SRMV2STAGER:SRM_FAILURE:lifetime expired 76/
  changing file state because request state has changed 61.GGUS 87826,
  87526 in progress.
- UK/UKI-SCOTGRID-GLASGOW many job f./Get er.:Staging input file failed/
  no such file or dir./failed to remove file.GGUS 87838 in progress:
  associated with disk071,065 suffered from the know high-load issue
  (with a load of over 300) due to issues with DPM and many IO
  connections. Rfiod was paused on the machine on the afternoon 27th
  which is what probably caused the staging-in fails.Sorted,Oct.29.
+ NL/IL-TAU-HELP FT f./SOURCE file doesn't exist/GRIDFTP_ER.:server er.
  500:i/o err.GGUS 87849 solved:file deleted Oct.29.
- NL/TR-10-ULAKBIM ~2200 FT f./DEST.:CONN.:SRM.GGUS 87857 in progress:all
  FT keep failing,Oct.28.The srmv2 service was started again and checked
  with a small file transfer.Works,Oct.29.

5) Sat Oct.27
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
- CERN/CERN-PROD->CSCS-LCG2 1|49 times FT f./SOURCE:INVALID_SIZE:empty
  file size returned.Savannah 133352.
- DE/FZK-LCG2 FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_FAILURE:
  lifetime expired.Savannah 133353.
+ FR/RO-07-NIPNE many FT f./SOURCE:CONN.:SRM.GGUS 87859:transfers
  completed soon after this,no err.anymore the same day.
- FR/IN2P3-CC still 190|4h FT f./SOURCE:AsyncWait.GGUS 87344 waiting for
  reply updated.

6) Sun Oct.28
      -> Cloud/site PROD efficiency ever<50% :
      -> Cloud/site DDM  efficiency < 50% : Nl/TR-10-ULAKBIM
        ........................................
+ ES/EELA-UTFSM FT f./GRIDFTP_ER.:globus_gass_copy_register_url_to_url
  transfer timed out.GGUS 87861 solved:no more err.,Oct.29.
+ ES/IFIC-LCG2 FT f./SOURCE:CONN.:SRM.GGUS 87865 solved:Disk server
  which caused failure was restarted.
- FR/RO-16-UAIC still FT f./SOURCE:INVALID_PATH:no such file or dir.
  GGUS 87629 assigned,updated:file doesn't exist at the site.
- US/Michigan Muon Calib DB replication delay:witer time not changed,
  experts informed.Elog.40457.
+ CA/CA-ALBERTA-WESTGRID-T2 500+ FT f./DEST.:CONN.:SRM.GGUS 87867
  solved:pool node damaged during the power outage,recovered power
  supply and rescued a pool node,Oct.29.
- UK/UKI-SCOTGRID-ECDF<-FZK FT f./GRIDFTP_ER.:server er.426:con.timeout.
  GGUS 87868 waiting for reply:issue on FZK side?
- DE/FZK-LCG2 100s t.101457,1016595 FT f./Get er.:staging input file
  failed.Savannah 133369,98536:asked to replicate missing files from
  other sites.

7) Mon Oct.29
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : FR,IT
        .......................................................
- FR/RO-16-UAIC still FT f./SOURCE:INVALID_PATH:no such file or dir.
  Savannah 133371:Still sonar FT f.due to missing files on DATADISK at
  the site.
- FR/RO-07-NIPNE 1000s FT f./DEST.:CONN.:SRM.GGUS 87885 assigned:also
  SOURCE f.Under investigation.
+ FR/IN2P3-CC 4K+\h Ft f./DEST.:CONN.:SRM.GGUS ALARM 87872 verified:
  restart of the SRM server helped the same day.Savannah 133378.
- DE/FZK-LCG2_MCTAPE->WUPPERTALPROD FT f./AGENT:CONF._ER.:No site found
  for host atlassrm-fzk.gridka.de.GGUS 87871 assigned:no anymore this
  error,file was successfully transferred the same day.
+ IT/INFN-MILANO-ATLASC FT f./SOURCE:CONN.:SRM.GGUS 87896 solved:DT,but
  but due to CEST to CET transition DT ended 1h earlier.SRM service has
  been restarted.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.5.2.1 data12_8TeV Reco merge task 1010497,1010537,1010805
 failures: AthenaPoolConverter ERROR poolToObject: caught error: FID
 "059E1C8C-1D2C-6646-AFC5-44053465F29A" is not existing in the catalog.
 Savannah 133215:group production EGAMMA.Many job failures up to 25
 attempts each at BNL,TRIUMF and CERN.Tasks are still "running" or in
 "submitted" state.No comments published so far.

-+ 17.2.3.7.1 data12_8TeV Reco merge tasks 1009507,1012203 failures:
 :Caught signal 11(Segmentation fault).No primary vertices reconstructed.
 Savannah 133172:group production Tau.All jobs failed multiple attempts
 at various sites running for CERN and US clouds.Other affected tasks:
 1006989,1008813,1009507,1009696,1011691,1011732,1012224,1012322,1012345,
 1012367,1012707,1012362. Tasks ABORTED:under investigation.Added NTUP_
 TAU and NTUP_TAUFULL creation and merging to the monitor to study this.

++ 17.0.6.4.11 data11_7TeV Reco merge task 1013580 failures:ATH_FAILURE -
 Athena non-zero exit.Savannah 133258:group production HIGGS.Other
 affected tasks:1013558,1013560,1013561,1013572,1013462,1013464,1013466,
 1013468,1013476,1013482,1013488,1013517,1013533,1013579.All jobs failed
 multiple attempts at various clouds.Wrong tag.All tasks ABORTED,this
 issue dealt with in a dpd ticket.

-+ 17.2.4.6.2 mc12_8TeV Reco merge task 961845 failures:Unknown dataset
 exception.Savannah 133319:group production SUSY.All jobs have failed up
 to 9 attempts each at 13 different sites running for ES cloud.Understood
 :the erased dataset resurrected.Many jobs in "transferring" soon after
 that.Task is in "submitting" state.

++ 17.2.2.6 mc12_8TeV simu task 1012647 failures:Put er:invalid argument
 lcg_cp.Savannah 133351:MC12 production - HIGGS.918 job failures at
 various sites in DE cloud.Task is still "running",no comments added.

-+ 17.3.4.3 valid1 DigiMReco task 1022744 failures:TRF_UNKNOWN |"module
 Type(eta): 5 unknown, using barrel" |"Could not create SiReadoutCellId
 object!".Savannah 133365:T1_McAtNlo_Jimmy.All jobs failed up to 10
 attempts at various sites running for US cloud.Task ABORTED.

-- 17.2.5.2.2 data12_8TeV Reco merge tasks 1023821-1023829 failures:TRF
 _UNKNOWN | "Standard std::exception is caught handling incident
 0xffe8185c" | "CaloCondBlobBase::getAddress: Index out of range: 33792.
 Savannah 133396:group production JetMET,SM WZ.Many jobs failed multiple
 attempts at CERN-PROD.Tasks are still "running" or in "submitted"
 state,no comments published so far.

-- many AOD->NTUP_HSG5ZMETU looping jobs (riing >2days) killed by pilot
 at CERN-PROD.Savannah 133328.No comments published since Oct.26.

-- LeslieGroer - 2012-10-29

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2012-10-30 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback