TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120403 (2012-04-03, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-04-03

General News

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


     World-wide weekly ATLAS ADCoS summary (Mar.27-Apr.2,2012)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Mar.27-Apr.2,2012) Panda production service
     - completed 1,654,955 managed group, MC production, validation
       and reprocessing jobs
     - average ~236,422 jobs per day
     - failed 148,869 jobs
     - average efficiency:
        -- jobs  ~91.1%
     - active tasks: 838
        -- distribution by cloud:
 CA:77 CERN:16 DE:90 ES:28 FR:212 IT:98 ND:53 NL:84 TW:34 UK:70 US:142

B)   - NL/SARA-MATRIX DT (SE,SRM warning:disk controller change), Mar.27.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:security updates),Mar.27.
     - ND/NDGF-T1 DT (SE,SRM warning:dCache upgrade on some nodes),Mar.28.
     - ND/NDGF-T1 DT (SE,SRM  warning:Maintenance fimm.bccs.uib.no),Apr.2.
     - UK/RAL-LCG2 DT (lcgce03 warning:replacing faulty disk,batch),Apr.2.
     - US/BNL FTS upgrade (~15-30 min.), Apr.2.
     - CERN:DQ2 0.1.37 client installed on Panda Monitor prod.nodes,Mar.28
     - CERN:ATLAS DDM Dashboard upgrade to 2.0 M3.3, Mar.30.
     - CERN:Central Catalogs update, Apr.3. Savannah 126879.
     - New LCG 62b,61e configuration released to AFS, Mar.28,29.
     - New DB Release 18.9.1 subscribed to sites, Mar.30.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    -------------------------------------------------------------------

2) Wed Mar.28
       ...................................................................
+ CERN/CERN-PROD->BNL FT f./GRIDFTP_ER.:server err.500.No route to host.
  GGUS 80702 solved:not BNL,but CERN related,as FT CERN->LYON,FZK has the
  similar failures,the list provided.There was a temporary problem at Cern
  12:19-16:43 Mar.28,no problems after.
+ US/SLACXRD FT f./gridftp_copy_wait:Conn.timed out.GGUS 80686 solved:
  network speed is just too slow at 106KB/s and the transfer exceeded 2h.
  limit.Networking problem between France and US is a known issue without
  good understanding.No major problem seen (for 24h),Apr.1.
+ DE/WUPPERTALPROD_PRODDISK FT f./GRIDFTP_ER.:server err.426:Conn.reset by
  peer.GGUS 80707 solved:LHCCONE network issue?Blacklisted,offlined:
  Savannah 127481.DT announced.Some issue with DT stop in GOC DB,Mar.30.
  Auto-whitelisted,set online after passing the tests,Apr.1.

3) Thu Mar.29
      ............................................
+ CERN/CERN-PROD_LOCALGROUPDISK still FT f./SOURCE file doesn't exist.GGUS
  80709 solved:known issue due to inconsistencies in the catalog.Is being
  fixed.
- CA/CA-MCGILL-CLUMEQ-T2 FT f./DEST.:SRM_FAILURE.GGUS 80715 in progress:FT
  succeeded in some time the same day,then went to DT:Mar.30-Apr.3.

4) Fri Mar.30
     -> Cloud/site DDM efficiency  <50% : FR,IT/INFN-T1
       ............................................
+ CERN/ATLAS-AMI-CERN in red/not available:experts informed.Elog 34756,72,
  82,91,92,95,96,98,808:seems a problem with Apache f/e load balancing in
  FR,submitted a ticket to CCIn2P3 support,Apr.1.
+ CERN/CERN-PROD job f./stage-out:WN->EOS writing err.GGUS 80765 solved:
  high error rate has gone done to ~5%, which seems to be acceptable since
  ATLAS is overloading the EOS SRM endpoint.ATLAS will move to direct
  "xrdcp" transfers for production jobs at CERN in the next days anyway,
  which should make this issue go away,Apr.2.

7) Mon Apr.2
       .......................................................
- CERN/ATLAD_DDM_DELETION monitoring SLS in grey (for 3h),Elog 34830,34832
  ATLAS_DDM_Tracker has the same problem.Experts informed.
- CERN/TOM heartbeat reappeared at 7:30pm.Reported to T0 ops.before.Elog.
  34815,34817,34829.
- CERN/DQ2 create dataset time problem,"spiking" since noon.Occasionally
  the DQ2 indicator in conTZole goes to red.Experts informed.Elog.34843.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
      (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.0.6.4.1 valid1 Reco merge tasks 751347-751354 failures:TRF_UNKNOWN |
 "Unable to build inputFileSummary from any of the specified input files.
 Savannah 93073:group production Higgs HSG2.All jobs failed up to 9
 attempts each in various clouds:DE,CERN,CA,FR.Tasks are in "submitted" or
 "failed" state.No comments added.

-+ 17.2.1.1 valid1 DigiMReco tasks 754957-755175 failures:TRF_UNKNOWN|
 "Failed toget valid AttributeList for channel number 957742633 (key Eta
 MassEnergyCorr_AntiKt_4_TopoTower)"|"Unable to build inputFileSummary
 from any of the specified input files".Savannah 93284:VALIDATION SampleA.
 All jobs failed multiple attempts at various clouds.Tasks ABORTED,no
 comments added.

++ 17.0.5.6.1 mc11_7TeV Reco merge task 751917 failures: TRF_UNKNOWN |
 "Unable to build inputFileSummary from any of the sp...Savannah 127483:
 group production EGAMMA.All jobs have failed repeatedly on the AOD->DPD
 making step at various sites in US cloud due to the error coming from the
 inputFilePeeker which can't open input AOD file.Similar to the bug above,
 but different release/cache,task type,etc.so goes to another Savannah
 tracker.Task ABORTED.Seen before:input produced with AtlasProduction-
 17.1.4.6 and the cache used in this task was AtlasPhysics-17.0.5.6.1.
 The cache to be checked and the request resubmitted.

-+ 17.0.6.4.7 mc11_7TeV Reco merge task 752538-750 failures:ATH_FAILURE -
 Athena non-zero exit.No keys among `GSFElectronAODCollection,GSFElectron
 Collection' for type `ElectronContainer' in ObjKeyStore.Savannah 127557:
 group production Higgs HSG2.All jobs failed multiple attempts in various
 clouds.Tasks ABORTED.

++ 17.2.0.4 valid1 Reco merge tasks 754084-754184 failures: RuntimeError:
 input AND output AOD is not supported.Savannah 93241,93249:SampleT
 production - r3459_r3460_p963.Understood:a mistake in the task definition
 .Tasks ABORTED and resubmitted.

-- LeslieGroer - 2012-04-02

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2012-04-03 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback