TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120320 (2012-03-20, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-03-20

General News

  • ADC and ATLAS Computing News [20120320]
    • ADC Weekly meeting
    • LHC ramped up to 4TeV Friday last week
    • 900 GeV collisions expected Wednesday
    • Full data taking at 8TeV in earnest expected to begin April 1
    • Comp@P1 - most shifts covered (56 shifters)
    • Harder time filling AMOD/ADCOS shifts (14/12 shifters currently, training 4/3 more)
    • Hammercloud machine moved to better hardware (4 core, 16GB RAM) and mysql updated. Did not affect sites.
    • One hour intervention in production next week. Put Panda on hold. New naming convention for task.
    • LFC migration in the French cloud (20 Mar)
    • Fast Tracker (FTK)
      • Aim: Reconstructs tracks in hardware after L1 trigger in < 100us and propagate to L2
      • https://twiki.cern.ch/twiki/bin/viewauth/Atlas/FastTracKer
      • Need to produce lots of studies this year for LOI for upgrade - in production by Fall
      • Use prodsys mechanisms to split up the large jobs for processing on the grid
      • Generate FTK input files from RDO
        • Simulate the FTK response
        • Need to slice detector into segments for job scaling (~128)
        • Each segment needs merging at the end
        • Reconstruct events, using FTK simulation inputs

ScreenShot057.png

* CA Cloud Squad Actions [20120228]

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


     World-wide weekly ATLAS ADCoS summary (Mar.13-19,2012)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Mar.13-19,2012) Panda production service
     - completed 1,322,660 managed group, MC production, validation and reprocessing jobs
     - average ~188,951 jobs per day
     - failed 53,152 jobs
     - average efficiency:
        -- jobs  ~96.1%
     - active tasks: 388
        -- distribution by cloud: CA:11 CERN:26 DE:89 ES:17 FR:57 IT:6 ND:38 NL:22 TW:5 UK:62 US:55

B)   - DE/FZK-LCG2 DT (LFC,SE,SRM,CE,BDII OUTAGE:maintenance), Mar.12-15.
     - UK/RAL-LCG2 DT (FTS OUTAGE:problems with b/e DB ~3.5h), Mar.14.
     - UK/RAL-LCG2 DT (LFC,SE,SRM OUTAGE:network problems), Mar.16.
     - FR/IN2P3 DT (LFC OUTAGE:LFC migration), Mar.20-21.
     - New DB Release 18.7.1 subscribed to sites, Mar.17.
     - New pilot version v52a announced and released, Mar.16,19.
     - HammerCloud intervention for DB uptate (~2h.), Mar.20.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    -------------------------------------------------------------------

4) Fri Mar.16
     -> Cloud/site PROD efficiency ever<50% : -
     -> Cloud/site DDM efficiency  <50% : DE,CERN
       ............................................
+ CERN/SLS ATLAS-Frontier availability 86% due to ATLAS-KIT-Frontier not
  available.Savannah 92676:Oracle DB at FZK fixed,FTS+Frontier are back.
  Availability is back to 100% as of Mar.16,the ticket can be closed.
- DE/FZK-LCG2_DATADISK low(25%) success for FT/SOURCE:GENERAL_FAILURE:
  RQueued.Elog 34396.Related to FTS DB issue at FZK?See Savannah 92676.

6) Sun Mar.18
     -> Cloud/site PROD efficiency ever<50% : -
     -> Cloud/site DDM  efficiency < 50%: DE
       ........................................
+ CERN/ATONR,ATLR DB monitor yellow/blocking+waiting session.Elog.34424,41
  Recovered.
+ CERN/AGIS error 503,service temp.unavailable 4:00-5:54.Restarted,fixed.
  Elog 34416,18-19.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
      (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------


++ 17.1.4.6 valid1 DigiMReco task 736292 crashes: TRF_SEGFAULT | "Caught
 signal 11(Segmentation fault).Savannah 92526:SampleT production - r3373.
 All jobs crashed up to 6-7 attempts in a row each at various sites in CA
 cloud due to the reproducible error on the same event (RAWtoESD step).
 Task ABORTED.Should be fixed for release 17.1.4.7(or 17.2.0.1).

++ 17.0.6.4.5 mc11_7TeV Reco merge tasks 741421 failures:Too many/too
 large input files.Group production TRIGGER L1CALO.Othe affected tasks:
 741406-14,16,18,19,21,43,47,449,549,571,575,577.All jobs failed multiple
 attempts in various clouds.Elog.31411-3.Understood:wrong task definition,
 tasks ABORTED,to be redefined.

-- LeslieGroer - 2012-03-19

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng ScreenShot013.png r1 manage 42.4 K 2012-03-20 - 16:00 LeslieGroer  
PNGpng ScreenShot056.png r1 manage 216.6 K 2012-03-20 - 15:58 LeslieGroer  
PNGpng ScreenShot057.png r1 manage 141.7 K 2012-03-20 - 16:01 LeslieGroer  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2012-03-20 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback