TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120710 (2012-07-10, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-07-10

General News

  • ADC and ATLAS Computing News [20120710]
ScreenShot012_copy.png
    • ADC Weekly meeting
    • High luminosity beam Wed/Thu but some issues since then
    • Run been extended 2 months - mid-Feb, followed by HI run
    • Overcoming CPU time issues (lost heartbeats) Rod Walker Slides
      • Some jobs violating limits (cputime, vmem, scratch disk)
      • Do batch systems send a soft kill (15) with enough time for pilot to cleanup?
        • kill athena process, create log tar (20s?), store log tar (60s?)
      • Proposal: Use pilot to provide better grace period
        • e.g. timer in pilot and kill before schedconfig.maxtime is up (assume this is published accurately)
      • Helps ProdSys identify issues (though does not save wasted cputime)
      • Don't send long jobs to short queues
        • broker etask.cpucount vs. schedconfig.maxtime
        • Complicated by distribution of job lengths in a task, different cores, hyperthreading - factor of 2?
        • Don't want to artificially limit access to resources
      • Base decisions on scout jobs and HS06 (but is this per node, per core, per slot?)
      • Proposal: update etask.cpucount at job avalanche time - implement this week
        • Ignore difference in cores and accept possible factor of 2 in cputime
      • Need sites to set MaxWCTime, MaxCPUTime (seconds) --> Schedconf.maxtime
      • Possible proposal: ATLAS asks for 48hrs limit on physical core --> site with full-HT should have 96hrs limit
      • Refined proposal:
        • Add HS06 per slot for schedconfig
        • Etask.cpucount: use site (avg from BDII) to get HS06s (per job slot)
        • Brokerage uses schedconfig.maxtime(s) * HS06_site > cpucount (start with HS06 = 1 for all sites)
        • Add HS06 rating to schedconfig
      • Second proposal: "Hospital" Queues
        • Provide a site or queue with high limits on vmem/cpu - at least one per cloud?
        • Tails of task job distribution may need higher limits
        • Identifying these jobs for the brokerage is still some work
    • MC Production - close to 1B mc12 fullsim and 400 AFT-II produced
      • 160M/57M events almost done to recover from Trigger Frontier bug
      • Anticipate at least another few weeks of high intensity requests - some groups delayed behind Higgs
    • Still some services and sites recovering from leap second bug last Monday
    • Increased critical LFC servers to three
    • NDGF LFC migrated to CERN successfully last Tuesday
    • DB release 9.6.1 subscribed to sites for urgent distribution Tue
    • Slides presented on Federated Xrootd Monitoring by Rob Gardner

  • Security News [20120207]
  • gLite/EMI News [20120605]
  • ROC_Canada News [20120710]
    • Current shift - Simon Nderitu 9-13 July, 2012
    • Next Week - ? Need new schedule

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

I. General summary:
    ---------------

A)  During the past week (July 3-9,2012) Panda production service
      - completed 1,377,154  managed group, MC production, validation
        and reprocessing jobs
      - average ~196,736 jobs per day
      - failed 125,315  jobs
      - average efficiency:
         -- jobs 91,7%
      - active tasks: 1,136
         -- distribution by cloud: CA:59 CERN:37 DE:126 ES:155 FR:122 IT:123 ND:58 NL:125 TW:29 UK:137 US:165

B)  - ND/NDGF-T1 DT (CE,SE,SRM warning:maintenance at DeiC/KU), Jul.2-3.
     - NL/SARA-MATRIX DT(SE,SRM warning:restarting Postgres on SRM),Jul.3
     - ES/PIC DT (CE,SE,SRM,LFC OUTAGE:core network upgrade), Jul.4.
     - TW/Taiwan-LCG2 DT (CE,SE,SRM,BDII warning:10G network m.), Jul.3-4
     - ND/NDGF-T1 DT (SE,SRM warning:rolling pool upgrade), Jul.4-6.
     - NL/SARA-MATRIX DT (SE,SRM warning:SRM instability), Jul.4-6.
     - IT/INFN-T1 DT (SE,SRM OUTAGE:file system check), Jul.9-11.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:s/w updates on head nodes), Jul.9.

     - New DB release 19.6.1 subscribed to sites, Jul.3.
     - New trf caches 17.0.6.16, 17.2.3.6.2, 17.0.6.4.13, 17.2.4.1,
       17.2.3.7.1, 17.2.3.3.3 were distributed during the week.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Jul.3
      -> Cloud/site DDM  efficiency <50% : CERN
        ........................................

2) Wed Jul.4
      -> Cloud/site DDM  efficiency <50% : NL/SARA, UK/QMUL
        ................................................................
- FR/BEIJING-LCG2_LOCALGROUPDISK many FT f./DEST.:NO_SPACE_LEFT.GGUS
  83855 verified:not a site admin.issue.Blacklisted in DDM,Savannah
  130108:FR cloud should acknownledge the ticket.

3) Thu Jul.5
      -> Cloud/site DDM efficiency <50% : NL/SARA
        ............................................

4) Fri Jul.6
        ............................................

5) Sat Jul.7
      -> Cloud/site PROD efficiency ever<50%:DE/FZK+HEPHY;US/BNL-CLOUD
      -> Cloud/site DDM  efficiency <50% : -
        ..............................

6) Sun Jul.8
        ........................................
- CERN/SLS Panda availability 50%.Elog.37403.Under investigation.

7) Mon Jul.9
        .......................................................
+ CA/CA-SCINET-T2 1.4k job f./lost hearbeat.GGUS 83961 solved:The data
  centre lost power so all jobs were lost.Power is restored and all
  services are back online.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.3.4 mc12_8TeV evgen task 892468,892467 failures: TRF_UNKNOWN |
 "EFFICIENCY IS TOO LOW! At least: 98% expected" | "Error in Algorithm
 Finalize" and TRF_UNKNOWN |"!!ALPGEN TERMINATES NORMALLY: NO MORE EVENTS
 IN FILE !!!"|"Terminating event processing loop due to errors" and
 TRF_INFILE_TOOFEW, 64031, Transform input file contains too few events.
 Savannah 130104,130170:MC12 production - HIGGS.>6400+13000 jobs failed
 repeatedly at various sites in US,DE clouds.Task are still "running".No
 comments published so far.

-+ 17.2.1.4 valid1 DigiMReco tasks 900532,33,48 failures:ATH_FAILURE -
 Athena non-zero exit.IndexError:list index out of range.Savannah 95879:
 VALIDATION ISF SampleA r3714.All jobs failed multiple attempts at
 different sites in FR cloud.Tasks ABORTED,no comments added.

-- 17.2.2.6.2 mc10_14TeV digi NTUP FTK task 900649 failures:Error details
 :Transformation not installed in CE.Savannah 130167:all jobs have failed
 4 attempts in a row each at CERN-RELEASE because they can't find the
 path to the transformation.The task is in "failed" state now as it seems
 to reach the maxAttempt.In accordance to ATLAS InstallDB the
 corresponding trf chache 17.2.2.6.2-trigmc is installed at CERN-PROD
 CVMFS area.

-+ 16.6.9.12 mc11_7TeV evgen 904146 failures:EFFICIENCY IS TOO LOW! At
 least: 98% expected | "Error in Algorithm Finalize".Savannah 130192:MC11
 production - SM.All jobs failed up to 7 attempts in various sites
 running for ES cloud.Task ABORTED,no comments added.

++ 17.2.1.4 mc12_8TeV DigiMReco task 874887 failures:GUID for HITS.797138
 ._002930.pool.root not found in DQ2.Savannah 129862,94647:MC12
 production - PHYSICS.6 jobs keep failing at UK cloud due to missing
 input files.Task set to "finished".Files declared lost at RAL.

-- LeslieGroer - 2012-07-09

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng DDM.png r1 manage 108.1 K 2012-07-10 - 19:09 LeslieGroer  
PNGpng ScreenShot012_copy.png r1 manage 79.6 K 2012-07-09 - 14:56 LeslieGroer  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2012-07-10 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback