TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120228 (2012-02-28, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-02-28

General News

  • ADC and ATLAS Computing News [20120228]
    • No ADC Meeting this week but AMOD notes ADC Weekly meeting
    • Not much production runnig Feb 24-25
    • SSB autoexclusions enabled
      • Small glitch with RAL but fixed now.
    • PFT enabled for DE cloud.
      • Few sites auto-excluded already. Enabling other clouds this week (TW - yesterday, IT, FR - today)
    • CERN pilot factories down on Feb 22 after CondorG upgrade
    • New pilot version 50d

  • CA Cloud Squad Actions [20120228]
    • User DaTRI requests approved / processed (last week)
SITE NumRequests NumDatasets NumSubscribed SizeofAllDatasets (GB)
AUSTRALIA-ATLAS-LOCALGROUPDISK 0 0 0 0
AUSTRALIA-ATLAS_SCRATCHDISK 8 255 256 198
AUSTRALIA-ATLAS_PHYS-SM 1 7 7 130
CA-ALBERTA-WESTGRID-T2_SCRATCHDISK 0 0 0 0
CA-MCGILL-CLUMEQ-T2_SCRATCHDISK 0 0 0 0
CA-MCGILL-CLUMEQ-T2_LOCALGROUPDISK 0 0 0 0
CA-SCINET-T2_LOCALGROUPDISK 5 26 25 54
CA-SCINET-T2_PHYS-TOP 0 0 0 0
CA-SCINET-T2_SCRATCHDISK 1 88 88 70
CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK 0 0 0 0
CA-VICTORIA-WESTGRID-T2_PHYS_EXOTICS 0 0 0 0
CA-VICTORIA-WESTGRID-T2_SCRATCHDISK 0 0 0 0
SFU-LCG2_LOCALGROUPDISK 0 0 0 0
SFU-LCG2_PHYS-SM 0 0 0 0
SFU-LCG2_SCRATCHDISK 0 0 0 0
TRIUMF-LCG2_LOCALGROUPDISK 1 1 1 1
TRIUMF-LCG2_SCRATCHDISK 0 0 0 0
TRIUMF-LCG2_PERF-TAU 0 0 0 0
    • Declared Downtimes
      • SFU-LCG2/ SCHEDULED downtime (OUTAGE) [19-02-2012 09:00 to 22-02-2012 03:00 UTC] - The main power cable to the whole building will be replaced.
      • SFU-LCG2 UNSCHEDULED downtime (OUTAGE) [22-02-2012 03:00 to 22-02-2012 23:00 UTC] - The power from the previous downtime is not on yet. Extending the downtime
      • SFU-LCG2/ UNSCHEDULED downtime (WARNING) [22-02-2012 05:30 to 22-02-2012 05:45 UTC] - The SE is back. Both CEs take more time to upgrade after the downtime.
      • TRIUMF-LCG2 SCHEDULED downtime (OUTAGE) [29-02-2012 22:00 to 29-02-2012 23:00 UTC ] - Linux kernel update in Oracle RAC server which hosts FTS database
    • HammerCloud autoexclusions
      • ANALY_ALBERTA-WG1 Auto-Excluded at Wed Feb 22 16:31:26 2012 CET
      • ANALY_ALBERTA-WG1 reset online at Wed Feb 22 19:30:16 2012 CET
      • ANALY_SFU reset online at Wed Feb 22 20:00:13 2012 CET
      • ANALY_SCINET Auto-Excluded at Thu Feb 23 18:32:03 2012 CET
      • ANALY_SCINET reset online at Thu Feb 23 20:00:18 2012 CET
      • ANALY_SFU_bugaboo reset online at Thu Feb 23 22:30:18 2012 CET
      • ANALY_SCINET Auto-Excluded at Fri Feb 24 23:32:47 2012 CET
      • ANALY_SCINET reset online at Sat Feb 25 03:00:09 2012 CET
    • other cloud squad related savannah and / or ROC ggus tickets [20110705]
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
  • Open Issues and Ongoing Discussions

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

panda_ssb_prod_20120228.png
panda_ssb_ana_20120228.png

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


     World-wide weekly ATLAS ADCoS summary (Feb.21-27,2012)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Feb.21-27,2012) Panda production service
     - completed 696,959 managed group, MC production, validation
       and reprocessing jobs
     - average ~99,566 jobs per day
     - failed 74,955 jobs
     - average efficiency:
        -- jobs  ~90.3%
     - active tasks: 1,032
        -- distribution by cloud:
  CA:64 CERN:55 DE:175 ES:25 FR:210 IT:67 ND:33 NL:86 TW:20 UK:97 US:200

B)
     - US/BNL ORACLE DT (Cond DB OUTAGE;h/w migration+11gR2 upgrade),Feb.22
     - UK/RAL-LGC2 DT(SE,SRM OUTAGE:CASTOR ATLAS update to 2.1.11-8),Feb.22
     - CERN/CERN-PROD DT (CE,LFC,SE,SRM OUTAGE:CASTORPUBLIC update),Feb.22
     - CERN/CERN-PROD DT (LFC,VOMS,FTS OUTAGE:LCGR DB DT ~30min.), Feb.27.
     - New DB Releases 18.1.1,18.2.1 subscribed to sites, Feb.22,23.
     - New pilot version 50d released, Feb.27.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    -------------------------------------------------------------------

1) Tue Feb.21
       ........................................
+ DE/FZK->CA-SCINET-T2 100% FT  f./GRIDFTP_ER.:server err.426.GGUS 79407
  verified:One of the BGP policies at Internet2/MANLAN got truncated which
  caused them to stop tagging the routes from GEANT correctly.This was
  fixed and Toronto is receiving the European routes again through the
  VLAN 2001.

2) Wed Feb.22
       ...................................................................
- CERN/pilot factory troubles,voatlas213-15.Started temporarily another
  factory (whuppertal) to cover missing pilots.In contact with CERN
  PF.Elog 34000.

3) Thu Feb.23
       ............................................

4) Fri Feb.24
       ............................................

5) Sat Feb.25
       ..............................

6) Sun Feb.26
       ........................................
- CA/CA-SCINET FT f./GRIDFTP_ER.:server err.500:conn.timeout.GGUS 79651
  assigned:perhaps due to asymmetric connections to LHCONE(GGUS:79407)?

7) Mon Feb.27
       .......................................................

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
      (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+  16.6.9.2 mc11_7TeV evgen task 715095 failures:Job killed by signal 15:
 Signal handler has set job result to FAILED.Savannah 126649:MC11
 production - SUSY.All jobs were killed repeatedly by the system at
 different sites in US region because they exceeded the CPU/wall time limit
 of 2 days (in CA) or even 3 days (in US).Task ABORTED.

-+ 17.0.6.2 mc11_7TeV DigiMReco tasks 724726-725033 SyntaxError: invalid
 syntax ."an unknown exception occurred".Savannah 91815,126585:MC11
 production - JETETMISS.All jobs failed multiple attempts at various
 clouds due to the reproducible error on the ESDtoAOD step.Tasks ABORTED.

-- 17.1.3.2 data11_7TeV Reco task 712903 failures:TRF_SEGFAULT |Duplicate
 request for folder /TRIGGER/LVL1/BunchGroupCont.Savannah 126646:group
 production TRIGGER HLT.All jobs failed up to 5 attemptps in a row each at
 CERN-RELEASE due to the reproducible error on the ESTtoAOD step.Task is
 in "failed" state.

++ 17.1.4.1 valid1 reco task 729177-9,729324 failures:Wrong DBRelease
 version:17.6.2.Need at least version 18.0.1.Savannah 91919:SampleT
 production - r3270.Many other tasks from tid 728128-729345 affected.Tasks
 ABORTED,new ones resbmitted using a later DBRelease.

++ 17.1.4.1 valid1 DigiMReco tasks 729454-729587 failures:TRF_UNKNOWN |Tag
 OFLCOND-SDR-BS7T-05-24 cannot be resolved for folder /CALO/JetCalib/EtaMass
 EnergyCorr".Savannah 91930,91679:T1_McAtNlo_Jimmy.All jobs failed
 repeatedly at various sites/clouds.Tasks ABORTED,suggested to use the
 latest DB Release and AtlasProduction 17.1.4.2.

-+ 17.1.4.1 valid1 Reco tasks 730158-730257 failures:PIXEL CL NOT FOUND IN
 SP id=.../received fatal signal 11(Segmentation fault).Savannah 91934:
 SampleT production - r3280.Large number of jobs/tasks failed in different
 clouds.Tasks ABORTED.No comments added.

++ 17.0.6.2.2 mc11_2TeV Digi tasks 730796,730801,730794,730804 failures:
 TRF_UNKNOWN|"out of memory: saving summary ..."|"commitOutput - caught
 exception:std::bad_alloc".Savannah 126671,91947:MC11 production -
 HEAVYION.All jobs failed repeatedly in NL,CA,DE clouds.T.730816,730829,
 730817,730810,730808,730812,730823,730806 (JETETMISS) affected too.Tasks
 ABORTED.Understood - too large number of events per job,resubmitted with
 a smaller number.

-- 17.0.6.2 mc11_7TeV DigiMReco task 728653,728111,728296,728276,728380,
 728358,728318 failures:TRF_UNKNOWN | 'poolToObject: caught error: FID
 "EA8EA4BE-7147-E111-AD30-782BCBCD72A9" is not existing in the catalog (
 POOL:.Savannah 91943:MC11 production - JETETMISS.Massive job failures at
 CERN and in DE cloud.Tasks are still "running".VO_ATLAS_SW_DIR should be
 changed(?),experts informed.

-- LeslieGroer - 2012-02-28

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng panda_ssb_ana_20120228.png r1 manage 163.6 K 2012-02-28 - 18:56 LeslieGroer  
PNGpng panda_ssb_prod_20120228.png r1 manage 71.8 K 2012-02-28 - 18:56 LeslieGroer  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-02-28 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback