TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120605 (2012-06-05, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-06-05

General News

  • ADC and ATLAS Computing News [20120605]
    • Lot of pressure to get tasks completed for ICHEP
      • Special Higgs group production (Wgamma Np0,3,4,5) essentially done
      • Request for ZZ->4l being considered
      • Some discussion about whether this XP can be accommodated in ProdSys
    • Site Availabiltiy stats
site_avail.png
      • CA-ALBERTA-WESTGRID-T2 SCHEDULED downtime (WARNING) [08-05-2012 16:30 to 08-05-2012 23:00 UTC ] - IB switch maintenance (2nd phase)
      • CA-ALBERTA-WESTGRID-T2 SCHEDULED downtime (OUTAGE) [17-05-2012 17:30 to 18-05-2012 23:30 UTC ] - IB switch maintanence 3rd phase
      • CA-SCINET-T2 SCHEDULED downtime (OUTAGE) [07-05-2012 13:30 to 09-05-2012 23:59 UTC] - Reconfiguring site for IB communication
      • CA-SCINET-T2/ UNSCHEDULED downtime (OUTAGE) [10-05-2012 00:00 to 10-05-2012 18:00 UTC] - Extension of downtime for network reconfiguration.
    • ADC Weekly meeting
    • About 0.85 fb^-1 collected last week
    • Production 90-100k running, analysis 25-30k (spikes to 45k)
    • Slow transfers from IN2P3 to TRIUMF (lasted about 10 days) traced to two slow data movers in France with recent Java updates
    • Pandamon alias switch (May 30th)
      • panda.cern.ch DNS alias changed from pandamonitor to pandalb
    • Some jobs affected by confusion in central services for data subscribed to SCRATCHDISK preventing replication to DATADISK
    • SARA stills sees SRM issues concurrent with large deletions (e.g. from LHCB)
    • Comments from Stephane about issue with migrating T2 SE's to new platforms. Need a bit more advanced planning perhaps?
    • Squads asked to monitor the DDM subscriptions, especially for DATADISK:
DDM.png

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (May 29 - June 4,2012)

I. General summary:
    ---------------

A)  During the past week (May 29 - June 4,2012) Panda production service
      - completed 1,560,433  managed group, MC production, validation
        and reprocessing jobs
      - average ~222,920 jobs per day
      - failed 70,159 jobs
      - average efficiency:
         -- jobs 95.7%
      - active tasks: 1056
         -- distribution by cloud: CA:100 CERN:90 DE:152 ES:56 FR:114 IT:92 ND:63 NL:66 TW:71 UK:130 US:122

B)   - NL/SARA DT (BDII,CE,SE,SRM,LFC OUTAGE:network issues), May 29.
     - NL/SARA DT (SE,SRM OUTAGE:switch emergency reboot), May 30.
     - NL/SARA DT (SE,SRM warning:SAN storage maintenance), May 31.
     - NL/SARA DT (BDII,CE,SE,SRM,LFC OUTAGE:namespace DB,network),Jun.4
     - IT/INFN-T1 DT (CE,SE,SRM OUTAGE:earthquake issues) May 29-30.
     - IT/INFN-T1 DT (CE,LFC,SE,SRM warning:network at risk), May 31.
     - CA/TRIUMF-LCG2 DT (FTS,Frontier OUTAGE:add DB storage), May 29.
     - TW/Taiwan-LCG2 DT (BDII,CE,SE,SRM OUTAGE:switch OS+h/w upgrade,
          CASTOR DB table index reconstruction), May 31,June 4.
     - TW/Taiwan-LCG2 DT (BDII,CE,SE,SRM OUTAGE:OS+h/w upgrade), Jun.4-5.
     - NDGF-T1 DT (SE,SRM waring:OPN down to Bergen,Norway), Jun.3-4.
     - Urgent group production tasks announced: Elog 36588, Jun.3.
     - New pilot version v 53c announced, Jun.4.

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue May 29
        ........................................
+ CERN/AMI A mistake was made in deploying a new release,the project tag
  validation was broken causing many jobs to fail.Fixed.Elog 36454-55,61.
+ FR/IN2P3-CC<->TRIUMF very slow transfer rate.GGUS 82363 updated,solved:
  identified 2 machines with low transfer(ccdcatsn27{7,9})to be checked.
  Disabled the pools on these machines last Saturday.Java upgrade had
  been deployed on them,Jun.4.

2) Wed May 30
      -> Cloud/site DDM  efficiency <50% : NL/TECHNION-HEP,US/UTA_SWT2
        .................................................................
+ US/UTA_SWT2 FT f./DEST.+SOURCE:SECURITY_ER.:Robot cert expired.GGUS
  82676 solved:also ~1K|4h deletion err./SRM_AUTHORIZATION_FAILURE.A
  host certificate has expired on a machine at UTA-SWT2,replaced.
- US/BU-ATLAS-T2_PRODDISK ~430|4h deletion errors.GGUS 80214 reopened:
  200|4h err.,May 31.
+ CA/CA-SCINET-T2_DATADISK FT f./SOURCE:SECURITY_ER:proxy expired.GGUS
  82690 verified:problem seems to be gone,May 31.

3) Thu May 31
        ............................................
- CA/CA-MCGILL-CLUMEQ-T2 All FT f./DEST.:CONN.:SRM.GGUS 82756 in progress
  :network issues between TRIUMF and McGill possibly caused by TRIUMF
  edge router.Many job f./Put er.:registration failed.Jun.2.

4) Fri Jun 1
      -> Cloud/site DDM efficiency  <50% : NL
        ............................................

5) Sat Jun 2
      -> Cloud/site DDM  efficiency <50% : NL/SARA
        ..............................

6) Sun Jun 3
        ........................................

7) Mon Jun 4
        .......................................................


III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+ 17.2.2.5 mc12_8TeV evgen task 827548 failures:TRF_UNKNOWN |"!! ALPGEN
 TERMINATES NORMALLY:NO MORE EVENTS IN FILE".Savannah 94891,129140:MC12
 production - HIGGS.Many jobs failed at different sites in US cloud.
 Submitted with the new JO (20 events/job),but it's clearly still causing
 some trouble.The same problem with t.827550,51,53,55,60,65,67 in various
 clouds:1000's of failed jobs,each having run for ~1-2 days before
 crashing and being retried many times.Tasks set to "finished",but
 t.827565,67 are still "running".The same issue with t.846555:50% failure
 rate, in "running" state.


-+ 17.3.2.1 valid1 DigiMReco tasks 846246,846301 failures:AttributeError:
 '<class 'AthenaCommon.AlgSequence.AlgSequence'>' object has no attribute
 'PixelDigitization'.Savannah 95025:T1_McAtNlo_Jimmy.All jobs have
 crashed at various sites in US cloud due to the error on digitization
 step.Tasks ABORTED,no comments published.

-- LeslieGroer - 2012-06-05

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng DDM.png r1 manage 108.1 K 2012-06-05 - 16:16 LeslieGroer  
PNGpng Screenshot-Site_Status_Board_-_Mozilla_Firefox.png r1 manage 161.5 K 2012-06-05 - 16:17 LeslieGroer  
PNGpng site_avail.png r1 manage 80.2 K 2012-06-05 - 17:00 LeslieGroer  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-06-05 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback