TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120904 (2012-09-04, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-09-04

General News

  • ADC and ATLAS Computing News [20120904]
    • New request for another 1M XP_Higgs events - about 9k cpu-days
    • ADC Weekly meeting
    • 0.9 fb^-1 collected last week (34% duty factor), peak luminosity 7.3x10^33cm^-2s^-1, Total delivered luminosity 13.7fb^-1
    • Xmas shutdown starts Dec 17, 2012
    • Cleanup of T1 disk space
      • T1-T1 Data consolidation halted
      • Reduction of mc11%AOD%tid%, data11%AOD%tid% on T1_DATADISK (from 2 to 1 primary copy) after decision at the last CREM meeting
      • Deletion of unmerged HITS, unmerged AOD when merged ones were available. Freed a few 100 TB on each T1
      • Identification of group datasets still on T1_DATADISK because they had no destination. Groups will be asked to do some cleanup
      • Lifetime of ESD data on disk. Final check with physics community. Aim is to go from 6 to 4 weeks.
    • Proposal to switch ToACache to the one generated by AGIS https://indico.cern.ch/getFile.py/access?contribId=9&resId=0&materialId=slides&confId=206068
      • 100% identical to DDM ToACache
      • Web UI under test
      • Rollback is an issue as information is cached for ~ day
      • Testing pilots still need to be completed
      • Propose test period in a couple of weeks for all clients and then commit to this
    • Discussion about deletion of GROUP datasets https://indico.cern.ch/getFile.py/access?contribId=6&resId=0&materialId=slides&confId=206068
      • Deletion of "old/buggy" (i.e. obsolete) datasets to free up space.
      • Deletion of "intermediate" datasets - frees up space in T1 DATADISK (2PB!). Transient datasets that were subsequently merged and replicated to GROUPDISK.
        • Send out alerts two weeks prior. Could affect user subscribed data on LOCALGROUPDISK as all copies will get deleted.
        • Confer with physics groups and coordinators
    • Report from French sites for T2D status https://indico.cern.ch/getFile.py/access?subContId=0&contribId=5&resId=0&materialId=0&confId=206068
      • Slow transfers from most FR sites (5 in France + BEIJING-LCG2) to/from TRIUMF mentioned - none achieve 5 MB/s.
      • Do see improvements in bandwidth with sites joining LHCONE.
      • Note that SONAR tests performance affected when IN2P3-CC is very busy
FRsites_SONAR_TRIUMF.png
  • CA Cloud Squad Actions [20120626]
    • User DaTRI requests approved / processed (last week)
    • Declared Downtimes
      • CA-VICTORIA-WESTGRID-T2 SCHEDULED downtime (WARNING) [29-08-2012 14:00 to 29-08-2012 15:00 UTC] - Short (5-10min) network interruption, to switch to backup path for UVic - VicTX.
      • TRIUMF-LCG2 SCHEDULED downtime (OUTAGE) [04-09-2012 17:00 to 04-09-2012 21:00 UTC ] - dCache upgrade to 1.9.12 golden release
      • CA-VICTORIA-WESTGRID-T2 UNSCHEDULED downtime (OUTAGE) [29-08-2012 16:00 to 29-08-2012 20:00 UTC] - site BDII problem
      • CA-VICTORIA-WESTGRID-T2 UNSCHEDULED downtime (OUTAGE) [29-08-2012 20:00 to 29-08-2012 21:00 UTC] - site BDII problem
      • CA-VICTORIA-WESTGRID-T2 UNSCHEDULED downtime (WARNING) [29-08-2012 21:00 to 30-08-2012 23:59 UTC - Research network bandwidth seems degraded, probably related to recent fiber switchover.
      • TRIUMF-LCG2 SCHEDULED downtime (OUTAGE) [05-09-2012 17:00 to 05-09-2012 20:00 UTC ] - dCache head nodes migration to new hardware
      • CA-MCGILL-CLUMEQ-T2 SCHEDULED downtime (OUTAGE) [01-09-2012 23:00 to 02-09-2012 23:00 UTC ] - Issue with the SE at lcg-cr. We are investigating
      • CA-MCGILL-CLUMEQ-T2 UNSCHEDULED downtime (OUTAGE) [31-08-2012 23:00 to 01-09-2012 23:00 UTC ] - Issue with the SE at lcg-cr. We are investigating
      • TRIUMF-LCG2 SCHEDULED downtime (OUTAGE) [04-09-2012 06:00 to 04-09-2012 08:00 UTC ] - Final MySQL tables export in preparation for service migration to CERN
      • CA-VICTORIA-WESTGRID-T2 SCHEDULED downtime (OUTAGE) [04-09-2012 17:00 to 04-09-2012 23:00 UTC ] - maintenance during LFC migration
      • CA-SCINET-T2 SCHEDULED downtime (WARNING) [04-09-2012 16:00 to 04-09-2012 22:00 UTC ] - Service disruptions while upgrades from 1Gbps to 10Gbps NICs on storage and other nodes
      • CA-SCINET-T2 SCHEDULED downtime (OUTAGE) [06-09-2012 02:00 to 06-09-2012 06:00 UTC ] - Main campus router software upgrade. Datacentre will be inaccessible during this period (though jobs will continue to run).
      • CA-MCGILL-CLUMEQ-T2 SCHEDULED downtime (OUTAGE) [01-09-2012 23:00 to 02-09-2012 23:00 UTC] - Issue with the SE at lcg-cr. We are investigating
      • CA-MCGILL-CLUMEQ-T2 SCHEDULED downtime (OUTAGE) [02-09-2012 23:01 to 03-09-2012 23:02 UTC] - Issue with the SE at lcg-cr. We are investigating
      • Australia-ATLAS SCHEDULED downtime (OUTAGE) [04-09-2012 05:00 to 04-09-2012 10:00 UTC ] - Compute host upgrade and rebuild
    • HammerCloud autoexclusions 
ANALY_MCGILL Auto-Excluded at Tue Aug 28 18:31:31 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Tue Aug 28 18:34:10 2012 CE
ANALY_MCGILL reset online at Tue Aug 28 23:00:13 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Tue Aug 28 23:03:02 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Wed Aug 29 09:09:51 2012 CET
ANALY_AUSTRALIA reset online at Wed Aug 29 10:00:48 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Wed Aug 29 13:35:04 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Wed Aug 29 14:05:04 2012 CET
IAAS Auto-Excluded at Wed Aug 29 19:46:52 2012 CET
IAAS reset online at Wed Aug 29 23:38:21 2012 CET
IAAS Auto-Excluded at Thu Aug 30 02:53:29 2012 CET
ANALY_MCGILL Auto-Excluded at Thu Aug 30 15:44:13 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Thu Aug 30 16:16:11 2012 CET
ANALY_MCGILL reset online at Thu Aug 30 17:01:12 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Thu Aug 30 17:03:01 2012 CET
ANALY_ALBERTA-WG1 Auto-Excluded at Thu Aug 30 22:36:05 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Thu Aug 30 22:39:02 2012 CET
ANALY_MCGILL Auto-Excluded at Thu Aug 30 23:07:11 2012 CET
ANALY_ALBERTA-WG1 reset online at Fri Aug 31 02:02:06 2012 CET
IAAS reset online at Fri Aug 31 03:04:16 2012 CET
CA-ALBERTA-WESTGRID-T2 Auto-Excluded at Fri Aug 31 19:03:21 2012 CET
CA-ALBERTA-WESTGRID-T2 reset online at Fri Aug 31 20:02:19 2012 CET
ANALY_VICTORIA-WG1 Auto-Excluded at Sat Sep  1 01:08:18 2012 CET
IAAS Auto-Excluded at Sat Sep  1 01:12:38 2012 CET
CA-VICTORIA-WESTGRID-T2 Auto-Excluded at Sat Sep  1 01:46:13 2012 CET
ANALY_VICTORIA-WG1 reset online at Sat Sep  1 02:01:48 2012 CET
CA-VICTORIA-WESTGRID-T2 reset online at Sat Sep  1 02:16:05 2012 CET
IAAS reset online at Sat Sep 1 02:16:07 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Sat Sep  1 05:08:33 2012 CET
ANALY_MCGILL reset online at Sun Sep  2 07:30:08 2012 CET
ANALY_MCGILL Auto-Excluded at Sun Sep  2 08:11:31 2012 CET
ANALY_MCGILL reset online at Sun Sep  2 11:31:54 2012 CET
ANALY_MCGILL reset online at Mon Sep  3 14:30:22 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Mon Sep  3 15:04:33 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Mon Sep 3 20:01:48 2012 CET

  • Security News [20120801]
  • gLite/EMI News [20120801]
    • EGI CSIRT ADVISORY [EGI-ADV-20120801] concerning gLite 3.2 middleware components no longer supported on 01 August 2012.

  • ROC_Canada News [20120710]
    • Current shift - Neil Knecht Sep 3-7, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

      World-wide weekly ATLAS ADCoS summary (Aug.28-Sep.3,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Aug.28-Sep.3,2012) Panda production service
      - completed successfully 1,402,474 managed group, MC production,
        validation and reprocessing jobs
      - average ~200,353 jobs per day
      - failed 63,648 jobs
      - average efficiency:
         -- jobs ~95,7%
      - active tasks: 1140
         -- distribution by cloud:  CA:55 CERN:83 DE:124 ES:63 FR:157 IT:118 ND:31 NL:145 TW:115 UK:125 US:124

B)   - TW/Taiwan-LCG2 DT (SE,SRM OUTAGE:DPM storage upgrade), Aug.28.
     - CERN/CERN-PROD DT (SE,SRM OUTAGE:EOSATLAS update to 0.2), Aug.29.
     - ND/NDGF-T1 DT(SE,SRM warning:some data might be unavailable),
       Aug.31,Sep.1,2,3.
     - New TRF caches were distributed:
       -- AtlasProduction 17.2.4.10 (32+64b)
       -- TrigMC 17.2.1.4.3 (32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Aug.28
      -> Cloud/site PROD efficiency ever<50% : TW (DT)
        ........................................
+ CA/CA-MCGILL-CLUMEQ-T2 FT f./DEST.:CONN.:SRM.GGUS 85578,85060 solved:
  the curls on the SE were out of date,updated the same day.

4) Fri Aug.31
      -> Cloud/site DDM efficiency  <50% : CERN
        ............................................
- CA/CA-MCGILL-CLUMEQ-T2 FT f./DEST.:/INVALID_SIZE+Checksum mismatch.GGUS
  85668 in progress:300+|h, 1K|4h FT f.Because of DT, site is
  automatically blacklisted in DDM,Sep.1.

5) Sat Sep.1
      -> Cloud/site DDM  efficiency <50% : TW
        ..............................
+ US/MWT2_UC FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 85703 verified:non-US FTS
  services weren't using the gridftp2 protocol which caused bottlenecks
  on the gridftp doors. The SRM door has been busy,but its worked
  through its backlog and the #threads is down.
- US/AGLT2 100s FT f./SOURCE:locality is UNAVAILABLE.GGUS 85712 in
  progress:do not route to TRIUMF or several other locations, due to
  the shutdown of the UltraLight router.Working to establish alternate
  routing.
+ CA/CA-VICTORIA-WESTGRID-T2_PRODDISK FT f./SOURCE:locality is
  UNAVAILABLE.GGUS 85705 verified:have been during a short time when
  some door nodes rebooted.Transfers are fine in last 8h.

6) Sun Sep.2
        ........................................

7) Mon Sep.3
        .......................................................
- CA/Australia-ATLAS 43% t.923422,923085,951163 job f./lost hearbeat.GGUS
  85727 assigned:under investigation.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.3.7.4 data12_8TeV Reco merge task 947681 failures:TRF_UNKNOWN out
 of memory:ERROR std::bad_alloc.Savannah 131548:group production Higgs
 HSG5.All jobs have failed mutiple attempts at NL/NIKHEF+SARA.Task is
 still in "submitted" state.All of jobs are aborting on memory
 allocation.Could this task be aborted?

++ 17.2.4.3.1 data12_8TeV Reco merge task 947665 failures:Reached
 maximum batch system time limit + kill looping job.Savannah 131654:
 group production HIGGS HSG2.Some jobs failed repeatedly at different
 sites in FR cloud running for 48h.Task set to "finished".

-- 17.2.2.4.2 data12_8TeV Reco merge task 946521 failures:Reached maximum
 batch system time limit + kill looping job.Savannah 131677:group
 production SM WZ and EW.2 remaining jobs have failed repeatedly at
 various sites rinning for UK cloud and hitting 48h limit.Task is still
 "running".No comments added so far.

-+ 17.2.4.10 data12_8TeV Reco task 953957 failures:ATH_FAILURE - Athena
 non-zero exit.TypeError: 'bool' object is not iterable.Savannah 131699:
 group production TRIGGER HLT.All jobs have failed 5 attempts in a row
 each causing 1140 job failures at CERN-RELEASE.Some configuration
 problem.Task ABORTED.

-- 16.6.7.20 mc11_14TeV simu task 961186 failures:RuntimeError:
 SimSkeleton:found problems with the method _do_metadata.Savannah 131760:
 MC11 production - SM.All jobs have failed up to 9 attempts at various
 sites running for DE cloud.Task is still in "submitting" state,no
 comments published so far.

-- 16.6.8.3 mc11_7TeV evgen task 954180 failures:Terminating event
 processing loop due to errors.Savannah 131761:MC11 production - TOP.All
 jobs have failed up to 11 attempts each at various sites running for UK
 cloud.Task is still in "submitting" state,no comments published so far.

-- LeslieGroer - 2012-09-04

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng FRsites_SONAR_TRIUMF.png r1 manage 1051.2 K 2012-09-04 - 19:56 LeslieGroer  
PNGpng gray.png r1 manage 0.1 K 2012-09-04 - 19:58 LeslieGroer  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2012-09-04 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback