Tier1 and Tier-2 Weekly Report 2012-08-14

General News

  • ADC and ATLAS Computing News [20120814]
    • CA availability for July all good (>94%)
ScreenShot015.png
    • WLCG availability also good
    • Special XP_Higgs production started on CA cloud (2.3M events, AFII) for the Higgs group (lepton+tau): e.g. JHUPythia8_AU2CTEQ6L1_ggH125_Spin{0,1,2}m_WW2tau
    • ADC Weekly meeting
    • Short meeting this week. No mention of EMI upgrade position this week.
    • Some issues in the US with users running private MC production. Not forbidden but discouraged. ADC will act if site services affected.
    • Too many user jobs at CERN affected LSF again
    • Short analysis of wallclock time for user analysis jobs over last three months (pathena, prun and ganga) to figure out what impact time limits may have. Fraction of jobs completed:
  < 1 hour < 12 hours < 24 hours
pathena 0.491 0.970 0.992
prun 0.862 0.990 0.997
    • Less than 3% of jobs would be affected with a time limit of 12 hours.
    • Setting job queue lengths per site based on published information in AGIS is the goal

    • HammerCloud autoexclusions 
      ANALY_ALBERTA-WG1 Auto-Excluded at Wed Aug  8 21:31:17 2012 CET
      IAAS Auto-Excluded at Thu Aug  9 09:36:40 2012 CET
      IAAS reset online at Thu Aug  9 10:33:56 2012 CET
      IAAS Auto-Excluded at Thu Aug  9 21:37:08 2012 CET 
      IAAS reset online at Fri Aug 10 01:37:07 2012 CET
      CA-ALBERTA-WESTGRID-T2 reset online at Fri Aug 10 07:34:48 2012 CET
      ANALY_ALBERTA-WG1 reset online at Fri Aug 10 12:00:33 2012 CET
     CA-ALBERTA-WESTGRID-T2 Auto-Excluded at Fri Aug 10 12:04:31 2012 CET
     ANALY_ALBERTA-WG1 Auto-Excluded at Fri Aug 10 12:31:41 2012 CET
     ANALY_ALBERTA-WG1 reset online at Fri Aug 10 14:00:42 2012 CET
     CA-ALBERTA-WESTGRID-T2 reset online at Fri Aug 10 15:02:28 2012 CET
     ANALY_AUSTRALIA Auto-Excluded at Sat Aug 11 06:43:53 2012 CET
     ANALY_AUSTRALIA reset online at Sat Aug 11 08:05:30 2012 CET
     ANALY_MCGILL Auto-Excluded at Sun Aug 12 00:02:01 2012 CET
     CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Sun Aug 12 00:04:00 2012 CET
     ANALY_MCGILL reset online at Sun Aug 12 00:30:13 2012 CET
     CA-MCGILL-CLUMEQ-T2 reset online at Sun Aug 12 02:32:04 2012 CET
     Australia-ATLAS Auto-Excluded at Sun Aug 12 05:36:15 2012 CET
     CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Sun Aug 12 05:36:22 2012 CET
      Australia-ATLAS reset online at Sun Aug 12 07:04:33 2012 CET
      ANALY_MCGILL reset online at Sun Aug 12 08:01:30 2012 CET
     CA-MCGILL-CLUMEQ-T2 reset online at Sun Aug 12 08:03:26 2012 CET
     CA-SCINET-T2 Auto-Excluded at Tue Aug 14 03:37:34 2012 CET
     CA-SCINET-T2 reset online at Tue Aug 14 05:32:57 2012 CET
     Australia-ATLAS Auto-Excluded at Tue Aug 14 09:49:41 2012 CET
     Australia-ATLAS reset online at Tue Aug 14 11:33:24 2012 CET

  • Security News [20120801]
  • gLite/EMI News [20120801]
    • EGI CSIRT ADVISORY [EGI-ADV-20120801] concerning gLite 3.2 middleware components no longer supported on 01 August 2012.

  • ROC_Canada News [20120710]
    • Current shift - Andrew Wong 7-10 Aug, 2012
    • Next Week - Sergey Chelsky

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Aug.7-13,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Aug.7-13,2012) Panda production service
      - completed successfully 1,740,957 managed group, MC production,
        validation and reprocessing jobs
      - average ~248,708 jobs per day
      - failed 72,792  jobs
      - average efficiency:
         -- jobs ~96%
      - active tasks: 623
         -- distribution by cloud:
 CA:16 CERN:49 DE:42 ES:25 FR:59 IT:30 ND:31 NL:124 TW:52 UK:39 US:156

B)   - ND/NDGF-T1 DT (SE,SRM warning:pool,network maintenance), Aug.7-8.
     - ND/NDGF-T1 DT (SE,SRM warning:pools DT at UiB), Aug.8.
     - ND/NDGF-T1 DT (SE,SRM warning:power work at DCSC), Aug.8-9.
     - ND/NDGF-T1 DT (SE,SRM warning:OTDR measurements on the OPN),Aug.13
     - TW/TAIWAN-LCG2 DT (SE,SRM OUTAGE:Castor DB reconf/upgrade),Aug.6-7
     - ES/PIC DT (CE-09,03 OUTAGE:migration from gLite to EMI), Aug.7-10.

     - New TRF caches AtlasProduction 17.2.4.7(32+64b),AtlasPhysics
       17.2.4.6.1(32+64b),MuonAlignment 17.2.4.6.1(32b) were distributed

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Aug.7
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ IT/INFN-T1 2K+ job f./Get err.:replica with guid...not found.GGUS 84929
  solved:due to communication problem between ATLAS DDM and job
  management system,not a site problem.
+ IT/INFN-GENOVA FT f./GRIDFTP_ER.:server er.500.GGUS 84955 solved:power
  failure caused a storage volume failure.Savannah 131007:whitelisted
  after the site fixed the issue,Aug.13.
+ FR/IN2P3-LPC_PRODDISK FT f./DEST.:CONN.:SRM.GGUS 84930 solved:DPM
  service crashed on clrlcgse01,restarted the same day.
+ US/MWT2_DATADISK FT f./FTS:FIRST_MARKER_TIMEOUT.GGUS 84932,84924
  verified:some new pools at IU were missing crucial firewall entries to
  allow outside gridftp access,set readonly to stop err.Corrected
  firewall settings.No err.,Aug.9.
+ US/UPENN_LOCALGROUPDISK FT f./DEST.:CONN.:SRM.GGUS 84957 solved:h/w
  problem brought out SRM down,fixed.No err.,Aug.8.

2) Wed Aug.8
      -> Cloud/site PROD efficiency ever<50%  ND/ARC-T2
      -> Cloud/site DDM  efficiency <50% : ND
        ................................................................
+ UK/UKI-SOUTHGRID-CAM-HEP many job f./Get err.:rfcp failed.GGUS 84961
  solved:unscheduled warning for 50TB disk server until Friday 17th
 (disk server down).The dpm daemon on the SE had stopped,restarted.
+ FR/ROMANIA07 many job f./Get:Err.reading token data header:Connection
  reset by peer.GGUS 84962 solved:problem with DPM server fixed.
+ ND/NDGF-T1->UNICPH-NBI FT f./GRIDFTP_ER.:server err.451:No write pool
  available.GGUS 84997 solved:the network link to Copenhagen went down at
  18:11 CET.The site is in DT.Savannah 131040.After DT FT succeed,Aug.13.
+ ND/ARC-T2 1.5K+ job f./LRMS error.GGUS 85000 solved:steno cluster DT,
  the active jobs were killed.
+ CA/TRIUMF many job f./Put err.:Internal name space timeout.GGUS 84999
  solved:caused by SE head node overload,rescheduled some background jobs
  .For longer term,the current head node will be replaced with a new
  powerful node.
+ IT/INFN-ROMA1 CALIBDISK All FT f./DEST.:CONN.:SRM.GGUS 850001 verified:
  DPM head node stopped on kernel panic,fixed.

3) Thu Aug.9
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: TW/ TAIWAN-LCG2
        ............................................
+ CERN/Panda server,bamboo nodes problems with atlpan.Savannah 131026:
  atlpan defined on all panda server/bamboo nodes locally.INS151683.
+ DE/TUDRESDEN-ZIH many job f./Get er.:Unknown host:SRM_FAILURE.GGUS
  85003 solved:DNS server problem fixed.
- DE/UNI-FREIBURG blacklisted again to recover lost files:Savannah 130910
  Blacklisting directly done in dq2.cfg.
+ UK/UKI-SOUTHGRID-OX-HEP->UKI-SOUTHGRID-OX-HEP FT f./SOURCE:Unable to
  connect to t2se36.physics.ox.ac.uk.GGUS 85033 verified:gridftp process
  seems to have died,restarted and the nodes are listening correctly.

4) Fri Aug.10
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
- ND/NDGF-T1_MCTAPE 2064 FT f. due to the internal issue (Network or
  Storage related).GGUS 85037 in progress:2-nd tape read pool has
  disabled itself during the night.Restarted and problem identified.The
  reads should work.Still 1171 FT f./SRMV2STAGER:Job failed with forced
  timeout after 21000s.The file is on the pool that had problems during
  the night.It is reading files all the time.
- DE/GOEGRID FT f./DEST.+SOURCE:AsyncWait/GRIDFTP_ER.:no route to host.
  GGUS 85052 in progress:still 100 FT f./SOURCE:ThePinCallbacks Timeout,
  Aug.12.

5) Sat Aug.11
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
+ CERN/CERN-PROD very slow LSF/bsub err.GGUS ALARM 85058 solved:can't
  submit jobs from lxplus,through creamCE,Aug.12.Offending processes
  killed,owner notified.System is coming back,Aug.12.
- US/MWT2 job f./Get err.:Failed to get LFC replicas:Bad credentials.GGUS
  85054 in progress:3 bad nodes taken offline.Still some f.,Aug.13.
  Another bad node offlined.
- FR/RO-14-ITIM many FT f./DEST.:CONN.:SRM.GGUS 85053 assigned.

6) Sun Aug.12
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : CA
        ........................................
+ CA/CA-MCGILL-CLUMEQ-T2 350 FT f./DEST.:INVALID_PATH:Invalid SURL
  specified.GGUS 85060 solved:there was a problem with the gpfs daemon on
  the storm node.It's OK for now,Aug.13.
- DE/UNI-BONN FT f./DEST.:CONN.:SRM.GGUS 85056 assigned.
+ FR/IN2P3-LPC 330 FT f./SOURCE:Stale NFS file handle.GGUS 85059 solved:
  had a filure on air cooler,repaired twice,Aug.13.

7) Mon Aug.13
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ CERN/atlas-SS08 FZK not available for a short time,recovered.Elog.38405
- DE/DESY-HH many job f./pilot:cmtsite command was timed out:[Errno 3] No
  such process(timed out).GGUS 85068 in progress:2 WNs are affected.Test
  cvmfs,err.on amd,interlagos nodes (64 slots) the time to setup software
  increases drastically.See also Savannah 129468.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

++ 17.3.4.2 valid1 DigiMReco task 932113 failures:TRF_EXC|"RuntimeError:
 ('Could not find RunDMCTriggerRunsInfo to get event/runNumber list: No
  module named RunDMCTriggerRunsInfo',)".Savannah 96611:T1_McAtNlo_Jimmy.
  All jobs have failed 5attempts at various sites running for CERN cloud.
  Task ABORTED.

++ 17.3.4.2 valid1 DigiMReco task 932402 failures:TRF_SVRINIT | "*** ***
 Wrong ATLAS layout: ATLAS-SLHC-01-02-01_VALIDATION *** ***.Savannah
 99613:T1_McAtNlo_Jimmy.All jobs have failed multiple attempts at various
 sites running for CERN cloud.Task ABORTED.

-+ 17.3.4.2 valid1 DigiMReco task 932944-5 failures:ATH_FAILURE-Athena
 non- zero exit:IndexError:list index out of range.Savannah 96644,96716:
 VALIDATION SLHC SampleA r3873.All jobs have failed up to 9 attempts in a
 row at various sites running for FR cloud.Tasks failed.No comments
 published so far.Some jobs of t.932946 also affected (ND cloud),set to
 "finished".

++ 17.2.2.4 data12_8TeV Reco task 933039 failures:ddm: Setupper._setup
 Source() could not decode VUID of prodDBlock.Savannah 131017:group
 production INDET SCT.All jobs have failed 9 attempts at various sites
 running for UK cloud.Typo in the task request(upper/low case letters,
 OK in AMI).Task ABORTED.

++ 17.2.2.4 data12_8TeV Reco task 933040 failures:ATH_FAILURE-Athena non-
 zero exit.Savannah 131018:group production INDET SCT.All jobs have
 failed 9 attempts at various sites running for TW cloud.Task ABORTED,HITS
 file needs to be digitised and the reconstructed by DigiMReco_trf.py not
 Reco_trf.py (advise the requestee).

-- 17.2.2.4 mc12_8TeV DigiMReco task 937185 failures:ATH_FAILURE - Athena
 non-zero exit:ZeroDivisionError:float division.Savannah 131124:group
 production INDET SCT.All jobs failed 10 attempts each at various sites
 running for IT cloud.Task "failed".No comments published so far.

-- 17.2.3.7.1 data12_8TeV MergeNTUP task 937088,98 failures:TRF_OUTFILE_
 TOOFEW,Transform output file contains too few events.Savannah 131125:
 group production Higgs HSG5.All jobs have failed multiple attempts at
 different sites in DE,ES clouds.Tasks are still in "submitted" state.No
 comments added so far.

++ 16.6.9.13 mc11_7TeV evgen tasks 934479-934488 failures:TRF_UNKNOWN -
 unknown transformation error:wrong ELF class:ELFCLASS64.Savannah 131045:
 MC11 production - SM.All jobs have failed multiple attempts in US,UK,DE
 clouds.Tasks ABORTED:the input libraries are for 64 bit but athena was
 setup for 32 bit,resubmitted for 64 bit.

-+ 17.0.6.17+16.6.7.26 valid1 digi tasks 934418-9 failures:TRF_UNKNOWN |"
 checkTagInfo: Could not retrieve TagInfo object from the detector store"
 |"Problems calling TagInfoMgr[0x14f927e0]+59".Savannah 96697:MC11 pp
 overlay test.Many jobs failed multiple attempts at various sites running
 for CERN cloud.Tasks are still "running",to be set to "finished".

-- LeslieGroer - 2012-08-14

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng ScreenShot014.png r1 manage 71.9 K 2012-08-14 - 18:29 LeslieGroer  
PNGpng ScreenShot015.png r1 manage 42.7 K 2012-08-14 - 18:32 LeslieGroer  
This topic: AtlasCanada > WebHome > GridActivities > ATLASCanadaTier-2Operations > T2WeeklyReport20120814
Topic revision: r1 - 2012-08-14 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback