TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130129 (2013-01-29, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-01-29

General News

  • ADC and ATLAS Computing News [20130129]
    • T2D acceptance: alpha: CA-MCGILL-CLUMEQ-T2 CA-SCINET-T2; bravo: AUSTRALIA-ATLAS CA-ALBERTA-WESTGRID-T2; charlie: CA-VICTORIA-WESTGRID-T2 SFU-LCG2
    • EMI-2 Update 8 is urgent for sites supporting ATLAS - fix for lcg-cp fails with files over 5 GB
    • EGI Trust Anchor release 1.52-1
    • ADC Weekly meeting
    • LHC running Pb-p, about 2 nb^-1/day. 14nb^-1 collected, goal 30 nb^-1. End run Feb 11.
    • DDM Operations
      • Use DATADISK at T1s for multicloud job inputs to solve PRODDISK issues
        • T1 PRODDISK capacity can probably be reduced to ~50/20TB
      • Removed RAW datasets
      • Removing unmerged simul.HITS (~991TB) and recon.AOD (865 TB)
      • Bug in site services fixed - improved thoughput dramatically for some sites (e.g. DE)
    • Grid Production
      • Removing fairshares at US, UK, DE, NL, TW T1s has helped balance GP vs MCpileup
      • Group production in DE, IT, CA, ND, UK, US clouds dropping off - being investigated
      • Tails of GP tasks still an issue even with automatic retries and lengthening queues
    • MC Production Report
      • A bit behind for Moriond requests, despite running ~80-90k simultaneous jobs.
      • About a month to finish MC12 requests and 1-2 weeks for MC11.
    • DPD Production
      • Removal of shares has helped throughput (20-30k simultaneous jobs)
      • Some of the changes made over the last few weeks have helped (task priority creep, auto-retries, etc), but still some hand intervention/micromanagement necessary
      • About a week of backlog on urgent CP tasks
      • Getting the tails under control will take some work. Discussions underway.
    • RAL Outbound Network Connectivity
      • Issue identified in October/November. Slow FTS transfers especially to US cloud. perfSONAR boxes helped identify the issue more precisely.
      • Main issue: 0.026% packet loss on outbound network from T1, especially correlated with distance.
        • Does not affect RALPP T3 (especially after fixed firewall issue in November) so backbone connection OK
      • Severely impacting the expected performance, especially as number of connections increase
      • Replacement hardware has to be used after power incident in November wiped out 10 switches at the datacentre.
      • Now ready to replace suspect switch - some discussion on exact timing but probably soon as site is only able to run at half-load due to the network issue
      • Also plan to upgrade to 40 Gbps link soon
    • Migrating SchedConfig from SVN to AGIS
      • Interface in place and ready to use in dev. Been testing since December.
      • ADC wants to wait until Moriond before full switch, despite the mishmash we have at the moment.
    • AMOD Report
      • lcg utils buggy: transfer of files larger than 5GB fail
        • patched version (with a hardcoded timeout option) not deployed on sites and no broadcast of this info provided by WLCG dev team
        • new (fixed) official version should be deployed very soon (by the end of the month)
      • Backlog of functional tests transfers from T0 to TRIUMF due to very high data activity (300MB/s)
      • Not generally known that the LHCOPN link is only 5 Gbps
    • Report on ADCoS

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Jan.22-28,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Jan.22-28,2013) Panda production service
      - completed successfully 1,321,243 managed group, MC production,
        validation and reprocessing jobs
      - average ~188,749 jobs per day
      - failed 139,057 jobs
      - average efficiency:
         -- jobs ~90.5%
      - active tasks: 3121
         -- distribution by cloud:
CA:159 CERN:114 DE:266 ES:191 FR:231 IT:184 ND:186 NL:646 TW:323 UK:567
US:254

B)   - CERN/CERN-PROD DT (LFC warning:LFC DB schema update), Jan.23.
     - CERN/CERN-PROD DT (LFC warning:LFC DB schema update), Jan.24.
     - DE/FZK-LCG2 DT (SE,SRM OUTAGE:network problem), Jan.23-24.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:updates at UiB pools), Jan.24.
     - UK/RAL-LCG2 DT (CE,SE,SRM warning:new equipment switch), Jan.28.

     - New M4.1 upgrade for ATLAS DDM Dashboard 2.0 released, Jan.24.
     - New LCG 64c configuration has been released, Jan.22.
     - New TRF caches were distributed:
       -- AtlasProduction 17.2.8.6(32+64b)
       -- AtlasPhysics 17.2.7.5.5(32+64b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Jan.22
      -> Cloud/site PROD efficiency ever<50% : FR/ROMANIA16
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ CERN/SS-LYON 50% avalability due to corrupt DB.Fixed.Elog 42311,42363
  .Savannah 99963.
+ DE/DESY-ZN FT f./GRIDFTP ER.:server err.500.GGUS 90646 solved:
  problems with LHCONE again,seems fixed the same day.
+ DE/HEPHY-UIBK FT f./SOURCE:GRIDFTP_ER.:conn.refused.GGUS 90704 solved
  :several dpm-gsiftp daemon failed during the night.
- US/BU_ATLAS_TIER2 many deletion errors/Conn.reset by peer.GGUS 89339
  assigned,updated.
+ FR/ROMANIA16 62% job f./lost heartbeat.GGUS 90665 solved:problem of   the cream limited the execution time of the jobs. After the   recompilation this problem disappear.
- ND/ARC t.1182239 job f./out of memory.Savannah 99969:after f. jobs
  were autoincreased 2->4GB memory,but some jobs still fail after that.
- IT/ZA-WITS-CORE_PRODDISK still 100% job f./stage-in.GGUS 90138   assigned,updated,unsolved:the site lost some files.
+ NL/IL-TAU-HEP 500 FT f./DEST.:CONN.:SRM.GGUS 90703 failed:no more
  failures,Jan.23.
+ UK/UKI-NORTHGRID-LANCS-HEP 6300+ FT f./DEST.:CONN.:SRM.GGUS 90704
  solved:srmv2.2 daemon had fallen over again,restarted.Escalating with
  DPM developers,they've tracked it down to a bug in the dpm code.

2) Wed Jan.23
      -> Cloud/site PROD efficiency ever<50%  DE/DESY
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ DE/MPPMU job f./SRM_NO_FREE_SPACE.GGUS 90746 solved:added extra
  space to the spacetoken the same day.
+ DE/FZK-LCG2<-T0 FT f./GRIDFTP_ER.:Unable to conn.to f01-060-110-e.   gridka.de/SORCE:CONN.:SRM.GGUS 90771 solved:the routing for LHCOPN
  to GridKa was incorrect,fixed after the DT,jan.24.
+ IT/GR-01-AUTH 41o deletion errors/Er.reading token data header:   Conn.closed.GGUS 90613 solved:no errors on Jan.27.
- IT/INFN-MILANO-ATLASC 200 FT f./DEST.:SRM-FAILURE.GGUS 90777 in
  progress:from the StoRM backend log files,it looks like all the
  PutDone requests are succeeding,but sometimes they take long time to
  complete (up to 8-10sec).Is there a timeout for the srmPutDone in the
  client command that could be increased?
+ UK/UKI-LT2-RHUL 1900 FT f./DEST.:SRM_ABORTED.GGUS 90764 solved:was
  due to high load on server caused by mysql.Stable for 2 days,Jan.28.

3) Thu Jan.24
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: -
        ............................................
+ CERN/CERN-PROD lxplus,lxvoadm conn.very slow.GGUS 90796 solved:IO   subsystem of afs154 was completely frozen (apparently triggered by
  a hardware problem).Fixed the same day.
+ DE/DE/DESY-HH_PRODDISK full.Savannah 135345:blacklisted,prod job f./
  stage-out.Cleanup done,whitelisted Jan.24.
+ DE/DESY-ZN_PRODDISK full.Savannah 135346:auto-blacklisted in DDM.
  Cleanup done,whitelisted Jan.24.
- UK/UKI-NORTHGRID-LANCS-HEP_PHYS-BEAUTY FT f./SOURCE:GRIDFTP_ER.:
  server err.500.GGUS 90778 waiting for reply:the disk server has
  suffered some kind of failure.Back online,after a raid system   re-verify and a round with the xfs repair tools,Jan.25.The fetch-crl
  cronjobs had been whacked on a number of other servers too - the
  configuration too for webdav misread the cron file and put in an extra
  newline character midway through the line - meaning the job wasn't   being run.
+ UK/UKI-LT2-QMUL 500 job f./staging.GGUS 90783,89733 solved:BDII
  problem at the site,switched over to the RAL bdii., no err on Jan.27.
+ ES/PIC 142 job f./Missing installation:No such file or dir.GGUS 90779
  solved:WNs td408,411,422 affected.Ran yaim in affected nodes to
  actualize them.They were stopped before the switch of the s/wa area
  to CVMFS.
+ CA/CA-SCINET-T2 job f./Get|Put er.:Unable to connect to 142.150.19.19
  GGUS 90791 solved:200+ same f.,Jan.27.A misconfiguration on the campus
  firewall not allowing access to 2 dcache pool nodes fixed,Jan.28.
+ FR/RO-07-NIPNE_DATADISK 10K+|4h FT f./DESt.:CONN.:SRM.GGUS 90808
  solved:had some cooling problems in datacenter,no errors on Jan.27.

4) Fri Jan.25
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : NL/JINR-LCG2
        ............................................
- IT/INFN-FRASCATI many t.job f./Get er.:Replica with guid...not found.
  GGUS 90818 in progress:file is available at the site,lcg-cp works.
  The same error appears at other sites.Not a site - prodsys issue?
- IT/INFN_MILANO-ATLASC_PERF-EGAMMA 40K+ FT f./AGENT:CONF._ER.:No site
  found for host t2cmcondor.mi.infn.it.GGUS 90777 in progress updated:
  In order to fix the free space reported by StoRM had to reconfigure it
  with yaim, and it started publishing wrong information in BDII,fixed
  Jan,25.Still 100s FT f./DEST.:SRM_FAILRE,Jan.25-27.From the StoRM b/e
  log files,it looks like all the PutDone requests are succeeding,but
  sometimes they take long time to complete (up to 8-10sec).Is there a
  timeout for the srmPutDone in the client command that could be
  increased?
+ UK/UKI-LT2-UCL-HEP many FT f./DEST.CONN.:SRM.GGUS 90826 verified:site
  in DT for DPM upgrade till Jan.29.
- DE/MPPMU_PRODDISK full.Savannah 135213 updated:autoblacklisted in DDM
+ ND/UNIGE-DPNC 800+ FT f./DEST.:CONN.:SRM.GGUS 90837 solved:the CRLs
  weren't updated on disk servers running Solaris,a side effect of the
  change of the head node of the SE done yesterday.Fixed.
+ ND/ARC-T2 814 job f./Failed to run downloader (pre-processing);Failed
  to run uploader (post-processing).GGUS 90840 solved:there was a bad
  configuration on 1 of the clusters (2 a-rex processes running).Fixed.

5) Sat Jan.26
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : ND/UNIGE-DPNC
        ..............................
- US/NERSC still FT f./DEST.:CONN.:SRM.GGUS 90619 assigned,updated:   experiencing f/s problems,put the storage element under maintenance
  till Mon. in OIM,for some reason AGIS doesn't find this.
- IT/INFN-T1_DATADISK nearly full.Savannah 135416:T0 export is to be   stopped until possible cleaning proceeds and make enough free space.
- DE/FZK-LCG2_DATADISK running out of space,excluded from T0 export.
  Savannah 135414.
+ ND/UNIGE-DPNC<-CERN FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 90841 solved:
  A modification of firewall done yesterday evening was preventing the
  services of DPM SE from contacting the mysql database.Fixed.
- UK/UKI-NORTHGRID-LANCS still 1K FT f./SOURCE:The available CRL has
  expired.GGUS 90778 in progress updated:1 server has trouble with crls,
  all the failures seem to be for files located on this disk pool.
  Fetch-crl cron had been whacked by the rolling out of the webdav   service on the node.Repaired.But still 300+ FT f. after that,Jan.27.
  Also HC tests f.,queues autoset to "test".Blacklisted:Savannah 135448.

6) Sun Jan.27
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : US/NERSC, UK/LANCS
        ........................................
- US/NERSC 1K/4h FT f./DEST.:CONN.:SRM.GGUS 90866 assigned:Downtime till
  Mon.announced in OIM, but not propagated for some reason to GOCDB and
  AGIS.Cross-linked with GGUS 90619.
- UK/UKI-SOUTHGRID-RALPP 800|4h FT f./SOURCE:AsyncWait.GGUS 90863
  assigned.
+ UK/UKI-LT2-QMUL_DATADISK 2K|4h FT f./DEST.:'userSpaceTokenDescription
  does not refer to an existing space.GGUS 90868,90871 solved:restarting
  the StoRM b/e+f/e helped.Seems triggered by a mysql update.300+ FT f./
  SOURCE:The available CRL has expired.
+ DE/HEPHY-UIBK 300 FT f./DEST.:CONN.:SRM.GGUS 90869 solved:dpm daemon
  died over the weekend,restarted.

7) Mon Jan.28
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
- CERN/SLS monitoring issue:ATLAS_DDM_Tracker,ATLAS_DDM_Deletion   unavailable.experts informed.Elog.42507-42508.
+ CERN/CERN-PROD "acrontab -l" f./No kerberos token.GGUS ALARM 90928
  solved:there was a problem with the hypervisor which switched the
  root fs to readonly.Should be fixed since ~14h40 when failed over to
  the stand-by.
- CA/VICTORIA-LCG2 Frontier/squid unavailable.GGUS 90895 assigned.
- ES/EELA-UTFSM->PIC 483 FT f./GRIDFTP_ER.:globus_gass_copy_register_
  url_to_url transfer timed out.GGUS 90932 in progress:Some of them in
  transferring state for many days.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

Moriond-related:

-- 17.2.7.5.3 data12_8TeV Reco merge BTAGSLIM (Moriond gr.1) task  1133304 failures: TRF_SEGFAULT|out of memory.Savannah 135063,99969  (group production BTAGGING).Only 3 jobs finished so far,others produced
 huge (>15Mb) logfiles and have failed up to 17 attempts in a row at  various US and CA sites due to memory allocation problem coming from  StreamNTUP_BTAGSLIMD3PDMaker.Other affected tasks:1133022,1133276,
 1133274,1133268,1133314,1133364,1133409,1133465,1133474,1133479,
 1133505,1133444 having multiple similar failures in different clouds:  US,FR,IT,DE,NL.Tasks are still "running".Auto-memory increase  implemented up to 4000MB doesn't help.Jobs seems to be hitting the  32-bit limit.

-- 17.2.7.5.3 data12_8TeV Reco merge tasks 1164476,1164653,1164657  (Moriond gr.2) failures: JetGetterBadInput.Savannah 135323:group
 production Tau.All jobs have failed up to 25 attempts in a row each at
 various clouds: US,FR,CA due to the error seems related to bad input
 files.The same problem with 100% failing Moriond gr.5,6,7+3 grop
 production HIGGS+JETMET tasks: - 1142550 (UK,12),1142529 (UK,10),  1142448 (IT,19),1142439 (CA,19),1142018 (UK,15),1141954 (FR,15);
 1142189 (FR,15),1142253 (UK,12);1142189 (FR,15),1142253 (UK,12);
 1171230 (25,US),1171228 (17,CA),1170972 (25,US),1170914 (17,FR),
 1170633 (17,FR).Tasks are still "submitting".No comments added so far.

-- 17.2.7.5.3 data12_8TeV Reco merge task 1164710 (Moriond gr.2)
 failures: ATH_COREDUMP | poolToObject:caught error.Savannah 100001,
 135326:group production Tau.All jobs have crashed 23 attempts at
 different sites in IT cloud.The same problem with Moriond gr.3(JETMET)
 tasks 1171204,1171025 failing in CA,IT clouds.Tasks are still in
 "submitting" state,no comments published so far.

-- 17.2.7.5.3 data12_8TeV merging task 1140746,1141628 (Moriond gr.5)
 failures:TRF_OUTFILE - output file error.Savannah 135336:group
 production HIGGS.All jobs have failed 25 attempts in a row each in UK
 and CA cloud due to "too few events in the output file" error.Tasks
 are still in "submitting" state.No comments added.

-- 17.2.7.5.3 1 data12_8TeV Reco merge task 1158934 (Moriond gr.7)
 failures:GUID for DAOD_4LHSG2...not found in DQ2.Savannah 135373:group
 production HIGGS.Jobs have failed 17 attempts in a row,weren't
 assigned to any site in ES cloud seems due to the error in the input
 DAOD file definition (or missing file?).Task is still "submitting".

Other tasks:

++ 17.2.7.9.3 data11_2p76TeV Reco HLT task 1185480 failures:ATH_COREDUMP
 | "Core dump from CoreDumpSvc on lxbrd38c02.Savannah 135338,39334:
 group production TRIGGER HLT.All jobs have crashed repeatedly at
 CERN-RELEASE on the ESDtoAOD step.Task ABORTED,under discussion.

++ 17.2.8.4 mc12_8TeV evgen tasks 1185427-1185432 failures:RuntimeError:
 inputGeneratorFile=group.phys-gener.alpgen...is incompatible.Savannah
 135339:MC12 production - SUSY.All jobs have failed multiple attempts
 in UK and DE clouds due to incompatibility of input generator files.
 Understood:original JOs were buggy,fixed and the tarball for these
 jobs have been updated.Tasks are in "running" state now.

-- 17.2.7.5.3 data12_8TeV Reco merge task 1164807,1166241 failures:Can't  find object in event store for JetCollection(1162448536)/AntiKt4LCTopo
 JetsReTagged.Savannah 135382:group production Tau.Only 29 jobs finished
 and 1814 failed up to 16 attempts at different sites in DE,UK clouds.  Tasks are still "running".No comments added so far.

-+ 17.2.7.5.2 mc12_8TeV Reco merge tasks 1185670-1185676 failures:
 ATH_FAILURE Exit code 10.Savannah 135386,135430:group production SUSY.  All jobs failed multiple attempts at various sites/clouds.Tasks are  still "submitting".Tasks are ill-defined with an NTUP_SUSY as input,
 but obviously should have the AOD.

-- 17.0.6.9.1 mc12_8TeV Reco merge tasks 1185577-1185591 failures:
  ERROR Cannot create tool CaloSwGap_v3 (No factory found).Savannah
  135384:group production Higgs.All jobs have failed multiple attempts
  in various clouds.Tasks are still "submitting".No comments published.

++ 17.2.7.9.2 mc12_5TeV DigiMReco tasks 1186747-1186752 failures: Run  timeError: key '--extraignorefilter' is not defined.Savannah 135405:
 MC12 production - HEAVYION.All jobs have failed repeatedly in various
 clouds:ES,NL,IT,US.Configuration error? Tasks ABORTED.Unfotunately,
 the new resubmitted tasks 1186965-1186970 fail with the same error.
 Tasks ABORTED.The configuration tag was fixed,resubmitted and running.

-- 17.2.7.5.2 mc12_8TeV Reco merge task 1187012 failures: AntiKt6Truth
 Jets_WZMomCalc.JetGhostAsso.TrackAssoc FATAL.Savannah 135431:group
 production SUSY:all jobs produced huge logfiles failed repeatedly at
 various sites in DE cloud.Task is still "submitting".Under discussion.

-- 17.2.2.2 mc12_8TeV Reco merge task 1162479,81,84 failures: "Read unknown parameter for EMShower Index 328".Savannah 135450: MC12 production - SUSY.All jobs have failed many attempts at different sites
running for CERN cloud.Tasks are still "submitting".No comments added
yet.

-- 17.2.7.5.3 data12_8TeV Reco merge task 1128571 failures:attempt to add a duplicate (StreamNTUP_WPRIMEMND3PDMaker.StreamNTUP_WPRIMEMND3PD
Maker.MET_CellOut_Eflow_JetAreaRhoEta5JVF_Filler) ... dupe ignored.
Savannah 135464:group production Exotics.All jobs have failed up to 9 attempts at various sites running for ES cloud.Task is still in "submitting" state.No comments published yet.

-- LeslieGroer - 2013-01-29

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2013-01-29 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback