TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121127 (2012-11-27, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-11-27

General News

  • ADC and ATLAS Computing News [20121127]
    • EGI trust anchors 1.51-1 released
    • T2 Data Acceptance Shares for November: https://twiki.cern.ch/twiki/bin/view/Atlas/DataDistributionSharesForT2#Shares_2012_Nov
      • All sites deemed above 90%
    • ADC Weekly meeting
    • LHC 0.55 fb^-1 delivered last week. Peak Luminosity 6.9E33. MD + Some cryo problems.
      • Gearing up for 25ns running in Dec before shutdown, then heavy ions in January
    • Reprocessing
      • 95+% done (all periods A,B,C,D,E,G,H-submitted yesterday).
      • UK behind due to power issues at RAL. Adding mulitcloud to help out.
      • Many T1 DATADISK close to full - assign some PRODDISK to DATADISK.
      • Except majority of job completed in next 3-4 days, with cleanup of problematic/failed jobs after that. Merge jobs may take a little longer.
    • T1 Disk Space
      • Cleanup has dropped usage from ~3PB to 1.5PB. Also some additions.
      • T1-T1 replication switched on (except for CNAF)
      • T0 export switched on for all DATADISK endpoints
      • BNL added 200TB before the weekend
      • Stephane identified marked many datasets on DATADISKs as secondary
        • Cleaning agent removed them during Friday and the weekend
      • More group datasets removed from DATADISK
    • AMOD Report
      • FTS3 tests on RAL :
        • Discovered STORM bug (fixed in STORM version EMI2-1.10.x)
      • Proxy certificate (for FT) problem (Friday–Saturday)
        • procedure to replace certificates still not 100% effective
      • LHCONE network incident :
        • caused problems to the whole Italian cloud and to the sw installation system (in Rome).
      • AFS issue on Sunday affected T0 as well as causing whole atlas.web.cern.ch web site to be unavailable for hours, it was fixed Monday morning. Unavailability of TiersOfAtlasCache web copy affected ATLAS production.
      • Power incident at RAL Nov 20. Still recovering (disk recovered, batch at 60%)
    • HammerCloud Report - new AFT test
      • change from mc10 to mc12, UserAnalysis (change from 16.0.3 to 17)
      • New AFT test with mc12 is healthy
        • Running since a week
        • Performance similar to old AFT
        • Input datasets have been replicated to all sites
        • Plan to switch tomorrow (turn off test with mc10)
          • Switch should be transparent, could take a bit longer to black/whitelist
        • Working on a new test to replace the DESD based data10 test (couple of weeks)
      • PFT update coming soon
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

     World-wide weekly ATLAS ADCoS summary (Nov.20-26,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Nov.20-26,2012) Panda production service
      - completed successfully 1,538,271 managed group, MC production,
        validation and reprocessing jobs
      - average ~226,182 jobs per day
      - failed 124,483 jobs
      - average efficiency:
         -- jobs ~92.5%
      - active tasks: 4098
         -- distribution by cloud:  CA:241 CERN:50 DE:363 ES:282 FR:439 IT:427 ND:226 NL:166 TW:423 UK:661 US:820

B)  Downtimes:
     - ND/NDGF-T1 DT(SE,SRM warning:upgrade,2 pool servers,CSC), Nov.20
     - ND/NDGF-T1 DT(SE,SRM OUTAGE:dCache upgrade,ndgf h/nodes), Nov.20
     - UK/RAL-LCG2 DT (all services OUTAGE:power incident), Nov.20-21.
     - UK/RAL-LCG2 DT (CE,SE OUTAGE:power incident), Nov.20-22.
     - UK/RAL-LCG2 DT (all services: warning:at risk,recov.), Nov.22-23.
     - CERN/CERN-PROD DT (ce203 OUTAGE:upgrade to emi2), Nov.23-Dec.3.
     - CERN/CASTORATLAS transparent upgrade to version 2.1.13-6, Nov.22.
     - FR/IN2P3-CC DT (cccreamceli06|8 OUTAGE:EMI updates), Nov.26.

    Other news:
     - CERN LCG 64b configuration has been released on AFS, Nov.23.
     - New ATLAS DBRelease-21.0.1 subscribed to sites, Nov.23.
     - New ATLAS releases and TRF caches were distributed:
       -- AtlasProduction - 17.2.7.7(32+64b)
       -- AtlasPhysics    - 17.0.6.9.2(32b)
       -- TrigMC          - 17.2.6.1.6(32b)
       -- CAFHLT          - 17.1.5.21.4(32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Nov.20
      -> Cloud/site PROD efficiency ever<50% : IT(GGUS:88553 verified)
      -> Cloud/site DDM  efficiency <50% : IT/INFN-T1,UK
        ........................................
+ DE/UNI-FREIBURG ~30 job f./Too little space left on local disk to run
  job.GGUS 88624 verified:pbs error and output files of local user
  could not be delivered and filled disks,deleted error and log data to
  free disks and user homedir.
+ DE/WUPPERTALPROD 1K+ job f./lost heartbeat.GGUS 88691 verified:lost
  all jobs in the batch system due to a file system error on the SGE
  server.Fixed:no errors anymore the same day.
+ ND/NDGF-T1 FT f./SOURCE:CONN.:SRM.GGUS 88638 verified:had a downtime
  for updates that got extended,errors are gone after that.
+ IT/INFN-T1 disk full(discrepancy between physicaal|SRM space),
  blacklisted in DDM,cloud IT set to brokeroff.Savannah 133937:fixed,
  Nov 22:SRM publishes 75 TB free.Whitelisted,online.
+ UK/RAL-LCG2 power outage,UK cloud set brokeroff.Savannah 133943,
  recovered,Nov.22-23.Whitelisted,online.

2) Wed Nov.21
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ CERN/ss-triumf,ss-fzk Site Services red.Experts informed.Elog 41082,
  4109393.Recovered.
- DE/FZK-LCG2 MCTAPE more FT f./SRMV2STAGER:Job failed with forced
  timeout after 43200 seconds.GGUS 87526 since Oct.18,unsolved:the site
  constantly keeping an eye on the system.Due to high activity on the
  tape b/e,frequent errors are unavoidable.
+ FR/BEJING-LCG2 ~1K FT f./DEST.:SRM_)ABORTED.GGUS 88713 solved:h/w
  failure fixed the same day,no new errors.
+ DE/PRAGUELCG2 ~500 FT f./GRIDFTP_ER:Gridmap lookup failure.GGUS 88717
  solved:problem with globus-gridftp-server processes on DPM pool nodes
  (they were started with wrong parameters),fixed the same day.
+ IT/INFN-ROMA2 Frontier/squid in red for 2 weeks,site not in Panda.
  Savannah 133976:new site and still setting up,panda queues are not
  defined yet so no job can use the Squid.
+ IT/GR-01-AUTH FT f./SOURCE+DEST.:HTTP_TIMEOUT:SRM.GGUS 88730 verified:
  the site completed DT.
- NL/SARA-MATRIX ~84K deletion errors/LFC exception:permission denied.
  Savannah 98924 updated:need to run a script that fixes this problem
  from a machine that would be trusted by LFC.Currently there is no such
  machine available, but there is a snow ticket with a request for it.
  Still many err.,Nov.22,24.~15K err.,Nov.25.Also ~500 FT f./SRMV2STAGER
  :StatusOfBringOnlineRequest:SRM_FILE_UNAVAILABLE|SOURCE:locality is
  UNAVAILABLE.~15K|4h deletion err./SRM_FILE_UNAVAILABLE,Nov.25.

3) Thu Nov.22
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: -
        .....................................................
- NL/TECHNION-HEP ~60% job f./TRF_SVRINIT | '*** COOL exception caught:
  No physical Update connection for "COOLOFL_INDET" is available.GGUS
  88760 assigned:cvmfs access problem?
- CA/AUSTRALIA-ATLAS still many FT f./GRIDFTP_ER.:globus_ftp_client:
  server err.500.GGUS 88407 in progress updated:many job f./transfer
  timeout,Nov.22.Network experts investigate.
+ UK/UKI-NORTHGRID-MAN-HEP 6900 FT f./DEST.:SRM_ABORTED|SRM_NO_FREE_
  SPACE.GGUS 88779 verified:related to f. of the FTS3 tests to which
  Manchester was added before RAL power failure.

4) Fri Nov.23
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : ND
        .....................................................
- ND/NDGF-T1 ~9K FT f./SRMV2STAGER:StatusOfBringOnlineRequest:
  SRM_FAILURE.GGUS 88802 in progress:one tape read pool is complaining
  about a f/s being full.
- CA/CA-MCGILL-CLUMEQ-T2 1200 FT f./DEST.:Local filesystem mask doesn't
  allow setting up correct ACLs for PtP.GGUS 88794 assigned.
+ FR/RO-07-NIPNE 900+ FT f./DEST.:CONN.:SRM.GGUS 88797 solved:Nov.24.
- IT/INFN-ROMA1 2600 FT f./DEST.+SOURCE:CONN>SRM.GGUS 88832 assigned:
  unscheduled DT for a power cut in the campus.Stll f.after DT(?),Nov.23
- US/AGLT2_PHYS-SM some DaTRI requests are awaiting subscription due to
  space.Savannah 134020,38152:freed more than 2TB to allow DS transfer
  the same day.Most FT started,still 1 waiting(requested Oct.26),Nov.25.
- UK/RAL-LCG2_PERF-JETS some DaTRI requests are awaiting subscription
  due to space.Savannah 134021:clean up to be done at the site soon.
- DE/UNI-FREIBURG PHYS-HIGGS some DaTRI requests awaiting subscription
  due to space.Savannah 134022:stopped these,requested the copy to
  another higgs site.Will continue to clean the space,it takes time to
  copy datasets to another site before being able to delete them.

5) Sat Nov.24
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
- UK/UKI-SCOTGRID-ECDF 600 FT f./SOURCE:the available CRL has expired.
  GGUS 88848,88577 reopened,in progress:the crls have expired on disk
  server(pool3) that had the problem with its certificate last week.Ran
  it manually(Nov.25),crls are updated.Will leave ticket open until a
  proper fix to the cron to be done (expected on Nov.26).
- DE/FZK-LCG2 1495 job f./stageing errors.GGUS 87526 updated,unsolved:
  :many FT f./SOURCE:Failed to pin file|locality is unavalable,Nov.25.
  Problem will most probably persist due to high load,no clear solution
  yet.
- CA/CA-MCGILL-CLUMEQ-T2 8K+ FT f./DEST.:Invalid SRM version for
  endpoint/AGENT:No site found for host storm02.clumeq.mcgill.ca.GGUS
  88843 assigned:also 1400 FT f./DEST.:Local filesystem mask does not
  allow setting up correct ACLs for PtP,Nov.24,26.

6) Sun Nov.25
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
+ CERN/bourricot down.Elog.41243.Seems fixed?
+ CERN/ATLAS AFS release area not accessible.GGUS ALARM 88856 verified:
  Atlas AFS folders were extremely slow to respond, which caused a
  saturation of the server due to the large amount of pending requests.
  Fixed.
+ TW/TAIWAN-LCG2 many FT f./SOURCE:CONN.:SRM.GGUS 88490 solved:seems a
  temporarily network issue between Taiwan and CA.FT is OK,Nov.26.
- DE/UNI-FREIBURG FT f./AGENT:CONFIGURATION_ER.:Channel closed for your
  VO or VO not authorized for transferring between CERN-PROD and
  CA-MCGILL-CLUMEQ-T2.GGUS 88850 in progress:similar f. from TAIWAN-LCG2
  and INFN-T1.FTS configration problem(?).Looks like this problem
  disappeared,Nov.26.All transfers to Freiburg succeed.
+ UKI-SCOTGRID-GLASGOW_PRODISK 100s FT f./DEST.:NO_SPACE_LEFT.GGUS 88858
  solved:problem with bourricot machine.Not a site issue.
- CA/CA-MCGILL-CLUMEQ-T2<-CA-ALBERTA-WESTGRID-T2 3K+(100%) FT f./DEST.:
  Local filesystem mask does not allow setting up correct ACLs for PtP.
  GGUS 88857 in progress:restarted storm services,but 1700+ FT f./DEST.:
  CONN.:SRM,Nov.26.

7) Mon Nov.26
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
- FR/GRIF-IRFU 100% FT f./SOURCE:SECURITY_ER.:available CRL has expired.
  GGUS 88855 assigned:IRFU storage is already using EMI,but Quattor
  people did deploy "fetch-crl upgrades" last week : either this is a
  coïncidence,or CRL download mechanism was broken at the same time.In
  each case,this should be easy to fix.
- DE/FZK-LCG2 DATATAPE many FT f./GRIDFTP_ER.:an end of file occurred
  (possibly the destination disk is full).GGUS 88877 in progress:still
  missing 1 of the previous 4 gridftp doors,which does not help.The
  queue is on the tape write pools and there is a huge backlog to store
  to tape.Recommended removing FZK from T0 export again - the tape
  system simply cannot take it.Export to FZK set to 0,FZK DATATAPE
  removed from T0 export:Savannah 134081.
- NL/AM-04-YERPHI stll many (96%) FT f./GRIDFTP_ER.:globus_gass_copy_
  register_url_to_url transfer timed out.GGUS 86265 in progress since
  Sep.21 updated.Blacklisted in DDM also for writing:Savannah 132529.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.7.4 data12_8TeV Merging repro task 1057646 failures:Payload ran
 out of memory:mergePOOL.exe (event merge) encountered a problem.
 Savannah 98943:BULK reprocessing AOD-merging for Period C.414 failed
 jobs,with 7 completed,out of 53 total in the task.Jobs are failing at
 UK.RAL-LCG2,UK.RAL-LCG2_HIMEM.Task is still "running".It seems that
 almost all events are skipped by EventSelector with an error message
 concerning the handling of LumiBlock metadata tool.

-+ 16.6.9.12 mc11_7TeV evgen tasks 1059560,62-65,79-80,82 failures:
 Syntax Error:invalid syntax.Savannah 133931:MC11 production - TOP.Many
 jobs failed in various clouds.Tasks are in "running","submitting" or
 "finished" state.Should have now been fixed - subsequent attempts
 should succeed.

++ 17.2.7.6 valid1 evgen task 1059514 failures:ValueError: received an
 instance of <type 'float'>, but <type 'list'> expected.Savannah 98944,
 98946:MC12 production - PHYSICS.All jobs ahve failed multiple attempts
 at various sites running for IT cloud.Understrood:non backward
 compatible change.Task ABORTED.

++ 17.2.7.4.1 data12_8TeV Reco merge task 1054286-8 failures:Attribute
 Error:'<class 'AthenaCommon.AlgSequence.AlgSequence'>' object has no
 attribute 'Kt5TopoJets'.Savannah 133932:group production JetMET.All
 jobs have failed multiple attempts at various sites in US,TW cloud.
 Understood - submitted to DESDs and now the problem was fixed in the
 new tags collected in the nightlies.Tasks ABORTED.

-+ 17.2.7.4.1 data12_8TeV Reco merge task 1059712 failures:'*** COOL
 exception caught: No physical Update connection for "COOLONL_LAR" is
 available.(CORAL : "ReplicaCatalogue::replicasForConnection" from
 "CORAL/Services/ConnectionService" )' | "Could not create a new
 conditions database.Savannah 133955:group production JETMET.140 jobs
 failed at ND/ARC.Task is "running".Tasks 1066048,1066097 (MC12
 production SUSY) affected also.For ND/ARC,one cluster had a typo in
 frontier settings.GGUS 88845 assigned:typo was fixed,Nov.26.

++ 17.2.4.3 valid1 DigiMReco task 1059569 failures:NameError: name
 'digitizationFlags' is not defined.Savannah 98969:singlepart_empty.All
 jobs have failed multiple attempts at various sites running for US
 cloud.Tasks ABORTED and resubmitted.

++ 17.2.6.2 valid1 ValidateD3PD merge task 1059584 failures:Folder
 /TRIGGER/HLT/HltConfigKeys does not exist.Savannah 98970:T1_McAtNlo_
 Jimmy.All jobs have failed up to 9 attempts at various sites running
 for CERN cloud.Task ABORTED.

++ 17.0.6.4 mc12_8TeV Reco merge tasks 1047915-1047953 failures:
 Attribute Error: 'TObject' object has no attribute 'GetEntriesFast'.
 Savannah 133968:MC12 production - TAU.All jobs have failed up to 10
 attempts each at various sites/clouds.Looks like problem during AOD
 creation, or some SW problem.Tasks ABORTED - the tag t84 is for mc11
 and not for mc12 samples.Tasks will be resubmitted.

-- 17.2.7.4.1 mc11_7TeV Reco merge task 1066982-1067308 failures:Can't
 find object in event store for Analysis:MuonContainer(1073853171)/Muons
 Savannah 134054:group production Tau.All jobs have failed multiple
 attempts at various sites/clouds.Tasks are in "failed" state.

++ 17.2.7.5 mc12_8TeV evgen task 1067915-1068033 failures:RuntimeError:
 No 'madgraph...evts' file found.Savannah 134055:MC12 production - SUSY.
 All jobs have failed multiple attempts at various sites/clouds.
 Understood:wrong release.Tasks ABORTED and will be resubmitted with the
 latest patch.

-- LeslieGroer - 2012-11-27

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-11-27 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback