TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130402 (2013-04-02, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-04-02

General News

  • ADC and ATLAS Computing News [20130402]
    • ADC Weekly meeting
    • Pilot Factories
      • Some discussion on tuning and number of factories needed at CERN
    • MC Production
      • Few large requests with long running jobs helping to fill sites
      • Some questions on handling HITS in MC12 AF-II workflow - larger files (380MB)
      • AF-II sim faster in mc12 than in mc11 so do 1000 events/job and write directly to TAPE (right from the beginning: since ~12 month); reduced number of steps
      • digi+rec in mc12 slower than in mc11 so only 500 events/job
      • Merging step could be adjusted.What is good size for HITS to TAPE?

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Mar.26-Apr.1,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Mar.26-Apr.1,2013) Panda production service
      - completed successfully 1,670,969 managed group, MC production,
        validation and reprocessing jobs
      - average ~238,710 jobs per day
      - failed 94,044 jobs
      - average efficiency:
         -- jobs ~94.7%
      - active tasks: 761
         -- distribution by cloud:
CA:36 CERN:12 DE:58 ES:26 FR:63 IT:38 ND:75 NL:48 RU:10 TW:46 UK:115 US:234

B)  Major Downtimes:
     - FR/IN2P3-CC DT(ccreamceli06-9 OUTAGE:migration to EMI2),Mar.25-26
     - ES/PIC DT (CE,SE,SRM OUTAGE:yearly electrical maintenance),Mar.26
     - NL/SARA DT (SE,SRM,CE,BDII OUTAGE:tape b/e maintenance), Mar.27.

C)  Other news:
     - New DB release 22.5.1 distributed/subscribed to sites, Mar.28.
     - New ATLAS release TRF caches were distributed:
       -- AtlasOffline    17.2.10 (32+64b)
       -- AtlasProduction -
       -- AtlasPhysics    17.2.7.5.10 (32+64b)
       -- AtlasMCProd     17.2.8.13.1, 17.2.1.4.7, 17.2.1.4.8 (32+64b)
       -- AtlasProd1      -
       -- AtlasTrigMC     -
       -- EvgenTP4MC11    -

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Mar.26
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : RU,ES,CERN,TW
        ........................................
- CERN/CERN-PROD<-T2s still ~1700 FT f./SOURCE+DEST.:proxy expired.GGUS
  ALARM 92166 waiting for reply updated,more errors during the week.
  Cross-linked to GGUS:92963 for FMPHI-UNIBA,Mar.29. ~400 f.,Mar.31.
+ NL/TECHNION-HEP FT f./SORCE:SRM returned an error,but with an invalid
  code.GGUS 92848 verified:f/e log filled all the disk and services
  failed.Disk space was recovered and server rebooted the same day.
+ US/NET2 again FT f./failed to contact on remote SRM.GGUS 92762
  reopened,solved:Bestman2 died.Services restarted,reduced load,DDM
  endpoints restarted.No f.|12h,Mar.27.Savannah 136738.
+ US/BNL_ATLAS_RCF ~1K job f./cmtside command was timed out.GGUS 92852
  solved:know issue when many jobs stat simultaneously in this queue,s/w
  installed in NFS produces timeouts due to high load.To be moved to
  CVMFS via NFS in the next 2 weeks.
- TW/TAIWAN-LCG2_MCTAPE ~1300 FT f./SRMV2STAGER:StatusOfBringOnline
  Request:SRM_FILE_UNAVAILABLE File has no copy on tape and no
  diskcopies are accessible.GGUS 92860 in progress:55 files are missing
  in disk server with no tape archive.Those files were dropped by
  migration failed due to some tape medium error exception,reported to
  CASTOR team,and should already have a patch for this case.Files can't
  be recovered,declared lost,Mar.28.
- ES/UAM-LCG2 many job f./cmtside command timeout.GGUS 92875 in
  progress:the list of affected WNs provided.Known cvmfs (2.0) problem.
  Savannah 100848,cwmfs experts contacted.
+ ES/PIC DATADISK 3K+ FT f./SOURCE:SRM_ABORTED.GGUS 92878 verified:due
  to some changes in chimera to improve the performance after the DT.
  Rolled back the same day.
+ ND/ARC-T2: 955(37%)job f./Failed to run downloader (pre-processing);
  Failed to run uploader (post-processing)".GGUS 92882 solved:WNs from
  *.gr.arctur.local affected.The faulty cluster is disabled for new
  jobs,needs an investigation what generated large amount of failures.

2) Wed Mar.27
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : FR/IN2P3-CPPM
        ................................................................
+ DE/MPPMU 147 job f./SITEROOT not found in command output.GGUS 92883 in
  solved:WN it01.t2.rzg.mpg.de affected.WN it01 had h/w problem,
  removed from scheduling ,Mar.28.
+ DE/HEPHY-UIBK ~660 FT f./failed to contact on remote SRM.GGUS 92884
  solved:dpm daemon died again.
- DE/GoeGrid ~350 FT f./SOURCE:ThePinCallbacks Timeout/AsyncWait.GGUS
  92915 in progress:still 1K+ f.,Mar.29,400+ f.,Mar.30. 1K+ f./SOURCE:
  TQueued|CONN.:SRM|SRM_ABORTED.Efficiency only 26%,Mar.31.
+ UK/UKI-NORTHGRID-LIV-HEP 409(43%) job f./lost heartbeat.GGUS 92905
  solved:WNs r21-n05.ph.liv.ac.uk to r21-n20.ph.liv.ac.uk affected.The
  vast majority of the failed jobs have occurred in the time frame
  affected by the interruption to external connectivity caused by the
  power cut.Resolved.
- FR/IN2P3-CPPM 884(eff.19%) FT f./DEST.:CONN.:SRM.GGUS 92910 in
  progress:also ~500|4h f./SOURCE:CONN.:SRM.The situation is stable
  Apr.1,after 600 new f.,Mar.29.
- IT/INFN-NAPOLI-ATLAS PHYS-HIGGS 132 FT f./AGENT err.:No status updates
  received since more than [360] seconds.Probably the process serving
  the transfer is stuck.GGUS 92916 assigned:cloud squad informed.No   problems seen by the expert on FTS,errors or excessive load,Mar.28.
- IT/INFN-T1 1K+ job f./staging input file failed.GGUS 92917 assigned:
  possible BDII permission denied problem,all queues auto-set to "test"
  or offline.Cloud squad informed,the f.are still there,Mar.28.

3) Thu Mar.28
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50% : US/SWT2-CPB, CA, DE
        ............................................
+ CA/MCGILL-CLUMEQ 3K+ FT f./SOURCE:SECURITY_ER.:User does not have
  valid permissions.GGUS 92947 solved:there is an authorization problem,
  emanating from the central ldap issues,fixed,mar.29.
+ US/NET2 again ~420 FT f./DEST.:COON.:SRM.GGUS 92762 reopened,solved:
  Some transient errors a few hours ago due to  stress testing SRM
  deletion. No more these failures for the past 4 hours.
+ US/SWT2_CPB new ~320 FT f./DEST.:CONN.:SRM.GGUS 92519 reopened,solved:
  0% eff.,more massive f. the same day.Transfers are fine in last 12
  hours,Mar.31.
+ US/MWT2 5K+ FT f./DEST.:CONN.:SRM.GGUS 92948 solved:difficulties with
  the network upgrade at UChicago,DT extended until 1200 CDT,Mar.29.
+ US/UPENN many FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 91122 reopened,solved:
  :caused by an unannounced update to xrootd that broke the installed
   system.Reverted to the old version and it is working again.
- IT/INFN-BOLOGNA-T3 many job f./Staging input file failed.GGUS 92929
  assigned:may be related to BDII issue,see GGUS 92917 for INFN-T1.
  Restarted BDII on:CEBO-T3-01/02,SEBO-T3-01,SGBO-T3-01/02,Mar.29.
+ NL/ITEP ~350 FT f./SOURCE:locality is unavailable.GGUS 92936 solved:
  site team reports the problem solved tha same day,no explanation.
+ DE/UNI-FREIBURG All FT f./SOURCE:globus_ftp_client: the operation was
  aborted.GGUS 92942 solved:firewall issue fixed the same day.
+ ND/NDGF-T1<->FZK again ~300 FT f./GRIDFTP_ER.:server err.426:conn.
  timeout.GGUS 92770 in progress since Mar.22 updated,solved:
  longstanding network problem between FZK and NDGF.Under debugging.Only
  26 such failures in last 24 hours,Mar.30.

4) Fri Mar.29
      -> Cloud/site PROD efficiency ever<50% :
      -> Cloud/site DDM efficiency  <50% : US, RU
        ............................................
+ US/SWT2_CPB new 22K+ FT f./DEST.:xrd/datadisk/rucio/data12_8TeV/71/0c   has trouble with canonical path.can't access it.GGUS 92519 solved:
  eff.0%,affects clouds.New f.after whitelisting.Savannah 136789.
  Transfers are fine in last 12 hours,Mar.29(8pm).
+ UK/UKI-LT2-QMUL 800+ FT f./SOURCE:CONN.:SRM.GGUS 92951 solved:one of
  the storage servers crashed,rebooted.Also have hit a Lustre bug on the
  clients,so have rebooted some of them as well the same day.
+ RU/RRC-KI-T1 600+(33%) FT f./CGSI-gSOAP running on fts4dae316f reports
  could not open connection to sdrm.t1.grid.kiae.ru.GGUS 92953
  solved:resolved (at 9:48 UTC+04:00): was a non-symmetric routing
  path to CERN's netblock 59624:1:188.184.128.0/17.But new f. over the
  weekend appeared,eff.~50%.problem with improper routing to CERN's
  LHCOPN part is resolved,Apr.1.
- DE/FMPHI-UNIBA_PRODDISK->CERN FT f./SOURCE:Could NOT load client
  credentials:proxy expired.GGUS 92963 in progress:looks more the FTS
  problem,proxy which comes with the command is expired already, not the
  site problem.Again CERN issue(?):GGUS 92166.More f.,Mar.31.

5) Sat Mar.30
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : RU
        ..............................
+ UK/UKI-NORTHGRID-LANCS-HEP 900 FT f./DEST.:SRM_ABORTED.GGUS 92973
  solved:restarted all the daemons.Savannah 136806.No more f.,Mar.31.
- US/NERSC ~500 FT f./DEST.:Checksum mismatch.GGUS 92768 reopened
  updated.
- DE/MPPMU many job f./lost heartbeat.GGUS 92975 assigned:WNs
  ot01.t2-ot14.t2 and pt01.t2-pt14.t2 affected.
+ NL/NIKHEF-ELPROD 450 FT f./SOURCE:CONN.:SRM.GGUS 92976 solved:no
  indications of problems on the SRM tbn18.nikhef.nl,transfers are
  nearly fine in last 12 hours,Apr.1.Efficiency is ~96%.

6) Sun Mar.31
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : RU
        ........................................
+ NL/WEIZMANN-LCG2 5K+ FT f./SOURCE:StoRM encountered an unexpected err.
  GGUS 92980 solved:also 6K+ deletion errors,no f. in 24h.,Apr.1.
+ CA/SFU-LCG2 230 FT f./SOURCE:The certificate has expired.GGUS 92979
  solved:host certificate were expired on gridftp doors.Stopped all   these doors.There are 9 more doors with valid certificates.Should be
  OK now.
- FR/BEIJING-LCG2 800+ FT f./SOURCE:Unable to connect to dpmds06.ihep.ac
  .cn.GGUS 92984 assigned:transfers are OK,Apr.1.
+ RU/RRC-KI-T1 250+ FT f./SOURCE:INVALID_PATH:Unknown Reason.GGUS 92953
  in progress updated:efficiency ~50% only during the weekend.Original
  problem with improper routing to CERN's LHCOPN part is resolved,Apr.1.
  Issues with se.grid.kiae.ru are to be tracked in separate ticket(?).
- IT/INFN-T1->LIP-LISBON FT f./SOURCE file size is 0.GGUS 92987
  assigned.
- ND/ARC 100% job f./LRMS error:(271)SGE error 100:assumedly after job.
  GGUS 92989 assigned:all *cos.lrz.de nodes affected,cloud squad
  informed:these nodes doesn't belong to NDGF(LRZ-LMU?).

7) Mon Apr.1
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50%: -
        .......................................................
- UK/UKI-SCOTGRID-GLASGOW DATADISK 1500 FT f./DEST.:NO_SPACE_LEFT.GGUS
  92990 assigned.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

++ 16.6.7.13 mc12_14TeV evgen tasks 1225292-95 failuris:SimSkeleton:
 found problems with the method _do_metadata.Savannah 136737:MC11
 production - UPGRADE.All jobs have failed 9 times at BNL/US cloud.
 Task ABORTED:a change was required in the simulation tag.

-+ 17.2.2.6.2 mc10_14TeV digi task 1225787 failures:Non-zero return code
 from FTKSimulationMerge(3).Savannah 136689:Manual FTK merging reco test
 All jobs have 6 attempts at CERN-RELEASE.Task ABORTED,s/w developers
 informed and discussed the error at the meeting.

-+ 17.2.2.6.2 mc10_14TeV digi task 1226786 failures: TRF_SEGVIO -
 Segmentation violation.Savannah 136769:Manual FTK merging reco test.All
 jobs have failed several attempts at CERN-RELEASE due to the following
 error: "Non-zero return code from FTKFullSimulation (8)(Error code 97)"
 Task ABORTED,s/w developers informed.

-+ 17.2.6.2 valid1 Merging tasks 1226810-1226847 failures:RuntimeError:
 skipEvents requested, but can't find which sub-step.Savannah 101035:
 MC12 VALIDATION New SampleA 64bit s1586.All jobs have failed multiple
 attempts at various sites/clouds in the very beginning.Tasks ABORTED,
 no comments published.

++ 17.2.8.13 mc12_8TeV evgen tasks 1226802-1226805 failures:Runtime
 Error: inputGenConfFile...is incompatible with inputconf.Savannah
 136771:MC12 production - SM.All jobs have failed so far at various
 sites in UK,ND clouds.Tasks ABORTED:a problem with upper and lower
 letters in the JO with the input dataset pattern to check against the
 one provided on the command line to be fixed.

++ 17.2.7.5.3 mc12_8TeV Reco mefge tasks 1226861,1226863 failures:Too
 many/too large input files. Total file size 242825501.Savannah 136773:
 group production JET/MET.All jobs have failed repeatedly at various
 sites in ES and DE clouds.Tasks DONE.Seems the corrections were
 done(?),the jobs have succeeded on the next attempts.

-+ 17.2.8.13 mc12_8TeV evgen task 1227552 failures: TestHepMC ERROR
 EFFICIENCY IS TOO LOW!At least:98% required.Savannah 136796:MC12
 production - EXOTICS.All jobs have failed failed 5 attempts in a row
 each at various sites running for US cloud.Task ABORTED and the
 requester has been informed.

-- 17.2.5.2.1 data12_8TeV Reco merge task 1044838 failures: No
 reconstructed Primary Vertex has been found.Core dump.Savannah 136802:
 group production EGAMMA.~60% of jobs crashed up to 15 attempts at
 different sites (US,CA) running for US cloud.Task is still "running",
 the last successfull job was done in November 2012.

-- 17.2.8.12 mc12_8TeV simu task 1226405 failures:Segmentation fault.
 Savannah 101052: MC12 production - SUSY.50%+ jobs failed multiple
 attempts at various sites running for ES cloud.Task is still "running".

-+ 17.2.2.6.2 mc10_14TeV digi task 1227740 failures: FATAL Error
 connecting branch: FTKTracksStream0.Savannah 136803:Manual FTK
 emulation test.All jobs but 1 have failed multiple attempts at CERN-
 RELEASE.Task ABORTED,no other comments added.

-- 17.2.7.5.10 data12_8TeV Reco merge task 1227459 failures:memory or
 time limit.Savannah 136810:group production SUSY.All jobs but one have
 failed so far at different sites running for UK cloud.Task is still in
 "running" state.

-- 17.2.1.4.4 valid1 DigiMReco task 1228210 failures: TRF_SEGVIO -
 Segmentation violation.Savannah 101059:McAtNloJimmy_CT10_ttbar_Lepton
 Filter.All jobs have crashed repeatedly at various sites in FR and DE,
 running for FR cloud on the Digi step.Task is still "submitting".

-- LeslieGroer - 2013-04-02

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2013-04-02 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback