TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130212 (2013-02-12, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-02-12

General News

  • ADC and ATLAS Computing News [20130212]
    • ADC Weekly meeting
    • Extended LHC running to Wednesday (Feb 14) to collect 5nb^-1 pp 2.76TeV. p-Pb delivered 31.2nb^-1.
    • HLT reprocessing at BNL (slides)
      • Currently runs at CERN. Testing doing this at BNL on the grid (backup and take additional load if necessary).
      • Need fast feedback
      • Special conditions - up to date release (via CVMFS), database access (Frontier), trigger menu
      • Teething issues in summer 2012. Successfully tests in October
      • Working on remaining issues plus policies
    • DPD Production (slides)
      • Most jobs from prior to Xmas are done
      • Some tails in JETMET and TAU
      • About 3k high priority jobs, 10k other
      • SL6 production readiness is a concern given that some T1s (SARA) could be fully migrating soon
    • MC Production (slides)
      • Mostly done - about 32k to go, few days
    • DDM Report
    • Rucio Renaming
      • Want to rename current files to meet rucio specifications i.e. /rucio//L1/L2/filename where L1/L2 are the 2 fi rst bytes of MD5(LFN) e.g. PFN : /rucio/user/jdoe/35/be/004406.EXT0. 00011.root
      • https://twiki.cern.ch/twiki/bin/viewauth/Atlas/DDMRucioPhysicalFileName
      • New naming convention already in use for DATADISK, SCRATCHDISK, CALIBDISK, PRODDISK + group area, (but not LOCALGROUPDISK)
        • would like to enable for LOCALGROUPDISK soon but need to consult with end users on timing (consult before S/W week)
      • TAPE endpoints still follow the old naming convention
      • This change is completely transparent for grid tools (fi les are still registered in the LFC) and for the federations (the LFN in the LFC still follow the old convention).
      • Plan is to rename old files centrally via WebDAV
        • Requires sites to deploy WebDAV - would like this prior to June
        • Longer term - may get rid of SRM.
        • Problems with DPM deployment fed back to developers
        • "The bulk renaming will be done on all sites before the end of the year. This is not disruptive for analysis and production and doesn't require any downtime from the sites."
      • A new flag (isDeterministic) will be introduced in AGIS to identify sites where all fi les follow the new naming convention. Means LFC lookup can be skipped.
      • Changes have been done in the Panda server to send the scope of the files to the pilot.
      • Some changes are required in the pilot so that the output are written according to the new convention and that the LFC lookup is skipped for the sites where isDeterministic=True
    • AGIS schedconfig migration
      • Final debugging steps followed by a shakedown period. Things seem stable.
      • missing maxwdir variable - will be fixed
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

      World-wide weekly ATLAS ADCoS summary (Feb.5-11,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Feb.5-11,2013) Panda production service
      - completed successfully 1,253,045 managed group, MC production,
        validation and reprocessing jobs
      - average ~179,006 jobs per day
      - failed 160,623 jobs
      - average efficiency:
         -- jobs ~88.6%
      - active tasks: 1235
         -- distribution by cloud:CA:80 CERN:20 DE:225 ES:39 FR:91 IT:120 ND:77 NL:48 TW:28 UK:249 US:258

B)   - IT/INFN-T1 DT (CE08 OUTAGE:upgrade to EMI2,SL6), Feb.1-7.
     - IT/INFN-T1 DT (SE,SRM warning:GPFS f/s b/g operations), Feb.5-8.
     - IT/INFN-T1 DT(SE,SRM warning:h/w storage failure), Feb.7-10,11-13
     - ND/NDGF-T1 DT (SE,SRM warning:update of java), Feb.5.
     - NL/NIKHEF_ELPROD DT(CE OUTAGE:upgrade CreamCE-UMD2),Jan.31-Feb.13
     - TW/TAIWAN-LCG2 DT (cream05 CE OUTAGE:hard drive failure),Feb.6-7.
     - ES/PIC DT (wms02 OUTAGE:migration EMI-1 to EMI-2), Feb.8-11.

     - CERN: rolling interventions on INT8R,INTR ATLAS DB, Feb.5.
     - New ATLAS DDM Dashboard 2.0 upgraded to M4.3 release, Feb.6.
     - New pilot version 56c announced, Feb.11.
     - ATLAS Production task monitoring has been updated, Feb.11.
     - New ATLAS release TRF caches were distributed:
       -- AtlasProduction 16.6.70.2(64b),17.2.8.8(32+64b)
       -- MCProd 17.2.1.4.2(32b)
       -- AtlasPhysics 17.2.7.5.7(32+64b)
       -- P1HLT 17.1.6.6(32b)
       -- EvgenTP4Mc11 17.2.8.6.1(32b),17.2.8.7.1(32b)
       -- TrigMC 17.2.8.7.1(32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Feb.5
      -> Cloud/site PROD efficiency ever<50% : IT/BOLOGNA,TW/Taiwan-LCG2
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ IT/INFN-BOLOGNA-T3 83% job f./IOVDbSvc ERROR ** COOL exception caught:
  No physical Update connection for "COOLOFL_INDET" is available.GGUS
  91158 assigned:the problem with the SRM service solved,Feb.8.
- TW/Taiwan-LCG2 54% job f./Put:Err.in copying the file from job workdir
  to localSE.GGUS 91160 in progress:lcg-cp is hang from all WN to CASTOR
  write pool frequently,caused by --srm-timeout option.Castor doesn't
  implement such SRM attribute for Castor srm interface,instead,Castor
  uses stager timeout to control request timeout.Can increase Stager
  time out to have workaround for this,but it doesn't really solve this
  issue(it just increases walltime and gets bad cpu efficiency).Removed
  the option "--srm-timeout 3600" from the pilots sent to TW on 02-05 in
  the evening.The last time out error observed was 02-06 01:21.
+ ND/ARC job f./seg fault due to cvmfs problem.GGUS 91156 solved:have
  taken the affected nodes, ti9.uppmax.uu.se, ti11.uppmax.uu.se out of
  service,they are under investigation.
+ NL/IL-TAU-HEP still 0% eff.,FT f./DEST.+SOURCE:CONN.SRM.GGUS 91123
  waiting for reply:Lustre f/s problem fixed,Feb.6.Blacklisted in DDM,
  Savannah 135703,whitelisted Jan.8.
- FR/RO-07 1% eff.,817 FT f./SOURCE:HTTP_TIMEOUT:SRM.GGUS 91153 updated:
  Savannah 135683.Site asks to test,Feb.7,whitelisted,Feb.8.Time-out
  comes from 3 dpm-disks,Feb.11.The only solution is to put them
  offline, and migrate the data to other dpm-disks in the next days.
+ US/IllinoisHEP FT f./SOURCE:CONN.SRM.GGUS 91189 solved:due to 2
  problematic sites:R0-07,IL-TAU-HEP,f.stopped after blacklisting those.

2) Wed Feb.6
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
- DE/GoeGrid 163 FT f./DEST.:AsyncWait.GGUS 91195 in progress:there was
  overloading of the SRM door on the main storage server.The parameters
  has been tuned and the server has been restarted the same day.Still
  the same f.,Jan.8.
+ NL/SARA-MATRIX 700+ FT f./SOURCE:locality is UNAVAILABLE.GGUS 91198
  solved:dCache on 2 file servers has stopped,restarted.
+ DE/UNI-FREIBURG ~300 FT f./SOURCE:locality is UNAVAILABLE.GGUS 91200
  solved:rebooted hanging pool.No more problems in transfers,Feb.7.
+ DE/UNI-DORTMUND 400 job f./Put er.:Invalid argument lcg_cp.GGUS 91222
  solved:possibly not a "real" site error,but the lcg-cp bug,
  re-configured pilot factories.If the error vanishes,the problem is
  identified.No more f.,Feb.7.
+ DE/WUPPERTALPROD 2K+ FT f./DEST.:SRM_ABORTED.GGUS 91231 solved:no
  err.,Jan.7.Grid-se was deleting 90k files of dark data, seems that it
  was a bit busy.
+ DE/CSCS-LCG2 2K+ FT f./DEST.:CONN.:SRM.GGUS 91233 solved: the
  /pnfs mountpoint failed during the night, and it had to be remounted.
  But still ~2.5K|4h f.The same problem happened again,and it happens
  even with the /pnfs mounted,SRM stops working at some point.Have tried
  increasing the memory of the SRM domain, besides another couple of
  variables,and opened a ticket against dCache support.Service works
  after restart,Jan.8.

3) Thu Feb.7
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: -
        ............................................
+ UK/UKI-LT2-Brunel ~150 job f./SITEROOT not found in command output.
  GGUS 91234 verified:affected node wn-a3-20-00,VO_ATLAS_SW_DIR is
  setup by the site,corrected (was pointing to NFS,not CVMFS).
+ UK/UKI-NORTHGRID-SHEF-HEP 40% FT f./SOURCE:Unable to connect to
  lcgse2.shef.ac.uk.GGUS 91245,91274 verified:rfiod on 1 disk server and
  dpm-gsiftp on lcgse2.shef.ac.uk were down.Restarted the same day.
+ NL/TECHNION-HEP FT f./SOURCE+DEST.:CONN.:SRM.GGUS 91236 solved:4700+|
  4h f.Storm f/e server log became huge and filed the whole disk,fixed.
  Savannah 135766.
- ES/UAM-LCG2 PHYS-HIGGS ~1K FT f./DEST.:NO_SPACE_LEFT.Savannah 135764,
  100322,100375.

4) Fri Feb.8
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : DE/FZK
        ............................................
+ DE/UNI-FREIBURG again 2K+ FT f./failed to contact SRM.GGUS 91200
  reopened,solved:in DT,air condition failed.
+ DE/FZK-LCG2 7K+ FT f./SOURCE:Unable to connect to f01-060-112-e.gridka
  /GRIDFTP_ER:server err.426.GGUS 91316,91317 solved:efficiency ~42%
  only.1K+ T0 export FT f. to DATATATE token,GRIDFTP_ER.,Feb.9.Caused by
  a very obscure routing effect, which is triggered with a weird time
  lag.Fixed,transfer works the same day.
+ NL/TR-10-ULAKBIM ~900 FT f./SOURCE+DEST.:GRIDFTP_ER.:Unable to connect
  to torik2.ulakbim.gov.tr.GGUS 91285 solved:Dpm ftp server on both dpm
  disks have been stopped.Fixed.
+ US/AGLT2 ~1.3K FT f./SOURCE:GRIDFTP_ER./FIRST_MARKER_TIMEOUT.GGUS
  91286 solved:restarting dCache on two pool nodes and 3 VM door nodes
  solved the issue.

5) Sat Feb.9
      -> Cloud/site PROD efficiency ever<50% : ND
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
+ CA/CA-VICTORIA-WESTGRID-T2 300 FT f./SOURCE:AsyncWait.GGUS 91318
  solved:after DT some configuration was missing from some of pools.
  Fixed.Transfers succeed, the efficiency is back to 99% the same day.
- FR/BEIJING-LCG2 414 FT f./DEST.:FILE_EXISTS.GGUS 91319 assigned:
  efficiency ~22% only.Blacklisted in DDM:Savannah 135812.Upgrade DPM
  to new version,Feb.10.Unblacklisted for tests,Feb.11.
- FR/RO-07-NIPNE again 1K+ FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 91153
  updated:blacklisted in DDM again.3 dpm-disks' problem.The only
  solution is to put them offline,and migrate the data to other   dpm-disks in the next days.Savannah 135683:queues set offline.
- UK/UKI-LT2-RHUL_DATADISK 475 FT f./DEST.:SRM_ABORTED.GGUS 91323 in
  progress.
+ NL/JINR-LCG2 5298|h FT f./DEST.:CONN.:SRM.GGUS 91324 solved:
  transfers work the same day in several hours.

6) Sun Feb.10
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
+ CERN/CERN-PROD LSF doen't respond.GGUS ALARM 91329 verified:fixed
  the same day.
+ ES/IFIC-LCG2 1K+|4h FT f./DEST.:Request timeout due to internal err.
  GGUS 91327 solved:efficiency ~5% only.Affects DATADISK+CALIBDISK
  unavailable for last 18h.,Feb.11.Blacklisted in DDM:Savannah 135816.
  Srm server suffered a glitch,in service again,Feb.11.Whitelisted.

7) Mon Feb.11
      -> Cloud/site PROD efficiency ever<50% : DE/UNI-DORTMUND
      -> Cloud/site DDM  efficiency <50%: ND/SE-SNIC-T2,DE/UNI-FREIBURG
        .......................................................
- DE/UNI-FREIBURG 475 FT f./SOURCE:ThePinCallbacks Timeout.GGUS 91337
  assigned.
- ND/SE-SNIC-T2 FT f./SOURCE:locality is UNAVAILABLE/SRM_FILE_
  UNAVAILABLE.GGUS 91336 assigned.
+ ES/UAM-LCG2 FT f./DEST.:NO_SPACE_LEFT.GGUS 91360 solved:not a site
  problem.The space token is full and automatically blacklisted in DDM.
  Savannah 135833.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

++ 17.2.7.5.3 data12_8TeV Reco merge tasks 1170823,-0910,-1025,-1116,
 -1123,-1204,-1225 (Moriond gr.3) failures: JetGetterBadInput.Savannah
 135323:grop production JETMET.All jobs have failed up to 25 attempts in
 various clouds:FR,IT,ND,NL,TW,CA.Tasks ABORTED.Again the empty (for
 events/containers) input files.

 -+ 17.2.7.5.3 data12_8TeV Reco merge tasks 1166240,253,264,270,346,402,
 440,443,454,464,479,117427 (Moriond gr.2) failures: FATAL file
 "NTUP_TAU.01174277._000006.root.25" associated with stream
 "StreamNTUP_TAU" has exceeded the max file size of 10240MB.Savannah
 135809,39778:group production Tau.Many remaining jobs have failed up to
 25 attempts in a row each at different sites in US,UK,IT,CA,ES,DE,FR
 clouds as they exceeded the allowed size of output NTUP file.Tasks
 ABORTED,under investigation.

++ 17.2.7.5.3 Reco merge Moriond gr.6 tasks 1193078,1191568 failures:
 "Can't find object in event store".Savannah 135382:group production
 HIGGS.Many remaining jobs have failed up to 23 attempts at various
 sites in FR and DE clouds.Tasks set to FINISHED.

++ 17.2.7.5.3 data12_8TeV Reco MergeNTUP task 1194713 failures:TRF_ARG:
 Wrong number of inputNTUPXXXFile arguments: 0 instead of 1.Savannah
 135710:group production Exotics.All jobs have failed repeatedly in UK
 cloud.A wrong configuration in task definitions.Task ABORTED and will
 be re-submitted with correct configuration.

-+ 17.2.7.5.3 data12_8TeV Reco merge task 1194536 failures:TRF_UNKNOWN|
 AntiKt4TopoEMJetsReTagged_JetTagJetFilter_D3PDJ...Savannah 135769:group
 production BTAGGING.Multiple job failures (50 finished,547 failed) at
 different sites in US and CA cloud due to the FATAL error coming from
 JetTagJetFilterAlg.Task set to finished.See similar Savannah 135660 for  another task:part of this run 204442 (after LB 224) had solenoid off
 thus this failure.

-+ 17.2.8.3 mc12_8TeV evgen tasks 1195890-94 failures:TRF_UNKNOWN
 attempt to add a duplicate (TopAlg.Tauola) ... dupe.Savannah 135807:
 MC12 production - HIGGS.All jobs have failed repeatedly in IT,ES,ND,UK
 clouds.Tasks ABORTED.This error is due to two calls to Tauola - JOs
 need some modification, which the requesters are looking at.

-+ 17.2.7.5.7 data12_8TeV Reco merge tasks 1197260-62 failures:
 TRF_UNKNOWN attempt to add a duplicate (TopAlg.TopWGFilter) ... dupe
 ignored.Savannah 135817:group production TOP.All jobs failed multiple
 attempts at various sites running for CA,US,NL clouds.Tasks ABORTED.
 Looks like a problem with scanning the logs for errors?

-+ 17.2.5.2 mc12_8TeV evgen tasks 1197639-46,63-86 failures:TRF_UNKNOWN
 Unwanted ATOOLS::exception caught.Savannah 135819:MC12 production - SM.
 All jobs have failed repeatedly in UK and ND clouds due to the errors
 coming from Sherpa.Tasks ABORTED and resubmitted (with 64 bit).
 Hopefully, should fix the problem.

-- LeslieGroer - 2013-02-12

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2013-02-12 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback