TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130416 (2013-04-16, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-04-16

General News

  • ADC and ATLAS Computing News [20130416]
    • ADC Weekly meeting
    • Reprocessing Campaigns
      • 2012 physics delayed streams - full info at https://twiki.cern.ch/twiki/bin/viewauth/Atlas/Summer2012Reprocessing
        • B-physics
          • Output: AOD, TAG, HIST (same tags as for 2012 bulk reprocessing)
          • Limited merged AOD file size to 5 GB
          • Slice tests run for validation - look good
          • DQ webdisplay not ready yet, because of the Rucio naming scheme (screwed up script to find files on CASTOR)
        • HadDelayed
          • Requested by SUSY physics group
          • Output: AOD.HIST.DESDM_TRACK.DESDM_RPVLL.DESD_CALJET
          • Using release 17.2.10.1 - limited merged AOD file size to 5 GB
          • Running sliceTest tasks right now
      • 2011 2.76 TeV pp reprocessing - https://twiki.cern.ch/twiki/bin/viewauth/Atlas/Special2011Reprocessing2p76
        • Using release 17.2.8.11 (same as for the 2013 pp tier0 processing)
        • Improved beam spot from Express stream reprocessing
        • Outputs: ESD.AOD.NTUP_HI.TAG
        • Done
      • Draft schedule for run-2 preparation from the Offline Activity Coordination Board: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/Run2OfflineSchedule
        • 06/2013: Reprocessing of 2010 low-mu data (and minbias MC)
        • 04/2014: launch digi/reco for data challenge with release 19, full reprocessing of at least parts of the 2012 8 TeV data as required for the data challenges
    • Pilot Factories
      • nqueue -- how it is treated
        • APF 2 by design does not pay attention to nqueue.
        • In the case of CERN factories, the script which generates the factory config does consider AGIS 'nqueue' and modifies max_pilots_pending appropriately.
      • Messages to the cloud support squads, site admins
        • Setting parameters via curl should work as before, for now CEs with ce_flavour=LCG-CEs will not get pilots
        • Clouds who are going to deploy APF 2 should consider the above (nqueue treatment)
      • Longer term
        • Should we keep nqueue (and some other parameters) set by curl, or change the way to set it via AGIS?
    • AMOD
      • Long standing issue: intermittent failures in CERN FTS (credential delegation) importing data to CERN (see by CMS as well)
      • BNL was blacklisted for several hours on Sunday because failing analysis jobs (and AFT)
        • The local/setup.sh file was corrupted - No indication of Frontier server to be used
        • Job falls back to direct oracle access, which is disabled
    • Application to make UKI-­SOUTHGRID‐RALPP and UKI-­SOUTHGRID-BHAM-HEP T2Ds
      • Rates to and from TRIUMF still low (10MB/s at best, lots of times below 5MB/s)

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

      World-wide weekly ATLAS ADCoS summary (Apr.9-15,2013)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Apr.9-15,2013) Panda production service
      - completed successfully 1,987,987 managed group, MC production,
        validation and reprocessing jobs
      - average ~283,998 jobs per day
      - failed 112,987 jobs
      - average efficiency:
         -- jobs ~94.6%
      - active tasks: 1153
         -- distribution by cloud:
CA:72 CERN:11 DE:183 ES:73 FR:119 IT:57 ND:44 NL:97 TW:69 UK:227 US:201

B)  Major Downtimes:
     - IT/INFN-T1 DT (SE,SRM warning:Storm network interface), Apr.9.
     - IT/INFN-T1 DT (SE,SRM warning:power supply upgrade), Apr.11.
     - UK/RAL-LCG2 DT (SE,SRM,CE OUTAGE:network intervention), Apr.9.
     - UK/RAL-LCG2 DT (SE,SRM,CE warning:RAL-CERN OPN mainten.), Apr.10.
     - FR/IN2P3-CC DT (cccreamceli05,7,10 OUTAGE:migr.to EMI2), Apr.8-9.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:PostgreSQL security updates), Apr.10
     - ND/NDGF-T1 DT (SE,SRM warning:upgrade of one TSM instance),Apr.11
     - CERN/CERN-PROD DT (ce201-8 warning:NFS network reconf.),Apr.10-11
     - CERN/CERN-PROD DT (ce204 OUTAGE:upgrade to EMI2), Apr.15-22.

C)  Other news:
     - New DB release 22.7.1 distributed/subscribed to sites, Apr.10.
     - New DQ2 Clients stable release 2.4.0 announced, Apr.11.
     - New ATLAS release TRF caches were distributed:
       -- AtlasOffline    17.6.50 (64b), 17.7.0 (64b)
       -- AtlasProduction 17.3.10.1 (32+64b)
       -- AtlasPhysics    17.2.7.5.12 (32+64b)
       -- AtlasMCProd     -
       -- AtlasProd1      -
       -- AtlasTrigMC     -
       -- EvgenTP4MC11    -

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Apr.9
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ IT/INFN-ROMA1 squid down.GGUS 93201 solved:restarted.
+ IT/INFN-ROMA2 squid down.GGUS 93212 solved:fixed the same day.
- IT/INFN-ROMA2 many FT f./DEST.:checksum mismatch.GGUS 93216 assigned:
  still f.,Apr.10.Unfortunately,after updating to EMI2 version,there are
  several problems related to StoRM (ie: jobs can not write output file
  even though everything seems configured correctly).
+ IT/INFN-MILANO-ATLASC ~4K FT f./SOURCE:CONN.:SRM.GGUS 93203:cross-
  linked to GGUS 93184 solved:h/w issue fixed,side effect:clean up done
  to /var partition and restarted the srm service.
+ US/BU_ATLAS_Tier2 3800(~70%) job f./stage-in|out err.GGUS 93202 solved
  :fixed at about 4:00 a.m. EST by the site team.
+ US/NERSC LOCALGROUPDISK 9K+ FT f./GRIDFTP_ER.:an end of file occurred
  (possibly the destination disk is full).GGUS 93204 cross-linked to
  GGUS 92768 solved:due to full localgroupdisk,no errors,Apr.14.Savannah
  136950.
+ DE/GoeGrid 1500 FT f./SOURCE:AsyncWait.GGUS 93230 solved:SRM door was
  not available since 9:00 to 11:00 on 11.April,due to many errors.
  Restarted.No f.,Apr.13.
+ DE/HEPHY-UIBK FT f./SOURCE:CONN.:SRM.GGUS 93233 solved:started dead
  dpm daemon,Apr.10.

2) Wed Apr.10
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
- CERN/SLS monitoring for CERN-PROD_ATLASDATADISK,ATLASMCTAPE and
  CALIBDISK shows less than 100% availability.ELOG 4384,experts informed
  Just a monitoring issue.
+ ES/NCG-INGRID-PT frontier squid is down.GGUS 93235 solved:the log file
  /var/log/squid/access.log size grow to 100GB and the system f/s became
  full.Fixed,the service is online again the same day.
+ IT/INFN-ROMA1 squid down again since 4pm.GGUS 93201 reopened,solved:
  went down with no apparent error messages (or explicit shutdown),
  restarted.Will put in place a better monitoring and keep alive
  procedure.
+ IT/INFN-T1 still many FT+job f./DEST.:CKSM operation aborted/Put er.:
  file exist.GGUS 93085 solved:one of gridftp servers wasn't working
  properly,fixed.Recent transfers are succeeding, and jobs are no longer
  failing with stage-in errors,Apr.11.
+ FR/RO-07 still job f./GEt:System error in connect:No route to host.
  GGUS 93205 verified:still many job f.,Apr.10-11.A route problem in
  some nodes fixed,Apr.12.
- FR/GRIF-IRFU still many FT f./SOURCE:SRM_ABORTED.GGUS 93156(non TEAM)
  in progress updated:some SQL requests are "slow" on the DPM (10-20sec)
  and SRM timeouts,Arp.12.The proposal to raise the timeouts on the FTS
  side,Apr.15.
+ DE/CSCS-LCG2 FT f./TRANSFER_MARKERS_TIMEOUT.GGUS 93254 solved:fixed
  by the site,Apr.11.
- RU/RRC-KI-T1 1.2K+ job f./Replica with guid...not found.Savannah DDM
  101160:all of the input files are shown as 'ready' on the panda
  monitor pages,but the pilot cannot stage-in the inputs ("replica not
  found").Cloud squad and the site admin informed.

3) Thu Apr.11
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50% : -
        ............................................
+ CERN/Physics Database Service http://phydb.web.cern.ch/phydb/ not
  reachabale:403.14 - Default page missing.Elog 43816.Fixed:the correct
  link published: https://phydb.web.cern.ch/phydb/ATLAS.
+ IT/INFN-BOLOGNA-T3_PRODDISK->INFN-T1 ~500 FT f./SOURCE:REQUEST_TIMEOUT
  GGUS 93267 solved:Storm was restarted,no more failures the same day.
+ DE/GOEGRID->ND,UK FT f./SOURCE:GENERAL_FAILURE:Running Duration 0.GGUS
  93230 updated,solved:the SRM door was not available since 9:00 to
  11:00 on 11.April,due to many errors.Restarted.No more f.,Apr.13.
+ US/SLACXRD many FT f./canonical path/SRM errors.GGUS 93293 solved:site
  fixed the problem,Apr.12.
+ CA/TRIUMF ~2.5K FT f./SOURCE:SECURITY_ER.:SRM Authentication failed.
  GGUS 93294 solved:dCache head node had an unexpected load issue,fixed.
  prod|analysis have been reset to online the same day.

4) Fri Apr.12
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ UK/RAL-LCG2 job submission problem from the CEs to the batch farm
  (~1h),SUM tests have failed.Fixed the same day.Elog 43836.
+ FR/RO-07 ~20-30% job efficiency due to a faulty storage server.GGUS
  93205 verified:prod.and analy queues set offline.A route problem in
  some nodes fixed.Back online,Apr.13.Savannah 136988 solved.

5) Sat Apr.13
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : RU,TW
        ..............................
- UK/RAL-LCG2 ~100 FT f./DEST.:checksum mismatch.GGUS 93315 updated,
  waiting for reply:new 2354 FT f./SOURCE:SRM_ABORTED the same day.The
  filename finsishes .root.1.There is a file in the dq2 which finishes
  .root.2.So is the failing transfer for a file which was found to be
  corrupt and so was recreated via a new WN job?Need to know size of *.1
- FR/BEIJING-LCG2 1K+ FT f./SOURCE:GRIDFTP_ER.:Input/output error.GGUS
  92984 reopened in progress:other 300 f./DEST.:FILE_EXIST,Apr.14.

6) Sun Apr.14
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : RU
        ........................................
+ CERN/DB dashboard:PVSS2COOL in red.No data-taking activities,alarms
  from online operations should be ignored.Elog.43852,43856.
+ CERN/PandMonitor:Error 503 service unvailable.Elog.43848:seems
  temporary issue,disappeared soon after reporting.
+ DE/FZK-LCG2 ~300 FT f./DEST.:AsyncWait/TQueued.GGUS 93320 solved:no
  err.,Apr.15.
+ DE/MPPMU still 2300+ FT f./GRIDFTP_ER.:server err.53o:login incorrect.
  GGUS 93195 updated,solved:gPlazma misconfig caused file tranfer to
  lost sync with job monitoring.A complete dCache instance restart was
  necessary,Apr.15.
- US/BNL job f./problems accessing geometry data. local/setup.sh file
  change/update is under investigation.Elog 43854.
+ US/UPENN all 1626 FT f./DEST.:CONN.SRM.GGUS 93326 solved:bestman
  restarted the same day.

7) Mon Apr.15
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50%: -
        .......................................................
- IT/INFN-ROMA2-LOCALGROUPDISK many FT f./GRIDFTP_ER.:An end of file
  occurred (possibly the destination disk is full).GGUS 93327 assigned:
  the disk is not full,and not blacklisted for writing.
- DE/IEPSAS-Kosice set to brokeroff due to DT for CE tomorrow.Savannah
  137000.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.7.5.31 mc12_14TeV Reco merge task 1228717 failures: Too many/too
 large input files. Total file size 27935589616 B. Savannah 136866:group
 production Tau.At least 40% of jobs have failed in US cloud due to
 "Too many/too large input files" almost twice exceeding the limit
 allowed (~15GB).The same problem with t.1228721-36.Task are still in
 "submitting" state.

++ 17.2.2.2 mc12_8TeV Reco merge task 1230458 failures:Too many/too
 large input files. Total file size 52773814020 B. Savannah 136945:MC12
 production - JETETMISS.Job failed in TW cloud,total input files' size
 exceeded ~52GB.Task ABORTED:the AODs have calibration HITS in,so they
 are kind of bigger than usual.Besides this,the DB release is wrong.

++ 17.2.10.1 mc12_5TeV evgen task 1232803 failures: TRF_SVRINIT |
 "Unable to read requested LHE file: events.lhe !!" | "Pythia 8 init...
 Savannah 136942:MC12 production - HEAVYION.All jobs have failed
 repeatedly at ND.ARC-T2 due to the PYTHIA initialization error.Task
 ABORTED,a newer pythia8 version is needed.

++ 17.2.1.4 mc12_8TeV DigiMReco tasks 1232725-1232730 failures:FATAL
 ***  Wrong ATLAS layout: ATLAS-GEO-20-02-01_VALIDAT.Savannah 136943:
 MC12 production - EGAMMA.All jobs have failed repeatedly at various
 sites running for UK,US,ES clouds on the second Reco step.Tasks ABORTED
 and will be resubmitted with fixed setup.

-+ 17.6.0.4 mc12_valid simu task 1232704 failures: trans: Payload core
 dump.Savannah 101146:MC12 validation - New transform.All jobs have
 failed multiple attempts at various sites in DE cloud.Task ABORTED,
 this was a test for the new tf transform framework.

-- 17.2.7.5.3 mc12_8TeV Reco merge task 1232186 failures:Can't find
 object in event store for JetCollection(1162448536).Savannah 136956:
 group production JETMET.All jobs have failed up to 10 attempts each at
 various sites running for UK cloud.Task is in "failed" state,no
 comments published.

++ 17.2.7.4 data12_8TeV Reco task 1234375-6 failures: "GUID for ...
 .tar.gz not found in DQ2".Savannah 136962,101162:AOD reprocessing of
 physics_Bphysics periodM.All jobs have failed in CA,UK clouds with:
 GUID for 20070101._0216416.tar.gz not found in DQ2 and GUID for
 20070101._0216399.tar.gz not found in DQ2.Fixed: the run numbers to be
 reprocessed were provided and included in the Conditions DB Release.
 Tasks are "running".

-+ 17.2.8.7.2 mc12_valid DigiMReco task 1229535 failures: Payload ran
 out of memory trans.Savannah 101167:MC12 production - PHYSICS.All jobs
 but 1 have failed 7 attempts at BNL_PROD_MCORE.Task set to "finished",
 experts check why athenaMP task didn't work well.

++ 17.2.1.4.9 valid1 Merging+DigiMReco task 1235615 failures:TRF_EXC
 (63400) ERROR During execution of Merging_trf.Savannah 101174:
 McAtNloJimmy_CT10_ttbar_LeptonFilter.All jobs have failed up to 7
 attempts each at various sites in FR cloud due to the error on the 1-st
 Merging step related to DB-release file.Task ABORTED,the task owner
 will have to modify the tag.

++ 17.2.10.1.1 valid1 evgen task 1232678-9 failures:TRF_EXC | "OSError:
 (3, 'No such process')"|Payload stdout file too big.Savannah 101175:
 MC12 production - PHYSICS.All jobs have failed multiple attempts in
 DE,US clouds.Tasks ABORTED.Understood:packed up the wrong input files
 to the evgen job, to be resubmitted with the correct inputs.

-- 17.2.8.13 mc12_8TeV evgen tasks 1237060,61,64,65,67 failures:  ATH_FAILURE | "Signal 15 received." Reached maximum batch...Savannah
 136998:MC12 production - SUSY,SM.All jobs have failed repeatedly so far
 at various sites in US,FR,ND clouds as they reached the maximum batch
 wall time limit running over 24-30 hours.Tasks are still "submitting".

-- 17.2.10.1.1 mc12_valid DigiMReco task 1237541 failures: ATH_FAILURE -
 Athena non-zero exit.Savannah 101191:MC12 validation - Multicore memory
 All jobs have completed the 1-st digi step at US/BNL_PROD_MCORE, but
 failed on the 2-nd Reco step.Task is still "submitting".

-- LeslieGroer - 2013-04-16

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2013-04-16 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback