TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121106 (2012-11-06, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-11-06

General News

  • ADC and ATLAS Computing News [20121106]

    • Reprocessing 
  The bulk reprocessing corresponding to the periods A to G of
  2012 data has now started. The campaign will start with periods
  D and B, then followed by periods A, C, E and G. All streams
  will be reprocessed and AOD, TAG, HIST and stream-dependent
  DESDs will be produced. Note that ESDs will be produced only
  for the small streams, excluding the main physics streams.

  The reconstruction r-tag which is being used is r4065 which
  corresponds to:
     o Atlas release 17.2.7.4
     o Conditions tag COMCOND-BLKPA-006-07 and DBRelease 20.7.1.1

  More information about the fast reprocessing of 2012 data can be found at:
  https://twiki.cern.ch/twiki/bin/viewauth/Atlas/Summer2012Reprocessing
  which includes links to the watching pages corresponding to the different
  steps of the reprocessing. A global monitoring page can also be accessed
  directly from: http://atladcops.cern.ch:8000/j_info/
      • Started on Nov 2
      • Using a few T2D's and may add more (need data subscriptions to accomplish this)
      • Few T1 issues - mostly addressed and catching up. TRIUMF doing well (50%+ done of 128k assigned jobs)
      • Reprocessing took over all the MC pile slots
        • Plan: set the share on T1s to run 10% MC (but no evgen)
        • Push more pileup jobs to T2s but needs tuning
      • Competition between high priority MC and reprocessing discussed for a while
    • MC Production
      • Some work on pileup issues at various T1s has helped
      • Separation of pileup and group production jobs in bamboo (job scheduler) improved situation (smaller jobs, start when pileup ready)
    • AMOD Report
      • Several T1s were recovered for T0Export (though TRIUMF off to catch up due to BGP issue at CERN dropping them to 1Gbps)
      • FTS problem is under control
        • new patch was implemented at CERN FTS T0 and T2 servers and should propagate to all sites (Tier-1 sites deadline is the end of this week).
      • Bugs in pilots found just before reprocessing, new version 55b was released
        • bug preventing pilot from reporting discovered file corruption after stage-in
        • bug preventing stage-out to castor at CERN
      • CERN switched 10% of their WNs to EMI-2 - no problems seen so far

  • CA Cloud Squad Actions [20120626]
    • User DaTRI requests approved / processed (last week)
    • Declared Downtimes
      • Australia-ATLAS SCHEDULED downtime (OUTAGE) [04-11-2012 21:00 to 07-11-2012 07:00 UTC ] - Migrating all equipment to new HPC Research Data Hall
      • CA-VICTORIA-WESTGRID-T2 SCHEDULED downtime (OUTAGE) [07-11-2012 16:00 to 07-12-2012 01:00 UTC ] - Cluster maintenance and firmware updates 

ANALY_SFU_bugaboo Auto-Excluded at Tue Oct 30 09:31:36 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Tue Oct 30 19:00:46 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Wed Oct 31 06:30:59 2012 CET
ANALY_AUSTRALIA reset online at Wed Oct 31 08:30:10 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Wed Oct 31 15:00:55 2012 CET
ANALY_AUSTRALIA reset online at Wed Oct 31 17:00:14 2012 CET
ANALY_ALBERTA-WG1 Auto-Excluded at Thu Nov 1 01:30:53 2012 CET
ANALY_ALBERTA-WG1 reset online at Thu Nov 1 02:30:08 2012 CET
IAAS Auto-Excluded at Thu Nov 1 06:01:48 2012 CET
IAAS reset online at Thu Nov 1 06:30:59 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Thu Nov 1 08:00:47 2012 CET
 ANALY_AUSTRALIA reset online at Thu Nov 1 09:00:13 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Thu Nov 1 16:31:04 2012 CET
ANALY_AUSTRALIA reset online at Thu Nov 1 19:30:07 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Fri Nov 2 00:00:58 2012 CET
ANALY_AUSTRALIA reset online at Fri Nov 2 04:00:12 2012 CET
CA-VICTORIA-WESTGRID-T2 Auto-Excluded at Fri Nov 2 15:01:52 2012 CET
ANALY_MCGILL Auto-Excluded at Fri Nov 2 16:30:57 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Fri Nov 2 16:31:54 2012 CET
CA-VICTORIA-WESTGRID-T2 reset online at Fri Nov 2 17:01:03 2012 CET
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Fri Nov 2 16:31:54 2012 CET
ANALY_VICTORIA-WG1 Auto-Excluded at Fri Nov 2 21:00:47 2012 CET
ANALY_VICTORIA-WG1 reset online at Fri Nov 2 21:30:12 2012 CET
ANALY_MCGILL reset online at Sat Nov 3 05:30:16 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Sat Nov 3 05:31:08 2012 CET
IAAS Auto-Excluded at Sat Nov 3 23:31:45 2012 CET
 IAAS reset online at Sun Nov 4 01:00:55 2012 CET
 IAAS Auto-Excluded at Sun Nov 4 03:01:45 2012 CET
 IAAS reset online at Sun Nov 4 04:01:03 2012 CET
ANALY_SFU_bugaboo Auto-Excluded at Sun Nov 4 16:00:55 2012 CET
 SFU-LCG2 Auto-Excluded at Sun Nov 4 17:01:49 2012 CET
 IAAS Auto-Excluded at Sun Nov 4 18:31:54 2012 CET
ANALY_SFU_bugaboo reset online at Sun Nov 4 21:00:08 2012 CET
 SFU-LCG2 reset online at Sun Nov 4 21:00:56 2012 CET
IAAS reset online at Sun Nov 4 21:01:00 2012 CET
IAAS Auto-Excluded at Sun Nov 4 22:01:51 2012 CET
IAAS reset online at Mon Nov 5 00:00:59 2012 CET
ANALY_SCINET Auto-Excluded at Tue Nov 6 07:30:58 2012 CET
ANALY_SCINET reset online at Tue Nov 6 08:00:09 2012 CET

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Oct.30-Nov.5,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Oct.30-Nov.5,2012) Panda production service
      - completed successfully 1,408,805 managed group, MC production,
        validation and reprocessing jobs
      - average ~201,258 jobs per day
      - failed 159,721 jobs
      - average efficiency:
         -- jobs ~90%
      - active tasks: 3777
         -- distribution by cloud:
 CA:361 CERN:261 DE:275 ES:244 FR:665 IT:310 ND:156 NL:430 TW:230 UK:366
 US:479

B)  DownTimes:
     - DE/FZK-LCG2 DT (cream1-3 OUTAGE:maint.,EMI2 update), Oct.22-Nov.5.
     - DE/FZK-LCG2 DT (cream4-5 OUTAGE:draining,retirement), Nov.5-19.
     - UK/RAL-LCG2 DT (lcgce03-8 OUTAGE:EMI-2 CREAM repl.),Oct.23-Nov.30.
     - UK/RAL-LCG2 DT (SE,SRM OUTAGE:Oracle DB/SRM problems), Nov.4-5.
     - NL/SARA-MATRIX DT (SE,SRM,CE warning:dCache restart), Nov.1.
     - NL/SARA-MATRIX DT (SE,SRM,CE warning:replace 1 pool CPU), Nov.2.
     - ND/NDGF-T1 DT (SE,SRM warning:dCache upgrade,pdc.kth.se), Nov.5-6.
     - CERN/EOSATLAS update DT (2 hours), Oct.31.

   Other news:
     - Bulk reprocessing of 2012 data started,Nov.2.Details in Elog 40600
     - New pilot versions 55a,55b released Oct.31,Nov.1.
     - New DB Release 20.8.1 subscribed to sites, Nov.3.
     - New ATLAS releases and TRF caches were distributed:
       -- AtlasProduction - 17.2.7.4(32+64b)
       -- AtlasPhysics    - 17.2.5.2.4(32+64b)
       -- TrigMC          - 17.2.6.1.5(32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Oct.30
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : CA,US
        ........................................
- CERN/ATLAS-AMI DB unavailable/red in SLS.Experts informed.Elog 40525,
  40564,40570,40591-2,40642.The same issue Nov.1-2,4:availability 0%.
+ DE/GoeGrid 500+ job f./Put er.:SRM_REQUEST_INPROGRESS timeout.GGUS
  87858 updated,solved:the partition on the chimera node hosting the DB
  was filled up,it caused problems with Storage System.Fixed the same day
+ DE/FZK-LCG2<-UK 557 FT f./GRIDFTP_ER.:server err.426:conn.timeout.GGUS
  87924 solved:file in question was tried to be read with gftp-2.0
  and fileserver tried to contact the destination machine,but failed to
  do so.Somewhere in UK ping+traceroute got stuck.No errors on Nov.5.
+ CA/CA-MCGILL-CLUMEQ-T2->CERN FT f./GRIDFTP_ER.:globus_gass_copy_
  register_url_to_url transfer timed out.GGUS 87932,87927 solved:was a
  likely network glitch(?),no problem anymore,Oct.31.
+ CA/TRIUMF-LCG2->NDGF 200+ FT f./SOURCE:CONN.:SRM.GGUS 87936 solved:no
  more errors in the last couple of hours the same day.
+ CA/TRIUMF-LCG2,SFU-LCG2<-FZK FT f./SOURCE:proxy expired.GGUS 87954
  solved:should be the known issue of FTS,restarted tomcat.Will update
  FTS with the latest patch which should fix this problem.
+ CA/CA-ALBERTA-WESTGRID-T2 Frontier,Squid is down.GGUS 87926 verified:
  The squid service is running on a VM.There was something wrong with the
  bridge networking.Resolved the same day.
+ US/AGLT2 ~4K calib.FT f./GRIDFTP_ER.:server err.500:No route to host.
  GGUS 87952 verified:new storage node (192.41.236.64) had an
  unconfigured firewall in place blocking file movers,fixed the same day.
- US/BU_ATLAS_Tier2 still 100s deletion errors/Err.reading token data
  header.GGUS 87784 in progress:more err.(~290|4h),Oct.31.

2) Wed Oct.31
      -> Cloud/site PROD efficiency ever<50%  CERN,DE
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ CERN/TRIUMF SS red in SLS.Memory leak detected,service restarted.Elog
  40546-7,40555-7.
+ US/SLACXRD_PRODDISK 185 FT f./DEST.:CONN.:SRM.GGUS 87972 solved: 1K+
  new f./SOURCE:CONN.:SRM.Fixed the same day.
+ US/BNL some FT f.Due to mistake when updating the SRM DB configuration
  the connection SRM server - DB  was lost.Fixed.Elog.40558,40560.
+ US/NET2 1000s FT f./SECURITY_ER.:Permission denied.GGUS 87993 solved:
  SRM upgrade,reverted to the previous version.Left some files owned by
  root instead of usatlas1,repair done.No err.,Nov.2.
- US/NERSC blacklisted untill issues with file deletion (SCRATCHDISK
  cleanup) are solved.Savannah 131508,97059.
- DE/FZK-LCG2->UKI-SCOTGRID-ECDF FT f./GRIDFTP_ER.:server err.426:conn.
  timeout.GGUS 87958 in progress:traceroot ECDF->FZK published.FZK
  ticketed to get SRM in their bypass.
+ DE/DESY-HH<-CERN FT f./DEST.:Marking Space as Being Used.GGUS 87978
  solved:transfers are done,no new errors for the last 24h,Nov.1.
+ UK/UKI-NORTHGRID-LIV-HEP many FT f./DEST.:CONN.:SRM.GGUS 87994 solved:
  network interruption caused the dpm service on hepgrid11 to segfault.
  The service has been restarted,Nov.1.

3) Thu Nov.1
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: -
        ............................................
- ND/NDGF-T1 few 100s FT f./GRIDFTP_ER.:server err.426:broken pipe.GGUS
  87999 reopened:~1300 f./AGENT:INTERNAL_ER.:Failed to process the
  transfer in [5]seconds/GRIDFTP_ER.:server err.426:File doesn't have
  expected length,Nov.2.More f.on Nov.4-5:GRIDFTP_ER.:server err.426:
  Conn.timeout.The dCache 2.3.x bug identified.Plan for the upgrade to
  2.4 n.
+ DE/FZK-LCG2 ~1100 (350 repro) job f./Get er.:Input file staging timeout
  GGUS 88008 solved:disk pool recovered,cross-linked to GGUS:87776.
+ DE/TUDresden-ZIH FT f./DEST.:CONN.:SRM.GGUS 88007 solved:had some
  trouble with the site,fixed the same day.
+ UK/UKI-NORTHGRID-LANCS-HEP FT f./DEST.:SRM_ABORTED.GGUS 88015 verified:
  all 470 FT f./SOURCE:CONN._ERR.:couldn't open conn.to fal-pygrid-30.
  lancs.ac.uk.Also failing Nagios test+excluded by HC.Issue with yum
  installed the latest version of dpm, which requires a reconfigure.
  Autoupdate on SE disabled,Nov.2.The problem turned out to be with
  reverse dns lookup on the SE.The cause of this is unknown but managed
  to put a workaround in place.
+ IT/INFN-T1 Frontier|squid monitoring problem.GGUS 88016 verified:had a
  problem in nagios event handlers,since changed ip address for one of
  the two squids.Fixed,Nov.5.

4) Fri Nov.2
      -> Cloud/site PROD efficiency ever<50% : CERN
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ CERN/CERN-PROD Inaccessible files on CASTOR/T0ATLAS ~12h.GGUS ALARM
  88064 solved:Diskserver has been repaired and the files are accessible
  again,Nov.5.
+ CERN/ATLAS-DDM vobox ss-t0 degraded & no T0 export information.VM had a
  problem,solved by restarting dq2.DQ2 was not properly restarted after
  reboot earlier in the evening.Elog.40610-40612.
+ UK/RAL-LCG2 Frontier Service broken on hosts lcgvo01.gridpp.rl.ac.uk
  for ~1.5h,resolved.Elog.40585,40587.
- UK/RAL-LCG2 one disk server problems caused some issues:stalled Raw
  data export,stage-in job failures.Elog.40586,40596,40598.backlog,but
  transfres go slowly,Nov.3.Elog.40622-24.Keep investigating.
- FR/GRIF-LAL<-TW 101 FT  f./DEST.:CONN.:SRM.GGUS 88044 assigned.
+ ES/PIC NTUP_SUSYBOOST root file not available.GGUS 88073 solved:the
  file is recovered,Nov.5.
- ES/PIC->CERN FT f./SOURCE:file doesn't exist.Savannah 98620.Similar to
  Savannah 98602 ?
+ NL/IL-TAU-HEP 200+ FT f.DEST.:StoRM encountered an unexpected error!
  GGUS 88090 solbed:still ~1200 same FT f.,Nov.4.Files transferred,Nov.5.
- IT/INFN-ROMA1 400+ job f./Exception caught by pilot watchdog:[Errno 10]
  No child processes.GGUS 88092 assigned:a few nodes had problems after
  an interruption with LSF,restarted Nov.3.Site in DT,Nov5.

5) Sat Nov.3
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : TW,IT
        ..............................
+ CERN/T0->DATADISK,DATAPREP:all T0 export f./GRIDFTP_ER.:server er.500:
  open/create err.:Operation canceled.GGUS 88107,88108 verified:also FT
  f./SOURCE:CONN.:SRM.EOSATLAS was down ~12:00-15:30 when it restarted,
  set up some monitoring to auto-restart next time this is happening.
- IT/INFN-LECCE 400 FT f./SOURCE:CONN.:SRM.GGUS 88109 assigned:still the
  same err.+~1K deletion err./couldn't open conn.to se-1.le.infn.it,Nov.5
- NL/SARA-MATRIX again 10K+ FT f./SRMV2STAGER:StatusOfBringOnlineRequest:
  SRM_FAILURE:Failed to pin file.GGUS 87531 in progress since Oct.18
  updated:new 14,600|h FT f.,Nov.4.High load on the SE?
- ES/IFIC-LCG2->CERN FT f./SOURCE file doesn't exist.GGUS 88098 assigned:
  Savannah 98620.
- UK/UKI-SOUTHGRID-RALPP<-IN2P3 FT f./No transfer markers received for
  more than 300 seconds.GGUS 88099 assigned:same ~100 f.,Nov.4.
+ DE/FZK-LCG2 FT f./globus _ftp_client: java.net.Connect Exception.GGUS
  88100 verified:no problem,Nov.5.
- FR/RO-02-NIPNE FT f./DEST.:SRM_FAILURE:Failed for all SURLs.GGUS 88102
  in progress.
+ US/BNL-ATLAS many FT f./First non-zero marker not received within 600
  seconds.GGUS 88103 verified:transient errors.

6) Sun Nov.4
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : UK
        ........................................
+ IT/INFN-FRASCATI 1K+ FT f./SECURITY_ER.:The available CRL has expired.
  GGUS 88113 verified:crl update cron didn't w. due to a failure in the
  migration from gLite to EMI(old cron fetch-crl replaced the new one).
  Fixed:all disk server have the right cron,crl updated the same day.
- IT/INFN-ROMA3 ~70 FT f./DEST.:CONN.:SRM.GGUS 88115 assigned:still 160
  FT f.,Nov.5.
+ US/AGLT2 ~1300 FT f./SOURCE:Trusted Certification Authority" has
  expired.GGUS 88114 verified:2 new dCache doors(VMs) 'rsync'doesn't seem
  to be running in cron.Fixed the same day.
+ UK/RAL-LCG2<-CERN many FT f./DEST.:CONN.:SRM.GGUS 88117 verified:DB
  problems have been fixed and Atlas services were restarted the same day
- UK/UKI-SCOTGRID-ECDF 3300|4h FT f./DEST.:SRM_ABORTED.GGUS 88119 in
  progress:issue with 1 disk server "pool5".Still ~700 deletion err./SRM_
  INTERNAL_ERR.:timed out,Nov.5.The machine is up (since Nov.5 afternoon)
  so transfers (and deletions) should work.However the cause of the
  collapse is still unknown so there may be some firmware updates over
  the next few days in an attempt to stabilise the machine.This may cause
  a few errors,so the ticket can be open until (say) the end of the week.

7) Mon Nov.5
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ CERN/CERN-PROD TO export f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FILE_UNAVAILABLE.GGUS 88134 solved:same disksaver as in GGUS:88064,
  accessible again.
+ DE/HEPHY-UIBK ~100|4h deletion err./Err.reading token data header:Conn.
  reset by peer.GGUS 88124 solved:the dpm daemon was stopped,fixed.
+ FR/IN2P3-CC `200|4h FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE.GGUS 88125,88121 verified:Atlas staging pools were disabled
  this morning (repository inconsistency due to a bug in dCache - ticket
  already opened at dcache.org).Those pools are online now.
- FR/IN2P3-LAPP LOCALGROUPDISK destination disk full.Savannah 133568.
+ FR/IN2P3-LPC 1000s FT f./DEST.:CONN.:SRM.GGUS 88176 verified:SRMV2.2
  service had crashed,restarted.
- CA/Australia-ATLAS FT f./DEST.:CONN.:SRM.GGUS 88152 in progress:
  Scheduled DT until Wednesday.
- NL/SARA_MATRIX many FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE:Failed to pin file.GGUS 88174 in progress.
- NL/IL-TAU-HEP still FT f./DEST.:StoRM encountered an unexpected error!
  GGUS 88090 reopened:changed priority to urgent,some tr. f. 24times|24h.
- IT/INFN-ROMA2 383|4h deletion err./SRM_INTERNAL_ERR.:Client transport
  failed to execute the RPC.libcurl failed to execute the HTTP.GGUS 88204
  assigned.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.3.7.1 data12_8TeV Reco merge task 1015189,1015190,1016479,1022523
 ,1023108,1023131,1023161,1023187,1023832 failures:Segmentation fault,
 address not mapped to objest.Savannah 133414:group production Tau.All
 jobs have failed up to 25 attempts each at various sites running for
 CERN cloud.Tasks are still in "submitting" state.No comments published
 so far.

++ 17.2.3.5.1 mc12_8TeV Reco merge tasks 1022733,1022737-8 failures:"ATH_
 JOP_NOTFOUND|"IncludeError:include file preInclude=JetTagD3PDMaker/Jet
 TagD3PD_smearTracks.py can not be found".Savannah 133415:group
 production B-TAGGING.All jobs have failed multiple attempts at various
 sites running for US,FR clouds.Tasks ABORTED:wrong tag requested.

++ 17.0.3.4+17.0.6.2 mc11_2TeV Digi+Reco task 1024112-3,1024115-6
 failures:an unknown exception occurred:This job will try to override
 pile-up luminosity configuration,but no pile-up will be set up!Savannah
 133416:MC11 production - HEAVYION.All jobs have failed multiple attempts
 in ES and NL clouds.Tasks ABORTED and fixed.

-- 17.2.5.2.1 data12_8TeV Reco merge task 997153 failures:Reached maximum
 batch system time limit/lost heartbeat.Savannah 133418:group production
 EGAMMA.Many jobs failed repeatedly while in some cases run for >3 days
 at various sites running for CA cloud(1697 job failures,only 80 jobs
 completed).Task is still "running",experts informed.

-+ 17.2.5.2.2 data12_8TeV Reco merge tasks 1023821-1023829 failures:TRF
 _UNKNOWN | "Standard std::exception is caught handling incident
 0xffe8185c" | "CaloCondBlobBase::getAddress: Index out of range: 33792.
 Savannah 133396,98505:group production JetMET,SM WZ.Many jobs failed
 multiple attempts at CERN-PROD.The same problem with many other tasks:
 1023111 1023120 1023134 1023135 1023197 1023213 1023302 1023310 1023317
 1023328 1023818 1023819 1023834 1023905 1024092 1024202 1024203 1024204
 1024206 1024229 1024230 1024231 are failing at various sites/clouds.
 Experts had identified the s/w problem,stopped the open-ended production
 ,notified the group contact.But the 'obolete' tasks are still RUNNING in
 etask.This needs to be synced somehow.

-+ 17.2.7.3 data12_8TeV Reco task 1024814 failures:input GUID not found.
 Savannah 98558:DESD and DAOD from DRAW for Slice-test Bulk reprocessing.
 10K+ reprocessing jobs failed at BNL and TRIUMF.Seems related to some
 DRAW ntups having no selected events and therefore not just empty,but
 missing.Failing jobs ABORTED,the task set to "finished".

-+ 17.2.6.3 mc12_8TeV evgen tasks 1028165,1027858,1027624 failures:TRF_
 UNKNOWN|"something wasnt write in file name transfer from the fragment."
 |"qcut was not sucessfully transferred from the fragment."RuntimeError:
 No 'madgraph...evts' file found.Savannah 133506:MC12 production - SUSY.
 Al jobs have failed multiple attempts in UK and US clouds.Tasks ABORTED,
 experts investigate.

-+ 17.2.3.7.1 data12_8TeV Reco merge task 1024920 failures:The dataset
 data12_8TeV.00213479.physics_Egamma.merge.log.f482_m1239_p1130_tid
 01024920_00 already exists!Standard std::exception is caught handling
 incident0xffdb089c" | "CaloCondBlobBase::getAddress: Index out of range:
 33792 | "CaloCondBlobBase::getAddress: Index out of range: 33792 >.
 Savannah 133569:Group production Tau.All jobs have failed 25 attempts in
 a row at various sites running for CA cloud.Similar to Savannah 133396?
 Task is set to "obsolete".

-- 17.2.4.6.2 mc12_8TeV Reco merge task 994097 failures:Could not
 register location:DQRepository Exception:Unknown dataset exception!
 Savannah 133570:group production SUSY.All jobs have failed 11 attempts
 at various sites running for DE cloud.Task is still "submitting".
 Similar to Savannah 133396? But another release,other Mc12 group prod.

-- LeslieGroer - 2012-11-06

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-11-06 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback