TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120821 (2012-08-21, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-08-21

General News

  • ADC and ATLAS Computing News [20120821]
    • Special XP_Higgs production (2.3M events) going slowly due to priority (simul should be done)
    • ADC Weekly meeting
    • Short meeting this week.
    • MC Production report https://indico.cern.ch/getFile.py/access?contribId=1&sessionId=0&resId=0&materialId=slides&confId=204759
      • Sherpa 1.4.1 - almost done with redo of AF-II MC but only about 25% of Fullsim
    • Nothing special from AMOD report
    • Postponed discussion of deletion of obsolete group production datasets
    • "Honourable" mention for CA-SCINET-T2 and CA-VICTORIA-WESTGRID-T2 in T2D report as having degraded SONAR performance - cloud support needs to respond
      • Others mentioned : FR: BEIJING-LCG2, IN2P3-LAPP, IN2P3-LPSC; IT: INFN-MILANO-ATLASC; UK: UKI-SCOTGRID-ECDF; US: SWT2_CPB?
T1-to-SCINET.png
SCINET-to-T1.png
T1-to-VICTORIA.png
VICTORIA-to-T1.png 
ANALY_AUSTRALIA reset online at Wed Aug 15 16:00:13 2012
ANALY_AUSTRALIA Auto-Excluded at Wed Aug 15 19:02:17 2012 CET
ANALY_AUSTRALIA reset online at Wed Aug 15 21:00:57 2012
ANALY_AUSTRALIA Auto-Excluded at Mon Aug 20 03:33:12 2012 CET
ANALY_AUSTRALIA reset online at Mon Aug 20 05:00:53 2012
ANALY_AUSTRALIA Auto-Excluded at Mon Aug 20 18:02:10 2012 CET
ANALY_AUSTRALIA reset online at Mon Aug 20 20:01:32 2012
ANALY_AUSTRALIA Auto-Excluded at Tue Aug 21 19:02:37 2012 CET

ANALY_MCGILL Auto-Excluded at Fri Aug 17 04:31:56 2012 CET
ANALY_MCGILL reset online at Fri Aug 17 06:03:47 2012
ANALY_MCGILL reset online at Mon Aug 20 12:31:25 2012
ANALY_MCGILL Auto-Excluded at Mon Aug 20 16:33:14 2012 CET

ANALY_SFU_bugaboo Auto-Excluded at Tue Aug 21 17:47:03 2012 CET

ANALY_VICTORIA-WG1 Auto-Excluded at Tue Aug 21 12:02:07 2012 CET

Australia-ATLAS Auto-Excluded at Tue Aug 14 09:49:41 2012 CET
Australia-ATLAS reset online at Tue Aug 14 11:33:24 2012
Australia-ATLAS reset online at Wed Aug 15 15:07:51 2012
Australia-ATLAS Auto-Excluded at Thu Aug 16 18:02:34 2012 CET
Australia-ATLAS reset online at Thu Aug 16 19:35:06 2012
Australia-ATLAS Auto-Excluded at Sun Aug 19 06:11:01 2012 CET
Australia-ATLAS reset online at Sun Aug 19 08:09:19 2012

CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Fri Aug 17 04:33:37 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Fri Aug 17 06:09:03 2012
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Mon Aug 20 01:41:49 2012 CET
CA-MCGILL-CLUMEQ-T2 reset online at Mon Aug 20 12:35:53 2012
CA-MCGILL-CLUMEQ-T2 Auto-Excluded at Mon Aug 20 16:37:49 2012 CET

CA-VICTORIA-WESTGRID-T2 Auto-Excluded at Tue Aug 21 19:06:08 2012 CET
CA-VICTORIA-WESTGRID-T2 reset online at Tue Aug 21 20:33:17 2012

IAAS Auto-Excluded at Fri Aug 17 09:38:27 2012 CET
IAAS reset online at Fri Aug 17 18:03:07 2012
IAAS Auto-Excluded at Tue Aug 21 09:43:56 2012 CET
IAAS reset online at Tue Aug 21 11:33:47 2012
IAAS Auto-Excluded at Tue Aug 21 13:41:09 2012 CET

SFU-LCG2 Auto-Excluded at Tue Aug 21 19:33:01 2012 CET

  • Security News [20120801]
  • gLite/EMI News [20120801]
    • EGI CSIRT ADVISORY [EGI-ADV-20120801] concerning gLite 3.2 middleware components no longer supported on 01 August 2012.

  • ROC_Canada News [20120710]
    • Current shift - Andrew Wong 7-10 Aug, 2012
    • Next Week - Sergey Chelsky

  • WLCG and General Computing News [20120221]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Aug.14-20,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Aug.14-20,2012) Panda production service
      - completed successfully 1,147,008 managed group, MC production,
        validation and reprocessing jobs
      - average ~163,858 jobs per day
      - failed 54,269 jobs
      - average efficiency:
         -- jobs ~95,5%
      - active tasks: 1310
         -- distribution by cloud: CA:56 CERN:92 DE:137 ES:64 FR:151 IT:78 ND:20 NL:186 TW:68 UK:140 US:318

B)   - ND/NDGF-T1 DT(SE,SRM warning:Power outage in Sweden,1 site),Aug.14
     - ND/NDGF-T1 DT (SE,SRM warning:pool updates in Linkoping), Aug.15.
     - ND/NDGF-T1 DT (SE,SRM warning:dCache pool updates in Oslo),Aug.17.
     - ND/NDGF-T1 DT (SE,SRM warning:network maintenance,1 s.),Aug.20-21.
     - ES/PIC DT (CE-10 OUTAGE:migration from gLite to EMI-1), Aug.14-17.
     - ES/PIC DT (CE-07 OUTAGE:migration from gLite to EMI-1), Aug.17-20.
     - ES/PIC DT (CE-08 OUTAGE:migration from gLite to EMI-1), Aug.20-22.

     - CERN SAM central grid-monitoring instance upgrade is ongoing.
     - New OSG Software version 3.1.8 released, Aug.14.
     - New pilot version 54a announced and released, Aug.15-16.
     - New TRF caches were distributed:
       -- EvgenTP4MC11 17.2.4.7.1(32b),
       -- AtlasPhysics 17.2.4.6.2(32+64b),
       -- AtlasProduction 17.2.4.8(32+64b),17.0.6.18(32+64b),
       -- TrigMC 17.2.4.4.1(32b).

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Aug.14
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ CERN/SS11(ss-bnl) showed degraded availability 30%.The agent was stuck.
  Fixed the same day.Elog.38423,38425-26.
+ TW/Taiwan-LCG2 many job f./Put err.:Invalid argument lcg_cp.GGUS 85138
  solved:had castor glitch few hours ago,solved.No more err.(12h),Aug.15.
+ NL/WEIZMANN-LCG2 2K FT f./DEST.:CONN.:SRM.GGUS 85139 solved:site in SDT
+ ND/ARC 700 job f./LRMS ere.:job killed/staging:Failed to prepare
  destination srm/Err.downloading tarball JO.GGUS 85144 solved:not a site
  problem.2 failure types:exceeded walltime for pile-up (maxCpuCount too
  low);badly defined evgen.
+ DE/UNI-SIEGEN-HEP 1K+/4h deletion err./Error reading token data header:
  Connection reset by peer.GGUS 85194 verified:declared the files as
  deleted to the deletion service;also changed the SE in ToA.Deletion
  backlog back to zero,Aug.16.
+ DE/LRZ-LMU FT f./SOURCE:SRM_ABORTED.GGUS 85195 verified:some dCache
  problems solved?No more err.in the last 12h,Aug.15.The exact details
  are described in dCache bug #7317 and #7300.

2) Wed Aug.15
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ CERN/CERN-PROD_ATLASLOCALGROUPDISK EOS all FT f./DEST.:CONN.:SRM.GGUS
  85201 verified:trouble with commands: /usr/sbin/eos-srm-max-bytes
  /eos/atlas/atlasgroupdisk exception:java.io.IOException: Cannot run
  program "/bin/sh": java.io.IOException: error=24, Too many open files.
  SRM stuck,Bestman restart seems resolved this(was running out of file
  descriptors when talking to EOS).
+ FR/IN2P3-LPC all FT f./DEST.:CONN.:SRM.GGUS 85207 solved:air cooling
  system has crashed,fixed.Savannah 131207:whitelisted after fixing.
- US/UTA_SWT2_PRODDISK FT f./DEST.:CONN.:SRM.GGUS 85225 on hold:this
  problem due to several factors:1)type of jobs currently running in the
  cluster do a lot of staging,and the files are predominately on a
  storage node that is heavily loaded;2)There seems be a fair amount of
  central deletions occurring;3)issue with how checksums are handled is
  adding to the load on the storage node.In the short term reduced
  the number of running jobs to ease contention in the storage system.
  Longer term modifications to the local site mover will be made.The site
  will be off-line sometime in the next few days for a migration of the
  local LFC to one at BNL.FT f./SOURCE file/user checksum mismatch,Aug.20
  due to the storage server went down a couple of days ago and the files
  being written during the failure are now corrupted,declared lost to
  deletion recovery service.Savannah 131312.

3) Thu Aug.16
      -> Cloud/site PROD efficiency ever<50% : CERN(t.942331 f.)
      -> Cloud/site DDM efficiency <50%: -
        ............................................
- IT/INFN-BOLOGNA-T3 many FT f./DEST.:CONN.:SRM.GGUS 85227 assigned:
  similar f. appear again.many HTTP_TIMEOUT:SRM f.,Aug.19.Blacklisted in
  DDM:Savannah 131326.Site is aware about this problem,most of the site
  people are on vacation.
- NL/WEIZMANN-LCG2 FT f./DEST.:CONN.:SRM.GGUS 85231,84722 in progress:
  still job f./Get+Put err.:could not open conn.to wipp-se.weizmann.ac.il
  .Found a solution to the Lustre attribute bug in Lustre,to be
  implemented in the coming days.Will allow STORM's on-the-fly check-
  summer to run instead of the old check-summer.The old check-summer ran
  in a 2nd pass after the transfer was complete, but before the "PutDone"
  Depending on file size and other SE activity, it often completed late
  enough to cause a time-out.
- FR/GRIF-IRFU<-TAIWAN-LCG2 still FT f./GRIDFTP_ER.:server err.500.GGUS
  84626,83757 in progress updated:also many FTS job f. in ATLAST1-IRFU.
  Seems something wrong in network level,waiting for result of traceroute
  from IRFU to do further debugging.
+ FR/IN2P3-CPPM_PRODDISK 100+ FT f./SOURCE:Unable to connect to mardisk07
  .in2p3.fr.GGUS 85268 solved:had a h/w issue,mardisk07 back online.
- DE/PSNC still FT f./DEST.:CONN.:SRM.GGUS 85199 updated,waiting for
  reply:check and remount storage.Job f./put err.One bad WN fixed,Aug.20.
- DE/GOEGRID FT f./SRM:Missing routing entry for se3-goegrid_1@local
  GGUS 85265 assigned:also SUM test f.(VO cert.in se-goegrid?).Job f./put
  er.,Aug.18.Blacklisted for writing to DATADISK,Savannah 131293.
+ TW/TAIWAN-LCG2 again many FT f./SRM_FAILURE:Unable to select castorFile
  by fileId :ORA-20010: File class 1 not found in database ORA-06512: at
  "STAGER.SELECTCASTORFILE".GGUS 85138 reopened,solved:mistake when doing
  castor operation so castor file class id became invalid.Fixed,no
  err.,Aug.17.
+ UK/UKI-SOUTHGRID-BHAM-HEP ~700 FT f./DEST.:CONN.:SRM.GGUS 85262:put
  into DT,changing over one of RAID controllers.Back to normal after that

4) Fri Aug.17
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : CERN
        ............................................
- CERN/CERN-PROD many FT f./SOURCE:GRIDFTP_ER.:globus_ftp_client:the
  operation was aborted.GGUS 85269 on hold:T0->CERN-PROD FT f./DEST.:
  CONN.:SRM/server err.530:Mapped user 'atlas003' is invalid/INVALID_SIZE
  empty file size returned.Been hit by the sudo/SElinux bug that changes
  permissions on /etc/nsswitch.con.Fixed.Most of the former 0-size files
  should now have the correct size again,56 files that need further
  manual action(list created).They aren't necessarily lost,but will be
  inaccessible now.
+ CERN/CERN TZERO FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_FILE_
  UNAVAILABLE:All copies of this file are unavailable for now.GGUS 85298
  solved:the diskserver hosting the file was down for about an hour on
  the 17th in the evening.File become accessible again after the machine
  had been rebooted.
+ ND/NDGF-T1 800+ job f./Failed in data staging.Failed to prepare
  destination srm/Failed checking source replica file/Cant read from
  source;Output file.GGUS 85292 solved:most of the failed jobs (750)
  have been canceled(killed) on a cluster(grad.uppmax.uu.se),which has
  serious problems.This site is disabled now.Jobs succeeded in a later
  run,Aug.18.

5) Sat Aug.18
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
+ US/SWT2_CPB 289 FT f./DEST.:CONN.:SRM.GGUS 85299 solved:storage server
  went off-line due to a problem with the NIC in the machine.Fixed.
+ US/SLACXRD->TRIUMF many FT f./SOURCE:SECURITY_ER.:not mapped.GGUS 85306
  solved:fixed,no errors,Aug.19.
+ DE/LRZ-LMU_DATADISK FT f./GRIDFTP_ER.:server er.451:Operation failed:
  Best pool too high.GGUS 85301 verified:dcache had troubles with space
  calculation,added some pools,corrected size info,Aug.20.Savannah 131292
- ES/IFIC-LCG2 all FT f./DEST.:CONN.:SRM.GGUS 85302 in progress:double
  disk failure on a RAID5 disk group affected ~3300 files in DATADISK.
  Again ~100% f.,Aug.20.Disk server crashed during recovery.Restarted.
  Some data still not accessible (~1000 files).
- IT/ZA-WITS-CORE->INFN FT f./SOURCE:Authorization denied:The expected
  name for the remote host (host@146.141.240.93) does not match the
  authenticated name of the remote host (se.core.wits.ac.za).GGUS 85303
  assigned.Blacklisted in DDM:Savannah 131294.

6) Sun Aug.19
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
- DE/UNI-FREIBURG job f./Get.:failed to get LFC replicas.GGUS 85308 in
  progress:list of bad WNs created.Some WNs had outdated CRLs.Recent jobs
  to ANALY_FREIBURG look fine,Aug.20.
- NL/AM-04-YERPHI FT f./DEST.:CONN.:SRM.GGUS 85309 assigned.
+ NL/SARA-MATRIX_MCTAPE FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE:Failed to pin file.Request clumping limit reached.GGUS 85311
  solved:next attempts succeed,no problem with the buffer size.
- IT/INFN-LECCE FT f./CONN.:SRM.GGUS 85310 assigned:100% failure rate,
  blacklisted:Savannah 131306.
- UK/UKI-SCOTGRID-GLASGOW many job f./Get:failed to get LFC replicas.GGUS
  85312 in progress.

7) Mon Aug.20
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : CERN
        .......................................................
- CA/CA-MCGILL-CLUMEQ-T2 100% FT f./DEST.:INVALID_PATH/StoRM encountered
  an unexpected error.GGUS 85060 reopened.Savannah 131321:blacklisted.
+ DE/DESY-HH FT f./SOURCE:AsyncWait/pool error Timeout.GGUS 85319 solved:
  issue with the DCache Pin Manager.The utility Domain was running crazy.
  It is restarted now and the file in question can be accessed.
- DE/LRZ-LMU job f./Too little space left on local disk to run job.GGUS
  85354 assigned,Savannah 131352.
- TW/Taiwan-LCG2 job f./Put err.:SRM_FAILURE.GGUS 85320 in progress:
  problem was caused by Oracle DB hung due to too many dead processes
  from Stager server.After restart Stager service,service seems back.
- UK/UKI-LT2-RHUL job f./lost heartbeat.GGUS 85321 in progress:some
  problem with s/w area is under investigation.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

++ 17.2.4.6.1 data12_8TeV MergeNTUP tasks 938872,938742,etc failures:
 TRF_ARG:Wrong number of inputNTUPXXXFile arguments:0 instead of 1.
 Savannah 131168:group production SM direct photon.All jobs have failed
 repeatedly in various clouds.Understood:no p1173 tasks submitted but
 only p1174 and they are not correct.Tasks ABORTED,new ones submitted
 with p1173 tag.

-+ 17.2.4.6.1 data12_8TeV MergeNTUP tasks 942731-942888 failures:TRF_ARG:
 Using different input and output types.Savannah 131238,131267:group
 production SM direct photon.All jobs have failed multiple attempts after
 the resubmission in various clouds due to the error which sounds as a
 bug(typo?) in the task/job definition of the input/output files.Tasks
 ABORTED,to be resubmitted with newer p-tag.

-- 17.2.3.7.1 data12_8TeV MergeNTUP tasks 937082 failures:TRF_OUTFILE_
 TOOFEW:Transform output file contains too few events.Savannah 131298:
 group production Higgs HSG5.Many jobs failed up to 25(!) attempts at
 different sites in US cloud.Task is still "running".No comments added so
 far.

++ 17.2.4.7 data12_8TeV Reco task 942331 failures:TRF_UNKNOWN|"Duplicate
 request for folder /TRIGGER/LVL1/BunchGroupContent associated to already
 requested Storegate key.Savannah 131220:group production TRIGGER HLT.All
 jobs have failed multiple attempts at CERN-RELEASE,6000*30mins wasted.
 Suggested running scout jobs until a decision is made.Task ABORTED and
 resubmitted with a correct tag.

-+ 17.2.1.4 mc12_8TeV DigiMReco task 923418,923822:1K+ waiting jobs in FR
 cloud Savannah 131239,131259,96815:MC12 production - JETETMISS+SM.19
 files are on SE but not in dq2 from DS,downloaded and checksum tested to
 be correct.Probably a left over from Savannah 96599:INCOMPLETE datasets
 at IN2P3-CC_DATADISK,the both DSs were createdin the period of the
 problem(not registered in LFC due to an incomplete update of schedconfig)
 ,copmlete now.Task 823822 DONE,923418 is still in"running".

 -- 17.2.4.5 mc12_8TeV evgen task 927654 failures:Too little space left
 on local disk to run job.Savannah 131299:MC12 production - HIGGS.Many
 jobs failed repeatedly at various sites running in UK cloud (also with
 "lost hearbeat" running for >2days.).Task is still "running",no comments
 published so far.

++ 17.2.4.6 data11_7TeV Reco task 942044 failures:computingSite:BNL-OSG2
 is unknown.Savannah 131297:group production INDET SCT.All jobs failed
 multiple attempts in US cloud.Task ABORTED,mis-submission.

-- 17.2.2.4.2 mc12_8TeV Reco merge task 930620 failures:Failed to
 transfer AOD.00921351._000195.pool.root.1: 1103 (No such file or dir).
 Savannah 131314,131335:group production Higgs.1 remaining jobs failed 17
 attempts at various sites running for FR cloud.The file was produced for
 FR cloud in US (fr/MWT2).According to dq2 it should be at IN2P3-CC, but
 according to lcg-ls it is not there,submitted GGUS 85336 in progress:
 lost file?Task is still "running".

-- LeslieGroer - 2012-08-21

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng SCINET-to-T1.png r1 manage 188.5 K 2012-08-21 - 19:09 LeslieGroer  
PNGpng T1-to-SCINET.png r1 manage 234.8 K 2012-08-21 - 19:09 LeslieGroer  
PNGpng T1-to-VICTORIA.png r1 manage 264.3 K 2012-08-21 - 19:09 LeslieGroer  
PNGpng VICTORIA-to-T1.png r1 manage 206.2 K 2012-08-21 - 19:09 LeslieGroer  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-08-21 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback