TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121204 (2012-12-04, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-12-04

General News

  • ADC and ATLAS Computing News [20121204]
    • ADC Weekly meeting
    • LHC 0.8 fb^-1 delivered last week. Peak Luminosity 7.6E33. Total delivered: 23 fb^-1. Prepping for 25ns bunch spacing.
      • Shutdown Dec 13, turnon Jan 7
    • Reprocessing
      • Cleaning up the tails, recovering lost files etc. Merge jobs still to be done for last two periods.
      • Working on procedures to just rerun failed jobs rather than full task (set task status to RUNNING; unfreeze datasets)
        • These procedures will go into JEDI
      • High priority tasks (>950) actually cause issues as force jobs to the corresponding T1 and don't allow T2 or MCP to assist
        • Discussing replacing the "priority creep" with more controlled mechanism
      • Hospital queues needed for some seriously ill jobs (mostly walltime issues, can be memory too) - one per cloud, preferably T1
    • T1 Disk Space
      • Few sites have added space recently (including TRIUMF: 750TB added to DATADISK) so no major issues this week
    • AMOD Report
      • Software brokerage transition completed - PanDA brokerage now uses AGIS rather than BDII information
      • FZK had some storage pool problems and recovering files at the moment
      • IN2P3 saw issues with long proxies on their SRM - dCache provided a fix GGUS: 88984, 89111
      • Some Frontier issues at RAL - caused issues in DE cloud as well as were failing over
    • AGIS Integration
      • Almost fully transitioned
      • Stopping SVN updates into JSON soon (might go the other way now)
      • Some incompatibilities with OSG to be settled
      • Completing the AGIS population of schedconfig and final testing of site/queue config this week
      • Web API developments needed:
        • User authentication and change tracking associated with individuals
        • Full history of changes, available via HTTP or something similar – ease of use similar to SVN
        • Training and transition for users before we tell them that the old interface (SVN) is deprecated. Possible metric – number of checkins per week to SVN. Educate those last few who cling to the old ways.
    • T2Ds Degraded Performance see report from Guido Negri
      • BEIJING-LCG2 – improving
      • CA-MCGILL-CLUMEQ-T2 – quite bad
        • Comments: Good performance with American T1s. Somewhat better perf in receiving data from T1s
      • CA-VICTORIA-WESTGRID-T2 – should improve transf TO T1s
        • Comments: Only slightly below T2Ds threshold. Should improve a little performances in transfers to Tier1s (NDGF, SARA, NIKHEF, CERN, RAL). Appeared in list on Aug 21, 2012
      • IN2P3-LAPP – quite bad
      • INFN-MILANO-ATLASC – should improve transf FROM T1s
      • SWT2 _CPB – should improve transf FROM T1s
      • UKI-SCOTGRID-ECDF – should improve transf TO T1s

  • Security News [20120801]

  • ROC_Canada News [20121030]
    • Current shift - Neil Knecht November 5-9, 2012
    • Next Week - Simon Nderitu

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 

      World-wide weekly ATLAS ADCoS summary (Nov.27-Dec.3,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Nov.27-Dec.3,2012) Panda production service
      - completed successfully 1,330,860 managed group, MC production,
        validation and reprocessing jobs
      - average ~190,123 jobs per day
      - failed 160,107 jobs
      - average efficiency:
         -- jobs ~89.3%
      - active tasks: 2305
         -- distribution by cloud:
 CA:181 CERN:91 DE:296 ES:165 FR:146 IT:142 ND:167 NL:324 TW:249 UK:393
 US:151

B)  Downtimes:
     - CERN/CERN-PROD DT (ce203 OUTAGE:upgrade to emi2), Nov.23-29.
     - CERN/INT8R,INTR DB rolling interventions (~3h), Nov.28.
     - ES/PIC DT (SE,SRM warning:at risk,tape system upgrade), Nov.29.
     - ND/NDGF-T1 DT (SE,SRM OUTAGE:dcache head node upgrade), Nov.29.
     - ND/NDGF-T1 DT (CE warning:Network outage to HPC2N), Dec.3.
     - IT/INFN-T1 DT (SE,SRM warning:StoRM upgrade), Nov.29.
     - FR/IN2P3-CC DT (SE,SRM OUTAGE:fixes on Dcache), Nov.30.
     - FR/IN2P3-CC DT (cccreamceli05 OUTAGE:EMI updates), Dec.3.
     - CA/TRIUMF-LCG2 DT (CE,SE,SRM,BDII OUTAGE:power UPS), Nov .30.

    Other news:
     - New ATLAS DBRelease-21.1.1 subscribed to sites, Nov.28.
     - New ATLAS releases and TRF caches were distributed:
       -- AtasOffline     - 17.1.6(32+64b)
       -- AtlasProduction - 17.3.3.4(32b)
       -- TDAQ            - 17.1.6:4.0.1(32b)
       -- TrigMC          - 17.2.6.1.7(32b),17.2.6.1.8(32b)
       -- HLT             - 17.1.6(32b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Nov.27
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ FR/IN2P3-CC many job f./Put er.:lcg_cp:Invalid argument.GGUS 88899
  solved:there was a backlog on gridftp door of dCache.The dCache issue
  is stabilized the same day,return to full production gradually.
+ TW/Taiwan-LCG2 ~150 job f./lcg-cp:err.while loading shared libraries:
  libcgsi_plugin.so.1:can't open shared object file:No such file or dir.
  GGUS 88919 in progress:NFS issue(?),and should be fixed.Seems not a
  site issue as seen at other clouds.No more error appeared,Nov.29.
+ IT/INFN-NAPOLI-ATLAS 5900+ FT f./DEST.:CONN.:SRM.GGUS 88920 verified:
  had a power supply issue,recovered all the services,Nov.29.
+ DE/ATLAS-KIT-Frontier not avalable,affected ATLAS-Frontier availabil.
  GGUS 88974 solved:restarting tomcat fixed the problem,Nov.28.

2) Wed Nov.28
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
+ CA/CA-MCGILL-CLUMEQ-T2_SCRATCHDISK->CERN-PROD FT f./GRIDFTP_ER.:globus
  _gass_copy_register_url_to_url transfer timed out.GGUS 88927 in
  progress.
- DE/FZK-LCG2 jobs f./stage-out err.:libcgsi_plugin.so.1:can't open
  shared object file.GGUS 88921,88877 in progress:finally the vendor
  reacted,fixed the pools.They will stay read-only until we migration
  the data onto new pools(ongoing).Have to wait until dCache removes all
  file declared lost replicas from the pools.Then can initiate a
  checksum verification for all files that are left.
+ DE/GoeGrid: 442 job f./stageout SRM_FAILURE.GGUS 88922 solved:caused
  by network misconfiguration,fixed the same day.
+ FR/IN2P3-CC FT f./DEST.CON.:SRM.GGUS 88984 solved: reduces the
  efficiency:50-90%(very low for channels to BEIGING,INFN-T1).Caused by
  SRM bug,which is hit when using large proxies(>16KB,due to multiple
  delegations) for authentication.After some time dCache becomes
  unresponsive and needs to be restarted.The dCache developers provided
  a patch,deployed on the SRM node at ~14:25,Nov.30.The SRM server has
  been restarted successfully.
+ US/BNL_ATLAS_RCF job f./lost heartbeat.GGUS 88975 verified:resource
  owned by a different community,provides compute cycle on an
  opportunistic basis.Jobs can be evicted,autoretried.No needs to report

3) Thu Nov.29
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: NL/IL-TAU-HEP,DE/DESY-HH
        .....................................................
+ UK/RAL-LCG2 FT f./DEST.+SOURCE:permission denied|SRM_FAILURE|SRM_
  ABORTED.GGUS 88989 solved:affecting also export of T0 data from CERN.
  Had a problem with Castor overnight.Fixed.
+ UK/ATLAS-RAL-Frontier 0% availability.GGUS 89063 solved:machines had
  a problem with networking,fixed.Savannah 99177:still 0% avail.,Nov.30.
  SLS monitoring has been updated pointing to new hosts,Dec.3.
+ CA/IAA-CSIC 450 job f./lost heartbeat.GGUS 89010,88375 solved:using
  the new VM image now;it should fix the condor problem and also a CVMFS
  problem,Nov.30.
+ DE/DESY-HH ~700 job f./No keys among `GSFElectronAODCollection,
  GSFElectronCollection' for type `ElectronContainer' in ObjKeyStore.
  GGUS 89026 solved:several GP_HIGGS tasks failing,a problem with the
  jobdefs,not a site problem.Savannah 99165,134205.
- DE/PSNC FT f./DEST.:CONN.:SRM.GGUS 88578 in progress updated:b/e
  storage failure,fixed.But last transfer failures have never succeeded.
  New 2K+ f.on Nov29 and Dec.1.
+ IT/INFN-T1 many(60% eff.|2h) FT f./DEST.:CONN.:SRM.GGUS 89033 solved:
  new StoRM release has been installed which has shown problems never
  occurred either during the test phase or on other end-points already
  upgraded.While the problem in under investigation,a rollback to
  previous storm release has been done.

4) Fri Nov.30
      -> Cloud/site PROD efficiency ever<50% : DE/FZK
      -> Cloud/site DDM efficiency  <50% : FR,DE/FZK,CA/IAAS
        .....................................................
+ FR/IN2P3-CC<-T0 still FT f.GGUS 88984 solved:new patch deployed on the
  SRM node at ~14:25.The SRM server has been restarted successfully.
- CA/CA-TRIUMF-LCG2 1000s FT f./SOURCE:CONN.:SRM|GRIDFTP_ER:globus_xio:
  Unable to connect|server er.426.GGUS 89045 in progress:following
  yesterday power issues,the UPS service provider needs the power feeds
  to be OFF(including bypass) for safety to investigate aasap in order
  to fix the UPS and minimize downtimes & number of cold starts.UDT.
  Still 840 job f./Put err.,Dec.1.Migrated part of WNs to EMI-2.Most of
  jobs failed on the new EMI-2 WN.Looks like to me there are still some
  bugs with lcg-util from EMI-2 WN(?) or dcache problem(?).
+ DE/ATLAS-KIT-Frontier 0% availability.Fixed.Elog 41359,41363.
+ DE/CYFRONET-LCG2 many job f./Put er.:lcg_cp:Invalid argument.GGUS
  89052 solved:probably due to a temporary DE cloud problem,restored,
  the errors disappeared.
+ UK/RAL Frontier not available in SLS for 14h+.GGUS 89063 solved:SLS
  monitoring has been updated. The file that needed to be modified was:
  frontier.cern.ch:/home/dbfrontier/monit/slsfrontier/atlas_frontiers.
  txt,Dec.3.
+ ND/NDGF-T1 DATATAPE still 256|h FT f./SRMV2STAGER:StatusOfBringOnline
  Request:SRM_FAILURE:Failed to pin file.GGUS 88802 solved:still same f.
  on Dec.2. 35 tape inputs of top priority jobs on BCCS pools declared
  lost and datasets resubscribed to force the jobs progress quickly.

5) Sat Dec.1
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : TW
        ..............................
+ NL/WEIZMANN-LCG2 6800+|h deletion errors|SRM_AUTHORIZATION_FAILURE:
  File not removed,permission denied.GGUS 89097 verified:cleaning
  atlaslocalgroupdisk.Files with wrong permissions have been removed.
+ ES/IFAE_DATADISK many FT f./SOURCE:AsyncWait.GGUS 89098 solved:due to
  some unstabilities on the hypervisor that contains site-bdii,fixed.
+ US/AGLT2 1K+ FT f./SOURCE:GRIDFTP_ER:globus_ftp_client:the operation
  was aborted.GGUS 89099 solved:2 of 5 dCache doors had memory issues.
  Memory was added,doors restarted.
+ DE/FZK-LCG2 MCTAPE 1K+ FT f./SRMV2STAGER:Job failed with forced
  timeout after 21000sec.GGUS 89102 solved:no more err.,Dec.2.
+ FR/GRIF-LPNHE $K+ FT f./DEST.:CONN.:SRM.GGUS 89103 solved:severe
  campus wide power outages,DT annouced.All critical problems solved,no
  further outages are expected,Dec.3.

6) Sun Dec.2
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
+ TW/TAIWAN-LCG2 400+ FT f./AGENT:INTERNAL_ERR.:No status updates
  received since more than 360sec.Probably the process serving the
  transfer is stuck.GGUS 89109 solved:files have been transferred.
- FR/RO-14-ITIM FT f./SOURCE:CONN.:SRM.GGUS 89108 assigned.
- FR/IN2P3-CC 8K+ FT f./DEST.:CONN.:SRM.GGUS 89111 in progress:still the
  same problem with SRM server despite the patch provided by dcache.org.
  Each time it happens,SRM needs to be restarted.Under investigation.
+ DE/FZK-LCG2 FT f./GRIDFTP_ER.:globus_ftp_client:server err.426.GGUS
  89110 solved:problem has disappeared the same day.
- DE/GoeGrid many job f./Put:SRM-FAILURE/CMT err.:Permission denied.GGUS
  89114 in progress.

7) Mon Dec.3
      -> Cloud/site PROD efficiency ever<50% : CA/SFU,UK/SOUTHGRID-BHAM
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ DE/HEPHY-UIBK FT f./DEST.:CONN.:SRM.GGUS 89119 solved:dpm-daemon was
  stopped,fixed.
- DE/MAIGRID_LOCALGROUPDISK FT f./GRIDFTP_ER.:server err.451:Best pool
  too high.GGUS 89120 in progress:not enough space in LOCALGROUPDISK.
  Probably,dCache was selecting a pool which was fully consumed,made
  the pool "rdonly".The situation is still problematic.
+ GRIF-LAL DATADISK 1000s FT f./DEST.:GENERAL_F.:Read-only file system/
  CONN.:SRM.GGUS 89121 solved:were on DT due to a electric power
  shutdown this WE.All storage service has been restored.
+ GRIF-LAL 2500+|4h FT f./GRIDFTP_ER.:server err.500/SOURCE:INVALID_PATH
  GGUS 89123 solved:see the ticket above.
- UK/UKI-SOUTHGRID-BHAM-HEP ~450 job f./libglobus_gssapi_gsi_
  gcc32dbgpthr.so' can't be preloaded.GGUS 89129 waiting for reply:due
  to the transition from Glite 3.2 to EMI2 WN.The automatic validation
  jobs don't seem to run(or s/w area hasn't been updated to EMI2 yet)?
+ UK/RAL_CG2 4053 job f./Get err.:Staging input file failed.GGUS 89141
  solved:restarted the stager and rebalanced the DB appears to have
  solved the problem.
- UK/UKI-NORTHGRID-LIV-HEP FT f./DEST.:SRM_ABORTED.GGUS 89161 assigned.
- IT/INFN-NAPOLI 900 FT f./DEST.:CONN.:SRM.GGUS 89155 assigned.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.1.4 mc12_8TeV DigiMReco tasks 1063832-1063859 failures:assert
 "AMSBIndex" in simdict AssertionError.Savannah 134127:MC12 production -
 SUSY.All jobs have failed multiple attempts at various sites/clouds.
 Tasks are in "failed" state.No comments published so far.

++ 17.2.7.5 mc12_8TeV evgen task 1072821-1072853 failures:AttributeError
 :'<class 'AthenaCommon.AlgSequence.AlgSequence'>' object has no
 attribute 'TestHepMC'.Savannah 134128:MC12 production - SUSY.All jobs
 have failed many attempts at different sites/clouds.Tasks ABORTED,to be
 resubmitted.Understood:the preExec instead of the postExec field used
 by mistake.

-+ 17.2.7.5 mc12_8TeV egen tasks 1075270,73,78 failures:TRF_SVRINIT |
 "Unable to initialise Pythia beams !!" | "Pythia 8 initialisation will
 FAIL!".Savannah 134182:MC12 production - EXOTICS.All jobs have failed 9
 attempts at various sites running for IT cloud.Tasks ABORTED and will
 be looked into.

-+ 17.2.7.6 valid1 Reco task 1059470 failures:FATAL DataHandle at
  0x40f38c30 already bound to DataProxy with key Pedestal.Cannot bind to
  proxy Pedestal as well.Savannah 99117:MC12 pp overlay test.All jobs
  have failed up to 9 attempts at various sites running for UK cloud.
  Task is in "failed" state.Under debugging.

++ 17.0.6.9.1 data11_7TeV Reco merge tasks 1074026-1074510 failures:ATH_
 FAILURE - Athena non-zero exit.No keys among `GSFElectronAODCollection,
 GSFElectronCollection' for type `ElectronContainer' in ObjKeyStore.
 Savannah 99165,134205:group production Higgs HSG2.15K+ jobs failed at
 various sites/clouds.Tasks ABORTED.Understood:the wrong input dataset
 was used (we don't use AODs as the input in the task definition).

-- LeslieGroer - 2012-12-04

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2012-12-04 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback