TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20121009 (2012-10-09, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-10-09

General News

  • ADC and ATLAS Computing News [20121009]
    • ADC Weekly meeting
    • LHC 1.12 fb^-1 delivered. Max lum 7.3x10^33^cm^2s^-1
    • MC Production team added some manpower Report
      • Some improvements to evgen
      • Significant contributions from Higgs group analysis activities (tau embedding)
    • AMOD Report
      • Drop in some production tasks on Monday (pending tasks with no input)
      • Misreporting of some site services corrected
      • Problems with VOMS proxies from BNL
        • dq2 cannot recognize the FQAN of proxies issued by BNL VOMS, OK for CERN proxies. Updated to latest panda/DDM version
      • Transfers failing from SARA -> CA ‘available CRL expired’. Suspected SARA, FTS bug GGUS 86889
        • Known FTS bug: GGUS:81844
        • Patch for proxy expiration issue available and deployed at RAL, PIC, TRIUMF, which fixes sporadic problem with delegation
        • All T1s should deploy the patch
        • Still need occassional tomcat restarts; FTS & VOMS developers still working on it
      • GROUPDISK - Report from Ueda
        • Goal - improve space usage by allocating GROUPDISK inside DATADISK
        • DQ2 uses namespace rather than space token; Works fine for dCache (not STORM)
        • ddm-ops changes the space token from ATLASGROUPDISK to ATLASDATADISK, and nothing else; no migration needed
        • Reallocate GROUPDISK space to DATADISK as it becomes available
        • Potential issue is for primary datasets in DATADISK which could potentially fill the space token. Secondary will be removed automatically when groupdisk files subscribed.
        • Expect group space managers to monitor and report
        • Monitoring plots and reports being updated to make this information more explicit - new DDM dashboard under development

  • CA Cloud Squad Actions [20120626] 

Ignoring all IAAS exclusions and resets (numerous this last week)
CA-SCINET-T2 Auto-Excluded at Tue Sep 25 07:14:10 2012 CET

  • Security News [20120801]

  • ROC_Canada News [20121002]
    • Current shift - Di Qing October 1-5, 2012
    • Next Week - Andrew Wong

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Oct.2-8,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Oct.2-8,2012) Panda production service
      - completed successfully 1,127,429 managed group, MC production,
        validation and reprocessing jobs
      - average ~161,061 jobs per day
      - failed 93,426 jobs
      - average efficiency:
         -- jobs ~92.3%
      - active tasks: 1296
         -- distribution by cloud: CA:134 CERN:108 DE:169 ES:222 FR:99 IT:90 ND:58 NL:112 TW:51 UK:91 US:162

B)   - DE/FZK DT (cream-1,3,5 OUTAGE:batch server crash,recovery),Oct.1-2
     - DE/FZK DT (cream-1,3,5 OUTAGE:batch server corrupt DB), Oct.2-4.
     - NL/SARA-MATRIX DT (SE,SRM warning:mas storage h/w failure), Oct.5.
     - CERN/CERN-PROD DT (ce208 OUTAGE:SAM errors), Oct.6.
     - CERN/CERN-PROD DT (ce205 OUTAGE:reinstall and s/w upgrade),Oct.3-7

     - CERN Site Services migration to new hosts, Oct.4.
     - New ATLAS releases and TRF caches were distributed:
       -- AtlasProduction - 17.0.6.21(32+64b),17.2.6.3-4(32+64b)
       -- AtlasPhysics    - 17.0.6.9.1(32b),17.2.5.2.2(32+64b)
       -- TrigMC          - 17.0.6.2.5(32b).

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Oct.2
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : ES/IFIC
        ........................................
+ CA/CA-ALBERTA-WESTGRID-T2->BNL FT f./SOURCE:INTERNAL_ERR.:Invalid SRM
  version [] for endpoint.GGUS 86614 solved:last error of this type was
  observed at 2012-10-02 21:28:13,and transfers have been going ok since
  then,so the problem went away by itself,Oct.4.
+ CA/TRIUMF-LCG2 1K+ FT f./SOURCE:CONN.:SRM.GGUS 86551 verified:problems
  with the network bandwidth manager node fixed.
+ US/SLACXRD->TAIWAN-LCG2 still FT f./SOURCE:CONN.:SRM.GGUS 86537
  reopened,solved:seems DOE was still blocking ASGC FTS agent to SLAC.
  Only TRIUMF<->SLAC errors,Oct.5 - see GGUS:86767.
- DE/FZK-LCG2_PERF-TAU DaTRI requests 401571-401580 awaiting_subscription
  due to quota_exceeded.Savannah 132618.
+ ES/IFIC-LCG2 many FT f./DEST.:HTTP_TIMEOUT:SRM.GGUS 86650 solved:2800+|
  1h FT f.Savannah 132659.Fully restarted StoRM services,updated
  lcg-vomscerts package,Oct.4.
- IT/INFN-MILANO-ATLASC->NDGF-T1 still FT f./DEST.:checksum mismatch.GGUS
  86129 updated:recently declared another 300 files lost to the
  consistency service.Blacklisted for fetches:Savannah 132696,Oct.4.

2) Wed Oct.3
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : -
        ................................................................
- NL/NIKHEF-ELPROD many job f./Get:No such file or dir.:/cvmfs/atlas.cern
  .ch/repo/sw/software/17.2.5/cmtsite/setup.sh.GGUS 86612 in progress,
  updated:ATLAS cache has grown,started to reconfigure cvmfs to use its
  local cache on the /tmp file system,Oct.4.
+ NL/csTCDie FT f./SOURCE:HTTP_TIMEOUT:SRM.GGUS 86700 solved:fixed SE
  problem,Oct.4.Savannah 132672.
+ NL/TR-10-ULAKBIM still FT f./SOURCE:HTTP_TIMEOUT:SRM.GGUS 86688 in
  progress:one storage box fixed,but still 1K|4h FT f.only to IT sites,
  Oct.4.many FT f./SOURCE:GRIDTFTP_ER.:VOMS error when processing cert.,
  Oct.8.Updated the latest lcg-vomscert on the server but forgot to apply
  it the disk boxes,fixed.Savannah 132612.
- NL/RU-MOSCOW-SINP-LCG2_DATADISK NO-SPACE-LEFT.Balacklisted:Savannah
  132692.
- DE/GoeGrid FT f./GRIDFTP_ER.:No route to host.GGUS 86707 reopened:still
  f.,both SOURCE and DEST.,Oct.4.Still many FT f.,Oct.7.Site is in the
  process of h/w upgrade,problems caused by dCache overload with internal
  transfers.The correct parameters were found,these problems should drain
  to the lowest level.

3) Thu Oct.4
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: ES
        ............................................
+ CERN/ATLAS-AMI in grey for ~9h,back to green at ~10am.Elog 39851.
+ CERN/CC,SS:installation of VOMS-aware software caused the stop of DDM
  activities between 11:00-12:30,rolled back.Elog 39860,67,81.Savannah
  132631.
+ NL/SARA-MATRIX_MCTAPE FT f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE:Failed to pin file,Request clumping limit reached.GGUS 86710
  solved:added 3.14 TB more to MCTAPE.It's ~20TB,Oct.5.
+ NL/JINR-LCG2 FT f./SOURCE:CONN.:SRM.GGUS 86718 verified:problem on the
  main link to Moscow fixed the same day.
- UK/UKI-LT2-UCL-HEP FT f./DEST.:SRM_ABORTED.GGUS 86753 in progress:
  success rate dropped again,Oct.8.The site adm put some suspicious
  extracts from logs in attachment dpm-log-errors.txt.Would welcome any
  guidance on interpreting these or focussing on any particular likely
  problem.
+ US/BELLARMINE-T3 FT f./GRIDFTP_ER.:server err.426/DEST.:Checksum
  mismatch.GGUS 86757 solved:adler32 checksum section back into the
  bestman2.rc config file after his recent update of bestman2,Oct.5.
- DE/PSNC still FT f./DEST.:CONN.:SRM.GGUS 86480,86117 updated:also prod.
  HC test job f.set the queue to "test".Preparing a new SE it should be
  up by the days,Oct.5.Balacklisted:Savannah 132729.

4) Fri Oct.5
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
- CERN/CERN-PROD 36% job f./Unable to access file in ANY stager(all
  prepare2get failed).GGUS 86778 in progress:CERN-PROD_TMPDISK token
  issue?Under investigation.
- CA/TRIUMF-LCG2<-SLACXRD FT f./SOURCE:GRIDFTP_ERROR:an end-of-file was
  reached.GGUS 86767 assigned:known issue.It arose when SLAC/WT2
  connected to LHCONE.Network experts at SLAC+Canada are investigating.
+ ND/NDGF-T1_DATADISK FT f./DEST.:Marking Space as Being Used.GGUS 86770
  solved:288 FT f./SRMV2STAGER:Job failed with forced timeout,Oct.8.The
  tape system admin found the reason for the slow restores,fixed.
- FR/RO-07-NIPNE still FT f./GRIDFTP_ER.:VOMS error when processing cert.
  GGUS 86676 reopened:under investigation.~800|h f.,Oct.8.
+ NL/IL-TAU-HEP new FT f.after whilelisting/GRIDFTP_ER.:server err.50:i/o
  er.GGUS 86486 in progress updated.

5) Sat Oct.6
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ..............................
- CERN/ss-ft(atlas-SS01),ss-triumf(atlas-SS05) degraded.Elog.39925,28,31,
  33,34.Aliases still pointing to the old machines,where the services
  have been shutdown?Savannah 132204.
- NL/SARA-MATRIX 288|1h FT f./SOURCE:The available CRL has expired.GGUS
  86889 in progress:alpplied the patch as the fix (GGUS:81844),restarted
  tomcat after applying the patch.However,it did not solve the problem.
  The same issue in TRIUMF FTS.Upgraded the certs on the SARA storage
  from 1.49-1 to 1.50-1,Oct.8.
- FR/IN2P3-CC,CA/TRIUMF FT f./ATLAS Data Management proxy expired.
  Savannah 132764,GGUS:81844.FTS bug,only PIC+RAL applied the patch,Oct.6
  See above also.
+ FR/GRIF-LAL_DATADISK 4625|4h FT f./DEST.:NO_SPACE_LEFT.GGUS 86892
  solved:had electric power shutdown,fixed.Savannah 132768.

6) Sun Oct.7
      -> Cloud/site PROD efficiency ever<50% : IT
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
- CERN/T1-s FTS bug causing FT failures:proxy expired.Savannah 98002:
  unfortunately the patch doesn't complete fix the problem.Other error
  messages observed:available CRL has expired.GGUS:86775,86889.
- US/AGLT2_CALIBDISK downgraded,SM SAM test(87%).GGUS 86897 assigned:1 of
  2 switch units in a stack died.Restored,but problems with some VMs
  which lost contact with iSCSI storage.1000s FT f./DEST.:CONN.:SRM.
- FR/IN2P3-LPC t.937891 job f./Missing DBRelease file.Savannah 132780,
  98007:DBRelease-16.8.1.1.tar.gz file has not been found at the site.

7) Mon Oct.8
      -> Cloud/site PROD efficiency ever<50% : ND/ARC
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ ND/NDGF-T1_MCTAPE 288 FT f./SRMV2STAGER:Job failed with forced timeout.
  GGUS 86770 updated,solved:the tape system admin found the reason for
  the slow restores,fixed.
- ND/SIGNET many t./job failures:Transformation not installed in CE.Bug
  with cvmfs 2.1.* and CVMFS_SHARED_CACHE=yes.Fixing the nodes back to
  CVMFS_SHARED_CACHE=no.cvmfs cache get's corrupted with time in shared
  mode.Investigating with cvmfs developers.
- IT/INFN-NAPOLI-ATLAS_SCRATCHDISK FT f./INVALID_SIZE:source file size
  differs from the one returned by the SRM.Savannah 132784:8 files were
  corrupted,they have been declared lost.
- DE/GOEGRID FT f./DEST.:CONN.:SRM.GGUS 86707,86905 assigned.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-- 17.2.3.7.1 valid2 Reco merge tasks 984198,985874 failures:Can't find
 object in event store for Analysis::MuonContainer.Savannah 97868:group
 production Tau.All jobs have failed multiple attempts at CERN-PROD.Task
 985874 "failed",984198 is still in "submitted" state.No comments added.

-+ 16.6.7.18 mc11_7TeV simu task 937891 failures:Reached maximum batch
 system time limit.Savannah 132662,132781: MC11 production - SM.12K+ jobs
 failed at 42 sites running for IT cloud with time limit.Task is still
 "running".Task had defaul 100,000s walltime limit,increased.Auto-reset
 from the scout jobs is turned on for EVNT tasks.

-+ 17.2.3.7.1 data12_8TeV Reco merge task 931368 failures:looping jobs.
 Savannah 132703:group production TRIGGER L1CALO.2 remaining jobs have
 failed >20 attempts at various sites running for UK cloud exceeding 2
 days CPU limit.jobs killed,task set to "finished",the owner asked to
 investigate.

-- 17.2.4.8.1 data12_8TeV Reco merge task 976352 failures:Get error:adler
 32 mismatch on input file.Savannah 97939:group production higgs test.
 Some jobs have failed 25 attempts each on CERN-PROD.Task is still
 "running".No comments published so far.

++ 17.2.4.8.1 data12_8TeV Reco merge tasks 976267,976309 failures:Get
 error:Staging input file failed.Savannah 132698:group production higgs
 test.One remaing job in each task failed 25 times due to 1 input file
 does not exist in input dataset.Added attempts to the jobs,they have
 completed 26th attempts.Tasks are DONE.

-- 17.2.2.6 mc12_8TeV simu task 974007 failures:TRF_UNKNOWN | "in sys
 Initialize():standard std::exception is caught|'FC transaction error The
 following ReadCatalogs:file:/cvmfs/atlas.cern.ch/repo/conditions/poolcon
 d/PoolFileCatalog.xml could have not been started.Savannah 132772:MC12
 production-PHYSICS.2,400+ jobs failed at ND.ARC.Task is still "running".
 No comments published.

++ 17.0.6.9.1 data11_7TeV Reco merge task 989183+some others failures:TRF
 _UNKNOWN | "jetFlags.inputFileType != RDO and CaloCalTopoCluster doesnt
 seem to exist in objKeyStore[inputFile]".Savannah 97999,95170:group
 production Higgs HSG2.All jobs have failed repeatedly at INFN-T1/IT.
 Tasks ABORTED:miss-submission.

-- LeslieGroer - 2012-10-09

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2012-10-09 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback