Tier1 and Tier-2 Weekly Report 2015-06-23

Tier1 and Tier-2 Weekly Report 2015-06-23

General News

ADC and ATLAS Computing News [20150623]
- ADC Weekly meeting
- MC Production
  - Grid running stably at high load (with some fluctuations).
  - In the last week (including some AF2 and MC12), processed 175M simulated events, 163M
  - There has (deliberately) been a heavy load from simulation in the last days. This was holding up some other tasks but that load is draining now digi+reco
  - Fixes
    - New minbias pileup with the truth jet fix to be used for 25ns and 50ns reconstruction.
    - Fix for duplicated event numbers / events
      - Prodsys2 config issue for sample extensions fixed
      - Duplicated events in AthenaMP on the same worker
      - Moving to AODSLIM for 25ns
  - MC efficiencies in MC15 simulation over the last month are about 0.89, 0.63 with pileup
- Derivation Production
  - Aiming for each derivation to be ~1% of input xAOD, ~4% per physics/performance group in data
    - Currently Jets, FTAG and HIGGS above that
  - For MC, EXOT and HIGGS are factor 2 over budget. More active groups have more samples so more volume.
  - CPU ranges from 30-500ms/event per train - trying to keep budget of ~10 sec. Will increase with higher luminosity
  - Highest CPU is from EXOT3, JETM1, 8,9 train. It is a very heavy jet algorithms train, fat jets+Rscan
  - Most tasks on data finish in the first 6 hours
  - Most tasks on MC finish in the first 13 hours then go up to 24 hours then we have tails. Stuck jobs are terminated and resubmitted manually
  - Output file merging is taking the most time due to the file validation. This needs to be done faster, discussing with software experts. In the meantime testing the merge job walltimes by reducing the number of files to be merged
- Reprocessing
  - https://twiki.cern.ch/twiki/bin/view/Atlas/Run2Reprocessing2015
  - Second production Express and CosmicCalo streams
    - The first iteration has successfully completed and we were about to launch the bulk processing, when a problem with the alignment was discovered.
    - New express stream reprocessing launched last Friday evening, joined by CosmicCalo on saturday.
    - Express stream completed in the early hours of last night and last few jobs of CosmicCalo should be done today
  - Relieving backlog on T0 by processing run 267638-9 - mostly done already with mcore at T1's
  - Bulk reprocessing to launch today
    - All 14 runs again with conditions update (new beamspot ONLY, derived from the 2nd express stream)
    - Split production into two parts: high mu and min bias
  - Some issues with errors at some sites with AtlasFieldSvc ERROR - different versions of Oracle DB or Frontier?
  - Some new conditions were locked Friday but not available until Satuday morning, causing express stream to crash
    - New recipe to check if conditions available before launch
  - Still see issues where can take 7+ hours before tids are put into the dataset containers
    - For the express stream, this can be 1/3 of the total time of the job. Containers are needed by beamspot and DQ before we could proceed. Can this be improved?
- CRC Report
  - Tier-0 backlog cleared by Monday
  - mcore jobs are running for reco
  - Testing physical nodes vs VMs - still see some differences
  - DDM issues over the week
    - Conveyor getting stuck at least two times
    - Finisher in trouble over the weekend
    - When too many FTS jobs are scheduled, makes new subscriptions too slow and time out
      - Discovered when moving 2M files from SARA-Matrix to RRC-KI-T1
    - May need some better monitoring tools to catch issues
  - Evaluating impact of duplicated event issue - is this only in MC?
  - HLT reprocessing jobs can take too long (10h) - should be 2-3h for optimal turnaround
- Distributed Analysis
  - Analysis jobs tend to occur in bursts, leading to peaks in pending jobs
  - Group production does not play a big role yet (mc15 simu & reco dominating activities)
  - Most users still finishing up Run 1 analyses
  - mc15/data15 just picking up, expect to increase a lot in the next weeks
  - DAST reports stable load, smooth operations
    - User complaints decreasing - most critical was when JEDI still used the dq2 API
    - Some slowness with sire-related issues and response of cloud-squads
  - Rucio clients recommended for users
    - Many users moved to rucio, some still using dq2
    - Still a few bits of missing functionality (e.g. dq2-list-replica-history)
    - A web interface to Rucio is available at https://rucio-ui.cern.ch
      - Currently geared towards administrative users but tools for end users will come soon
  - Hammercloud infrastructure moved to Agile - improved test submission speeds
  - Current AFTs quite old - plan to replace at least 2 soon - use data15
  - Testing Athena- and Root-based AFTs - about 100 queues tested, 100 to go (missing datasets)
- Proposal for new SAM tests via gfal librariers for Availability/Reliability Reports
- New PanDA Pilot (62.0)
- xAOD usage monitoring
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150519]
  - April 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift