Tier1 and Tier-2 Weekly Report 2017-08-01

Tier1 and Tier-2 Weekly Report 2017-08-01

General News

ADC Technical Coordination Meeting [20170731]
- Tape Usage - users' point of view
- Event Service for Production on Grid - status update
ADC Weekly Meeting [20170801]
- MC Production
  - 200-300k but big/long spikes are due to NERSC (HPC).
  - The global share itself looks fine.
    - Most of shares are being used for MC16 simulation and digit+reco.
    - EVNT jobs (single-core jobs) are almost zero but new requests to be submitted soon
  - ~100 M events in the production queue: MC16a 15%, MC16c 25%, MC15 60%
    - Need more MC16a/c to keep the grid full
    - Have some MC16c requests but need to decide if will introduce a “merge step (10k events) for all the EVNT
- Group Derivation Production
  - CP-trains launched (7 trains, 38 formats) - done
  - Physics trains launched (10 trains, 73 formats) - mostly done
  - Production with release 21 data17_13TeV essentially done
    - Investigasting a few failures in one task
- Reprocessing
  - Waiting for a fix for 2017 data for missing trigger info in physics_Background
  - Done with processing of pPb VdM runs with updated alignment
  - Waiting on final 2 data runs with low-pt di-tau reconstruction
  - Reprocessing of Heavy Ion 2015-16 data in release 21 - WAITING
    - Will be a significant campaign, only test runs requested so far
    - Reprocessing on the full data set expected in autumn of 2017
  - Running few luminosity data2016 runs with 2017 configurations
  - Waiting on 2016 data reprocessing test with R20.7 and new ID alignment
- CRC/ADCoS Report
  - 300k+ cores used all last week
  - Frontier Issues
    - Overlay dataset reconstruction tasks defined and released Wednesday late afternoon
    - Caused usual havoc for Frontier servers at RAL, TRIUMF and IN2P3-CC - autorecovered TRIUMF, RAL needed a few restarts
    - No jobs went to CERN and those few in other clouds that have CERN as main Frontier server worked OK
    - Can see the effects on CPU efficiency for all tasks over the few days other than analysis and production, which needed condition data
      - Most of the affected tasks failed after wasting several days of walltime
    - Some pausing of tasks Friday and paused all yesterday
    - Now releasing the throttle very slowly (max 600 running jobs)
  - Installed newer FTS version at RAL as ran into 32-bit integer issue after 2B events
  - Pilot FTS at CERN had only RFC proxy installed that was not recognised by sites with older versions of dCache (<2.16)
  - Fixed temporarily by installing a legacy proxy in the FTS server but sites need to upgrade their m/w
  - Tickets issued to about a dozen sites about upgrading
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2017 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2016 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift