Tier1 and Tier-2 Weekly Report 2018-05-08

Tier1 and Tier-2 Weekly Report 2018-05-08

General News

ADC Technical Coordination Meeting [20180507]
- ATLAS-Google collaboration and project goals
- Kubernetes Overview
- Rucio and Google Cloud Storage Integration
- ATLAS-GCE Integration
ADC Weekly Meeting [20180508]
- Group Derivation Production
  - Open-ended production setup for physics runs
  - Following software related error affecting small number of jobs - holding new requests until understood
  - Most of urgent production done - few still need push (TOPQ, EXOT15 DAOD_RPVLL, EXOT8 on data15/16/17)
  - March obsoleted datasets deleted
  - New 20.7.8 cache built last week, next one for HI
- MC Production
  - Shifting priorities a bit this week
  - MC16e part 1, essentially done (2.3B events)
  - MC16e part 2: 129M/272M events done, 58M FS done - obtain part of a special sample for 2017 MC (uses MC16d reco on MC16e HITS); provide more tasks to HPC, BOINC
  - Almost completed 435M urgent Zvvjets events
  - Final checks done for single-tag containers
- Data17 Reprocessing
  - Previous issues with duplicated events now fixed.
  - Most period containers can't be made because of missing events.
  - 40 runs have a mismatch between T0 and GRID reprocessing.
    - 2 have production issues (broken/failed jobs)
    - 21 runs have processed 100% of RAW inputs (according to PS), and have more events than T0 processing (good news).
    - 17 runs have a grid reproc. count smaller than T0 processing. But have processed 100% of the input RAW. Missing RAWs? Or is prodsys not seeing the whole RAW DS?
- CRC/ADCoS Report
  - 300k cores used steadily during the week
  - Increased rate of transfer for job production since last week (15-20GB/s, new record of 2.3PB/day)
  - Jobs with higher than average IO intensity are assigned only to the nucleus (200kb/s --> 400kb/s last week)
  - Some storage over the deletion limits for a while - takes time to clean up
  - T0 replication was quite slow - small change to data distribution policy helped
  - Bug in rucio caused conveyor to crash on Thursday causing backlog of transfers
  - Voms-proxy issue when deployed V3/java from epel
    - Caused general blacklisting on Wed 1pm CERN time, and rucio issues on Sunday
    - Reverted to UMD version
  - EOS issue Friday due to high load
  - Monitoring pages have got stuck occasionally
  - Discussion on the possibility to define lifetime for user rules to help cleaning LOCALGROUPDISK
    - Set a default lifetime in webUI that user can modify
- Monitoring Pledged Disk Resources
  - Pledges take efect on the 1st of April
  - Until now, ADC has not been extensively checking the amount of disk space deployed at Tier'1s and Tier'2s
  - Script now available to compare deployed resources associated to centrally managed space tokens against pledged resources registered in REBUS
  - Currently, 15 federations provide less than 90% of their pledged resources
    - This includes 3 Tier'1s
    - Will open tickets to inquire about their plans
  - In the future, we might require a higher threshold than 90%
  - Reminder: Installing new disks is not enough; they need to be allocated to space tokens (ATLASDATADISK)
- ATLAS CSCS Project Status
  - Utilize Swiss Supercomputing Centre for Tier-0 spillover and HLT (elastic provisioning in high luminosity era)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2018 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2017 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift