Tier1 and Tier-2 Weekly Report 2018-04-10

Tier1 and Tier-2 Weekly Report 2018-04-10

General News

ADC Technical Coordination Meeting [20180409]
- Summary of the HSF/WLCG Workshop
- Event Service for ATLAS@Home
ADC Weekly Meeting [20180410]
- Group Derivation Production
  - sharedWriter pre 21.2.21.0 is causing problems
  - Some datasets still to be deleted
  - Overviewing 65+ formats used in open-ended production in anticipation of data taking
    - Build test cache for large scale test
    - In the past have defined specific containers in rucio and populated them automatically with AODs coming from T0
    - Worked well for physics_Main stream
    - Some groups would like to run over ZeroBias or BPhysLS streams which likely will be done on grid spillover resources, so need to figure out how to do these in open-ended production
- MC Production
  - MC16e is main production now, first round is a full set of samples with MC16a statistics - about 2.5-3B Events Fullsim and 0.4B AFII
  - Most evgen done - submit more in next few days to kep evgen production going
  - Problematic Madgraph MC16d jobs almost done (300M evgen)
  - Some wider discussion about jobs crashing due to not enough LHE input
  - Some urgent trigger stuff making its way through the system, ut hitting some problems due to lack of validation - almost done
  - Some jobs paused for investigation - about 80% done
  - Started discussion about completing the move of all MC16 datasets to single tag containers
- Reprocessing
  - AOD.recon cleaning: part of the data17 cleaned up
  - Unmerged AODs being merged at the moment with more inputs than before. Now about 5GB, instead of 1-2GB. Will follow up with Data16 and data15.
  - data17 reco - few issues remains with few TAUMUH and ZMUMU DAODs in periods B and K.
  - Few other DAOD_RPVLL reprocessings going on
  - Fast reprocessing of 2017 BphysLS stream - almost done
  - Sorting out requests for data17 low-mu runs for AFP
- CRC/ADCoS Report
  - Issue with rucio certificate (expired) yesterday and panda earlier in the week
  - ADCR database (used by rucio, panda, etc) stuck on Tuesday and Thursday morning
    - Caused many queues to be excluded by HC because jobs could not contact Rucio
    - Cause is high load from Rucio from processing very large datasets - temporarily fixed by restarting database
    - DB admins reconfigured the DB on Friday to handle better the load
  - CERN-wide network down 10:30 - 11:15 Thursday morning
    - HC blacklisting was disabled to avoid massive exclusion
    - ADC services recovered quickly
  - Time of year to renew certificates on CERN services - large number
  - Panda certificate used to query Rucio for finished transfers expired Saturday morning
    - Fixed Monday morning but caused the grid (especially MCORE simulation) to drain over the weekend
  - DDM certificate used for Rucio transfers expired Monday afternoon - Fixed one hour later
  - Tracking down event generation errors with heavy disk I/O and failures with "LHS events exhausted" - wasting lots of time
  - Scout logic - changed to require 50% success before avalanching the task
  - New tab on ADC live page "special resources only"
  - Some potentially corrupted files on EOS caused by faulty switches - determined to not be affecting too much data
    - Request to add md5sum in rucio for future verification
  - Checksum mismatch for all transfers to INFN-T1 tape
  - No coverage for rest of April
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2018 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2017 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift