Tier1 and Tier-2 Weekly Report 2018-01-09

Tier1 and Tier-2 Weekly Report 2018-01-09

General News

NO ADC Technical Coordination Meeting
ADC Weekly Meeting [20180109]
- Meltdown and Spectre
  - Meltdown patch KPTI removes mapping of kernel memory in user space
  - Secret data still there, but can't be leaked
  - Performance hit 2% (HS06) - 15% (postgres)
  - Spectre not completely patched - no know exploits for now...
  - https://security.web.cern.ch/security/advisories/spectre-meltdown/spectre-meltdown.shtml
  - Patch in testing/QA
  - Need rolling reboots of CERN machines
- Group Derivation Production
  - Re-derivation campaign almost finished (only ~48 active tasks)
  - The whole campaign took about 6 weeks with an average of 95,000 slots
  - About 80 formats were produced
  - Main issues:
    - input file stage-in problems at certain sites/clouds (bulk of problems)
      - solution: re-assign to a different site/cloud
    - using > 200 input files per job
      - solution: setting MF=200 (ATEAM-448)
    - excessive memory consumption / input files too big
      - solution: run on fewer input files per job
    - merging empty files with MCORE
      - solution: switch to SCORE
    - rare software glitches, principally affecting BPHY7 and 8
      - solution: to be investigated
  - Some stats:
    - Average Max_PSS_per_core: 1.15GB
    - Reco average wall time per event 0.7s, DAODMerge 1.5s
    - Max PSS for reco jobs peak 8-9GB, merge < 2GB
    - About 15-20% failures (21.2.8.0, 21.2.10.0, 21.2.9.0)
    - Jobs that complete in < 5mins usually are failures. Most jobs finish within a day (actually peak at 20 mins, but long tail)
- MC Production
  - Steady use of 250k cores through break. Mostly MC since first few days of Jan
  - Main goal was to get MC16d reco underway, had MC16e evgen and some sim in reserve
  - Lots approved just before Xmas, now in queue
  - For last week: 150k pile + 70k sim (1B MC16d_CP almost done, 15/82M MC16d_PA)
  - Completed many tails of older samples
  - Some issues to resolve with merge vs non-merged HITS inputs and container definitions (need all tids in the same container)

- Reprocessing
  - https://twiki.cern.ch/twiki/bin/view/Atlas/Special2017Reprocessing
  - https://twiki.cern.ch/twiki/bin/viewauth/Atlas/2017FastReprocessing (WIP)
  - First part of data17 reprocessing - https://its.cern.ch/jira/browse/DATREP-101 (period B-E)
    - All runs treated with r10203 except the first one of each period (r10202 = r10203 + HLT monitoring)
    - In the tail end now. Some tasks might have double counting issues, need to investigate.
    - Need to fix one incorrect input for CosmicCalo stream
  - Second part of data17 reprocessing - test tasks
    - Here, r10222 is identical to first part tag but with updated conditions. r10224 is a clone of r10222 to test the forst after first event workflow.
  - DATREP-111: reprocessing of 2016 length scale calibration run for luminosity with 2017 configuration (1 VdM run)
- CRC/ADCoS Report
  - Couple of bugs found (Frontier access fails and is successfully retried but the TRF fails the job anyway)
  - Lots of transferring files in last week (100k+) due to short evgen jobs with small outputs causing CERN FTS load
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2018 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2017 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift