Tier1 and Tier-2 Weekly Report 2017-10-31

Tier1 and Tier-2 Weekly Report 2017-10-31

General News

ADC Technical Coordination Meeting [20171030]
- Compiling ATLAS code for other platforms (e.g. powerPC) e.g. Summit and Titan
ADC Weekly Meeting [20171031]
- MC Production
  - MCProd ~70% of global share. 150k-220k slots during week
  - About 800M events pending or queued, 130M waiting approval
  - mc15_13TeV - mostly evgen (60k slots)
  - mc16_13TeV - sim and pileup (240k slots)
  - Bunch of mini campaigns on HPC sites - 50 running requests
    - Often simulation delayed due to evgen not produced
    - 8M events produced last week
- Group Derivation Production
  - AthDerivation 21.2.5.0 validation status
    - Fixed issues with 5 trains in MC - build new cache soon
    - Will need to remake validation samples (1 MC, 2 data runs) * Open-ended derivation going on with 21.0.19.6/8 mostly * Using up to 80k slots (avg 50k)
- Reprocessing
  - Rel 21 2015-16 HI - tested but holding for missing monitoring component
  - Missing Exotics running
  - Preparing for fast processing of data17
    - Goal is to reprocess first part of data-taking before winter break
    - Created slice test for 25ns and 8be4 (mu~60), with single and two input files
    - PSS per core same for 1 or 2 files per job
      - 25ns ~ 2090MB, 8be4 ~ 2230MB (~150MB more)
    - CPU Time per event - same for 1 or 2 files per job
      - 25ns ~16 sec, 8be4 ~30 sec (reflects higher mu)
    - Slice test used 25k - probably overkill for tuning config
    - Still need to validate the 2 files/job workflow
    - Make and test merge tags
- CRC/ADCoS Report
  - Steady data taking 45/fb delivered
  - EOS buffer mostly full with primary data (4PB)
  - Tier-0 full of jobs - no spillover forseen in the short term
  - Moving 2-3M files/day (>1PB/day)
  - Backlog of EOS to Castor transfers (was up to 5 days, now 2)
    - Linked to high activity on CERN FTS instance
  - Problem due to wrong protocol conguration on some CERN GROUPDISK endpoint - Sanitization campaign is being run in AGIS to fix protocol inconsistencies between sites
  - TRIUMF-LCG2: old ticket GGUS:131004 (3 weeks) about failing transfer with timeout errors. Problem finnally understood "tape writing pools were getting full"
  - No CRC shifters for next 5 weeks
- Longish talk and discussion to improve ADCoS monitoring
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2017 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2016 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift