Tier1 and Tier-2 Weekly Report 2018-02-13

Tier1 and Tier-2 Weekly Report 2018-02-13

General News

ADC Technical Coordination Meeting [20180212]
- Harvester Grid Status
- Prefetcher
- Update on rucio mover tests
ADC Weekly Meeting [20180213]
- Group Derivation Production
  - AthDerivation,21.2.15.0 built yesterday
    - fixes serious problem with 21.2.14.0 - all jobs crashed when running over > 1 file in MCORE
    - Should have fixed problem with merging empty files
    - test tasks launched
    - Few % of jobs failed repeatedly in previous test tasks - resubmitted
  - Will send a new list of superseded derivations to be obsoleted/deleted this week (~ 1 PB)
- MC Production
  - MC16d bulk samples mostly done end last week - cleanups, some INFN issues still
  - Increased to 20% share to finish up
  - Some work to ensure P1 stays full with sim tasks
  - More MC16e coming soon, starting from evgen
  - Still some issues due to ds name length for some tasks - discuss with AMI
- Reprocessing
  - Remaining issues with first part of data17 being slowly treated - no issues from DQ
  - Speeding up testing for period B fix to aligmenent errors
  - Reprocessing of period F started - H,I,K setup
  - Running - 2017 5 + 13 TeV low mu data for jet/etmiss
  - Done - 2015 low-mu 13 TeV
- CRC/ADCoS Report
  - Running 300k+ slots all week
  - Memory monitor issue - showing 100% CPU usage due to linker problem
    - Fix in future releases of pilot
  - libblas library added into HepOSlibs (needed by user analysis releases)
    - sites were asked to update their libs
  - Few minor issues with CERN-PROD datadisk and EOSATLAS deadlock
  - Slow transfers to BNL fixed with restart of gridftp doors
  - Timeouts to FZK tapes - limit of 100 connections
  - Ran out of ports at RAL ECHO - fix needs changes on central router (next week)
  - timeouts to TRIUMF DATATAPE - temporary overload - looks OK now
  - INFN transfers under investigation (firewalls? slow response from team)
- Input data pre-production step
  - Proposal by Rod Walker
  - Motivated by high manual workload from lost or corrupted AODs and lost/missing/unavailable/bad checksum replicas
  - Aim: fix before tasks run
  - Detect failures by making disk replicas one produced with limited lifetime
    - Monitor and react to failures
  - Could be done for RAW (and maybe HITS)
    - Requires extra temporary disk space at T1's/nucleus?
    - Some tasks go longer than 30 days
    - How to avoid delays in starting next steps
    - Needs prodsys2 + rucio development
- Blas and HEP_OSlibs updates
  - New requirement/dependency BLAS - inserted into HEP_OSlibs metarpm for SL6 and CentOS7
  - Was in VO ID card for a while:
    - https://operations-portal.egi.eu/vo/view/voname/atlas
    - HEP_OSlibs_SL6-1.1.4-1.el6.x86_64.rpm
    - HEP_OSlibs-7.2.5-1.el7.cern.x86_64.rpm
  - Sites are required to upgrade to these versions
  - WLCG HEP_OSlibs baseline will be updated too
  - Experiment don't want to maintain OS dependencies in CVMFS
  - Solved eventuall by containers, which is where the focus is now
  - Hammercloud test thrown together quickly to check this:
    - http://hammercloud.cern.ch/hc/app/atlas/template/974/
    - About 25% sites still failing
- Tape File Families
  - data18
    - data18_13TeV, data18_1beam, data18_1p8TeV, data18_900GeV, data18_calib, data18_comm, data18_cos, data18_hi
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2018 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2017 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift