Tier1 and Tier-2 Weekly Report 2017-07-04

Tier1 and Tier-2 Weekly Report 2017-07-04

General News

ADC Technical Coordination Meeting [20170703]
- ADC Event Service and Object Store testing separation
- Machine Learning for ADC anomalous events
ADC Weekly Meeting [20170704]
- MC Production
  - ~150-200k running slots (rest derivation production)
  - The global share looks OK but small imbalance (between mc16 and mc15) in the EVNT share
  - About 770 M events in the production queue - Only a few % MC15
  - Looking into what is needed to produce O(20B) events for this run (~1.5 years)
    - Expect HPC to be important part of this
    - We need to improve the procedure of tasks submission. So far only Doug (from MC prod team) contributed to it.
- Group Derivation Production
  - Production with rel 21
    - CP-trains going well for both data15 and data16 (~2 weeks) - few jobs still running
    - Physics trains launched last week - 10 trains, 73 formats
      - data15 87% done, data16 29% done
  - Open ended production with 21.0.19.6
    - AtlasDerivation can't process non-reduced AODs (prior to run 327764) - reprocessed with f836 separately (11 runs)
    - Starting on reduced AODs - zero-bias stream (JETM5) and Main (16 trains, 100 formats)
  - Using ~100k slots in last ~week
- Reprocessing
  - Missing trigger information in physics_Background DAOD_IDNCB, DONE
    - 2016 72 runs (periods G-L) and 2017 all data so far
    - 2016 tested and trigger info recovered in tag r9653
    - 2016 All 72 jobs finished. Few exhausted tasks due to a muon trigger issue not yet fixed in the version used.
    - 2017 to come
  - Reprocessing some 2015 pp 13 TeV DRAW_EGZ data with mu=0 OFC for energy scale checks, RUNNING
    - Needed to extrapolate the e/gamma energy scale measurements from 2015 13 TeV pp data to the 5 TeV pp and PbPb data.
    - All jobs ran, but with data16 instead of data15 for the output... Reprocessing them again
- PanDA Pilot Update v70.0
  - Reset cpuConsumptionTime if > 10^9 seconds (mostly from frozen BOINC jobs)
  - Now look for a local copy of agis_ddmendpoints json files before copying from CVMFS or downloading from AGIS - helpful on some HPC's
  - WebDAV protocol added to GFAL2 and Rucio movers - needed for DynaFed
  - If env ATHENA_PROC_NUMBER not set, set it to job.coreCount
  - Core informa=on added to benchmark dictionary
  - Updates for Event Service
    - Remote I/O support
    - Zipping of output files now default
    - Support for multiple buckets
  - Number of changes for PanDa proxy for Event Service for BOINC
- CRC/ADCoS Report
  - Transfers dominated by Data Consolidation (derivations)
  - Discussion about sites with non-sufficient performance at today's CRC meeting: https://indico.cern.ch/event/648045/
    - Primary purpose: Identify/solve sites with large Wallclock fraction loss
    - Based on weekly activity - HC, job efficiencies, tickets and responses etc
  - ADC live page need some improvements: http://atlasdistributedcomputing-live.web.cern.ch/ATLASDistributedComputing-live/
  - EOS to CASTOR throughput test scheduled this week
  - No major issues last week.
  - Problem with quota on EOS inodes during weekend
  - Large number of queues with Pilot factory jobs extensively failing
    - Most frequent errors: CREAM_BLAH, CREAM_JOB_REGISTER, PERIODIC_REMOVE, CREAM_DELEGATE, PERIODIC_HOLD, UNKNOWN_ERROR
  - Deletion is not fast enough for IN2P3-CC and TRIUMF (most probably not a site issue) :
    - Problem due to very small files (<40 MB) to delete
    - Actions taken :
      - File reordering applied (delete big files first)
      - Increase deletion threads
      - Site blacklisted for writing
  - Sim@P1 : Problem with ca-proxy.cern.ch ? P1 machines were not able to access the machine (No squid available). Fix
- Information modelling of computing and storage services/resources - Balazs Konya, Lund University
  - Computing Resources Information Catalog (CRIC)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Reports [20170328]
  - 2017 Reports - click down to find reports of format WLCG_All_Sites_ATLAS_MMMYYYY.(html|pdf)
- Sites should be providing the 2016 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift