Tier1 and Tier-2 Weekly Report 2016-04-26

Tier1 and Tier-2 Weekly Report 2016-04-26

General News

ADC and ATLAS Computing News [20160426]
- ADC Technical Coordination Meeting [20160425]
  - Minutes are live, in google docs: Minutes
  - Main topic: ARC-CE and ARC Control Tower
  - Andrej comments from ARC-CE @ NERSC meeting
- ADC Weekly meeting
- MC Production
  - About 3.3B of 4.5B done (about 0.5B a week)
  - Lots of discussion about r-tag and p-tag used for merged MC
- Derivations
  - 62 derivations submitted on full 20.7 reprocessed data 2 weeks ago
    - Added new derivation: EXOT17
    - Problem defining some tasks because of error in getting tag info from AMI
  - Started open-ended derivation on new data
  - 43 derivations submitted on mc15c two weeks ago
    - Many tasks in registered waiting for xAOD to become available
    - About 7k tasks done last week
    - Many tasks in submitting - waiting for free slot. Raise priorities if urgent
  - Some jobs killed due to high mem usage (EXOT)
    - Creating too many instances of PDF weight tool - reduced. Seems to help with memory usage
    - Put into new cache (20.7.5.4) - currently being built
- HI Reprocessing
  - New strategy with cutLevel=2 working well (halves the cput time)
  - HardProbes and MinBias ~ 292M/488M done. About a month ahead of shedule
  - Still to submit remaining runs, then MinBiasOverlays etc
  - Also non-HI runs (13 TeV minBias, 5TeV VdM for luminosity group)
- HLT Reprocessing Validation
  - Based on first runs this week, submitted HLT-only and HLT+reco - completed in about 15 hours (requirement < 24)
  - Had some issued with tasks stuck in transferring due to BNL-CERN and BNL-SARA issues over the weekend
  - Expect to do this ~daily during data taking (but probably need more at the moment while tuning)
  - Submitting to WORLD-CLOUD rather than targeting a few T1s
  - Stuck on using MCORE due to bug in File Validation - open Jira ticket
- CRC/ADCoS report
  - Usual week ~200k slots, with spikes from HPC centres
  - LHC stable beams Fri/Sat
  - Some central services degradation - some network related
    - EI, Stratum1 CVMFS, Panda servers, Pilot factories
  - Some T1 DATADISKs filling up (BNL, FZK, NIKHEF) - rebalanced
  - BNL complained about not getting directly notified about network issues that they could have failed over during weekend if alerted
- Pilot Update v65.0
  - Event Service updates for large merges
  - CPUtime from jobReport (more accurate than from payload only)
  - Multiple files returned from AthenaMP (needs new AtlasDerivation)
  - Direct access support - testing pending new AthenaMP release
  - Support for 'putLogToOS' server command
  - Memory monitoring more robust
  - Send DB info with jobMetrics
  - Remove 'madevent' dir from working dir before creating job log
    - Unsustainable as keep adding dirs that need deletion
    - Better if TRF runs entirely inside workDir just like runAthena
    - Special files to be save in logs should be put either as an output file lites in jobReport or saved in special dir - all other files wiped by pilot
  - Sequential multi-jobs from same task
  - Server update for ND true pilots after out of local disk space
  - Fix for user jobs when user requested copy-to-scratch
  - Frontier is env variable now containing task id and job id to facilitate debugging
  - Alternative stage-out (bit controverisal and needs testing)
  - Few minor HPC updates and bug fixes
  - Plan for "hotfix branch" in future to not derail the long development cycle on reglar releases
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift