Tier1 and Tier-2 Weekly Report 2016-03-15

Tier1 and Tier-2 Weekly Report 2016-03-15

General News

ADC and ATLAS Computing News [20160315]
- ADC Weekly meeting
- MC Production
  - Grid is full again (last 3 days or so)
  - Finally started MC15c (Fri) and also some big MC15b production (MG W+jets and 5TeV samples)
    - Digit+reco from MC15c, Evgen and Simul from MC15b
  - Observed high failure rates Saturday
    - Suspect issue with test submissions before task submission. Cancelled r7726 today
    - Two JIRA tickets (TrigCond and TrigL2DielectronMassHypo_ZeeTight timeout) - hopefully resolved soon
- Derivations
  - Currently validating data reprocessing cache (AtlasDerivation, 20.7.3.5)
    - A few things to sort out before launching the derivation production
    - 62 derivations are scheduled to be part of the reprocessing
  - Re-based to 20.7.5 (mc15c) last week - production cache for mc15c is still ~1 week away
  - Number of L1Calo requests to be submitted this week
- Reprocessing
  - 2015 13 TeV bulk reprocessing - essentially done, but not quite
    - physics_Main DESDM and DAOD merges mostly completed (16 diff formats, 106 runs)
    - Issues with Super-mega project_mode config which was reasonably successful, but need to do this better next time
    - Huge variation in input datasets, from 15 events in 12k files to 98% skim selection
    - For the future should do something like:
      - target output file size of 2 GB, merging up to 200 files at a time
    - Three problems remain:
      - A few (8) datasets to be merged have > 1000 events in at least one input file
        
        Obsolete these manually and submit simple merges
      - There are 58/1695 exhausted first step merges, and 4 that failed altogether
        
        Was not spotted to using "ignoreErrors=True" in reco step, crashes in "HLT_xAOD__TrigPassBitsContainer_passbitsAux" (mostly) Ticket: ATEAM-269
        
        DPC say these can be aborted and so when make the physics containers use the unmerged dataset for these runs. Will be painful, but possible
      - At least three files deleted due to JEDI/PS2 inconsistency after reassignment
        
        Need to see how many this has affected
    - Remainder of Physics Containers will be made on Friday
    - Also some of the merged HIST files are corrupt (already from grid jobs), need to see how many this has affected, solution still not obvious
  - 2015 5 TeV (6 runs)
    - All pp mode done in 20.7.3.9: physics_Main, physics_MinBias, express, CosmicCalo (DESDM/DAOD merges also to be done)
    - All 4 streams now running in the doHIP mode, with release 20.7.5.2 (AOD and HIST only)
  - 2015 Heavy Ion (34 runs)
    - Following tests using express vs MinBiasOverlay, all 34 express stream runs now running in release 20.7.5.2 for beamspot derivation (AODs) and DQ sign off (HISTs)
- CRC/ADCoS report
  - Running slots
    - Low number of jobs up to Saturday when MC15c was launched, grid running at 200k since then
    - Producing 50-70 Mevts per day (pile)
    - Efficiency of MC15c is low. Large number of failed jobs (about 40% yesterday). Being investigated
  - HLT processing issues
    - Large number of failing jobs and some delays to start due to misconfigurations
    - Jobs showing as "urgent" to be moved to "recon" instead
  - Deletion failures at IN2P3 (fixed)
  - Status of Tests during the week
    - Amazon : ES got up to 100k running on Friday for about 5-6 hours (were these fully successful?)
    - NERSC : showed activated jobs but they do not seem to be running (news?)
    - Titan : Failing jobs, rejected by the submission system. Being checked
  - HI reprocessing
    - Expected to start in a few weeks, and would need to finish about the end of May
    - Checking resources needed, etc.
  - Need some changes to avoid running physics validation jobs on HPCs
  - Couple of issue with Frontier squids already fixed
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift