Tier1 and Tier-2 Weekly Report 2016-01-26

Tier1 and Tier-2 Weekly Report 2016-01-26

General News

ADC and ATLAS Computing News [20160126]
- ADC Weekly meeting
- MC Production
  - Still may be a bit of time to sign off on MC15c so probably later in Feb for start
  - Dedicated meeting on duplication issues
  - Testing overlay sample production at CERN
  - No good candidates for "100 events/job" similation yet
- Derivation Production
  - Waiting for reprocessing to start. Run in open-ended mode with 20.7.3.2
  - Setting up two trains for full 2015 data with 20.1.9.5: HIGGxDx and EXO,JET,TOP,TAU,SUSY
    - Waiting on validation results before launching full production
  - Working on new interface for MC15c to submit MC+derivation simultaneously. Need to finalize on background samples
  - Added the following parameters to the task definitions as default since last week: cpuTimeUnit=HS06sPerEvent;cpuTime=0;ramUnit=MBPerCore;cloud=WORLD
  - Some discussion about needing to delete obsolete derivations and reluctance of some groups to do so. Conferring with Physics Coord.
- Reprocessing
  - https://twiki.cern.ch/twiki/bin/view/Atlas/December2015Run2Reprocessing
  - Hope to start reprocessing tomorrow
  - Excellent performance over the weekend to produce 10 high priority runs in 20.7.3.8 for both CosmicCalo streams
  - Datasets to obsolete
    - The bulk reprocessing done in release 20.7.3.5 should be obsoleted, except for the 10 priority runs remain, at least in the short term
    - Issue that these requests were interspersed with others,so very few requests tasks can be obsoleted completely
- CRC/ADCoS report - none this week
- ProdSys meetings about duplicated events
  - Couple of meetings in last week - see slides for minutes
  - DEfT
    - Duplication caused by non-unique way input data can be entered
    - Propose new scheme for task input data - likely couple of weeks to produce
    - Implement new check of duplicated tasks running daily
  - JEDI
    - Duplications mainly from parallel job submissions - some fixes put in place end of last year
    - Still possible one file could be used in different tasks to produce same output
    - Could implement JEDI check at end of task execution
  - Event Index
    - Two duplication detection mechanism are implemented in EI:
      - On file level, runs at the moment when data is uploaded to the EI
      - On container level. Could be run by request for old data, will be implemented as a daily check for a new datasets
    - It was agreed that containers with duplicate events will be marked in AMI
      - Container level duplication check was started for dataset list provided (~11000 containers, MC15b). 30% was processed so far, 58 containers with duplicates were found
    - Need way to notify production managers when this occurs
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20151001]
  - September 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift