Tier1 and Tier-2 Weekly Report 2016-03-29

Tier1 and Tier-2 Weekly Report 2016-03-29

General News

ADC and ATLAS Computing News [20160329]
- ADC Weekly meeting
- MC Production
  - Grid mostly full during Easter - MC15c and some big MC15b production
  - Solved issue last week of very slow jobs with MC15b mu profile (r7726). Resubmitted with r7773
  - Seen some similar issues with MC15c mu profile (r7725) that still not understood
    - Jobs running up to 4 days. Could be in a trigger module
    - Tasks remain running but submitted new task with r7725
  - Some discussion about whether this related to MC15c or the complexity of the tasks. Needs further investigation
  - PanDA could identify stuck tasks from heartbeats?
- Derivations
  - Plan to build AtlasDerivation,20.7.5.2 tomorrow
    - MC15c production cache
    - Submit bulk submission (end of this week?) according to the default samples collected from the group contacts
  - AtlasDerivation 20.7.5.1 ready to run on the reprocessed data
    - EGAM-train about to be launched
    - Total of 62 derivations (to be confirmed tomorrow)
    - will probably launch production towards the end of the week
- Reprocessing
  - Three outstanding activities:
    - 5 TeV data in doHIP mode: completed. Physics Containers to come
    - Re-merging of some of the HISTs from the 13 TeV reprocessing
      - Needed now for 7 runs, maybe a few more to come, due to ROOT issue in merge
      - Ticket: https://its.cern.ch/jira/browse/ATLASRECTS-2927
      - Fix is in 20.7.5.3, re-done in that release.
      - Tasks submitted end of last week, but not progressing very fast due to INFN downtime
    - "debug_crashed" stream attempt
      - Attempt to make collections of about 7k selected events from ~50 files (5 different streams)
      - e.g. "containers"
        
        data15_13TeV.50ns.debugrec_crashed.daq.RAW:
        
        data15_13TeV.minBias.debugrec_crashed.daq.RAW:
        
        data15_13TeV.25ns.debugrec_crashed.daq.RAW:
      - But not working, not processed correctly by ProdSys2, not correct for AMI...
        
        https://prodtask-dev.cern.ch/prodtask/inputlist_with_request/6559/
        
        Needs more thought
- CRC/ADCoS report
  - INFN-T1 downtime for ~5 days but not announced until just before it happened
  - Seeing Pilot Factories degradation quite often during last few days
    - Not sure if monitoring or real - see elog thread 56857
  - CERN-PPSTAPE transfers were spoiling the dashboard for the whole weekend, token was removed
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift