Tier1 and Tier-2 Weekly Report 2015-01-27

Tier1 and Tier-2 Weekly Report 2015-01-27

General News

ADC and ATLAS Computing News [20150127]
- ADC Weekly meeting
- GDP Report
  - Two weeks now with decent load on the system
  - Still variety of ongoing issues, some minor, some major - most people back at work
    - Main issue replication of data and its distribution over the grid?
  - At the 150k level for simultaneous jobs, but grid still not full - production at about 100k
  - Distribution of inputs causing hanging tasks
    - Tasks cannot be assigned to a cloud because primary and secondary input datasets (minbias distribution, cavern files) are not located in the same cloud
    - Requires manual intervention to create rules ad-hoc
    - Suggestions include consolidating input containers and distribute them automatically; create a WORLD cloud; broker to cloud with the most inputs and allow rucio to take care of the rest of the data transfers
  - Reported already that Data loss recovery procedures are working well and being tested
    - Bookkeeping and sync of the metadata between DEFT, JEDI, Rucio and AMI is still a concern
  - Derivation production merging - ROOT bug caused read issues, explaining the problem with internal merging
  - Automatic increase of memory works
    - Paul incorporated detection mechanism in the pilot that JEDI is able to identify memory allocation failures and then memory requirements are automatically raised in the next job execution attempt - won't get all cases
  - Fixed brokerage of many jobs to sites running very few – An issue was identified where the weights for sites were too high and hence, got too many jobs. Fixed
  - Slow movement of input data causing difficulties in job distribution to sites and hanging tasks
    - Still an underlying issue and maybe one of the main reasons we cannot fill the grid. In some cases, the rucio transfer get “stuck” so tasks cannot be assigned to clouds or jobs cannot start in the sites. In some cases, retrying the replication fixes the problem but this slows down the workflow.
  - Child task is done before processing all the events produced by parent task
    - – This condition appeared due to a bug in DEFT and causes the chain to be inconsistent with the number of produced events. Fixed this week.
  - Tasks with high failure rate
    - Some tasks have been submitted that are buggy and causing massive failures (>80% errors).
      - e.g. jobs from task 4690876 failing with the python error “IndexError: list index out of range”. Aborted by production managers.
- MC Production
  - ProdSys2 Status
    - Most of the tickets can be closed or will have their priority decreased (we will go through them asap)
    - Number of events produced in evgen not passed to following tasks
      - Can be solved either using DEfT datasets or looking to rucio (only valid for prodsys2 datasets)
    - Tasks not starting due to incomplete datasets (evgen or pile jobs)
    - Implementation of MC15 evgen
    - 44M full sim, 152M reco events in system that could be processed already
    - Should finalize validation soon on 50ns request (mostly digi+reco)
    - Also have initial list of MC15 simulation update from validation by end of the week
    - New upgrade request to be submitted soon with high pileup, could be equivalent to 100M mc12a digi+reco
    - SUSY pMSSM request with 15k samples (x4 datasets) will be submitted shortly
    - TRT request (1398) is high priority for physics groups. Around 75M out of 120M finished in the request
    - Upgrade request (1480, 1481, 1482) is high priority for physics groups.
      - Some initial jobs in ‘assigning’ state due to incomplete dataset.
- Derivation production
  - ~ ~1 week delay in the preparation of next production cache (19.1.4.5)
    - Request from physics coordination to apply some fixes to lepton isolation
    - This cache is for the deployment of smart slimming and testing of trains, and we don’t expect it to be a full re-derivation (because not all groups need to re-run)
- SARA Data Loss Incident
  - Lost files 325k (150k non-logs)
  - All lists are attached to JIRA ticket: https://its.cern.ch/jira/browse/ATLDDMOPS-4826
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift