Tier1 and Tier-2 Weekly Report 2014-09-23

Tier1 and Tier-2 Weekly Report 2014-09-23

General News

ADC and ATLAS Computing News [20140923]
- ADC Weekly meeting
- Rucio Status
  - Panda integration - Main blocker is callback handling (rucio different from DQ2) - being discussed
  - Rucio Infrastructure
    - All machines have been migrated to a new AI project
    - Old aliases still work but should be changed
      - voatlasrucio-server-prod.cern.ch -> rucio-server-prod.cern.ch etc
      - voatlasrucio-redirect-prod-01.cern.ch will be decommissioned and replaced soon
  - Investigating how to scale Rucio daemons horizontally for scalability/redundancy
  - Rucio info will be synced with AGIS very soon, then automatic info propagation can start
  - Stress test was scaled back to 100k replicas/day last week
    - Fixed several issues related to FTS interaction so may increase again
    - Would like to stress deletion more but this may interfere with DQ2
  - Rucio clients - New version will become the default on cvmfs soon
- Derivation Production Status
  - First bunch launched today with AtlasDerivation-19.1.1.4 (couple of known issues)
    - Only 5 derivations (of 53) for Egamma, single train due to merging issue
    - Submitted 19 datasets which created 65 tasks in ProdSys2, can be monitored at:
    - http://bigpanda.cern.ch/prodsys/prodtask/task_table/#/?time_period=all&task_type=all&username=atlas-dpdproduction&request=900&task_id_lt=4172136
  - Total production expected ~ 2-2.5 PB (MC ~1.9PB)
  - Next Steps
    - Build a new cache, AtlasDerivation-19.1.1.5
    - Test the new merging trf in parallel for the Run-2 MC samples and the second round of production
- ADCoS and DAST shifts
  - Migrated to BigPanda
  - New twiki
  - Working towards a tutorial at December jamboree
    - 1/2 day on tools, 1/2 day on duties
  - Working on attracting new people
  - Reports to WLCG weekly relied on Experts but proved too much
  - Concern that no reliable information communicated from daily ops to shifters (use Elog)
- MC Production
  - DC14 Run II digi+reco athenaMP launched (16 Sept)
  - Issues with high memory being investigated
  - Some current evgen samples with high failure rate (ADCSUPPORT-3936)
    - Known issue with Alpgen on-the-fly – always get up to 10% failures; in ‘difficult’ corners of phase space can be (much) more; hard to prevent; ideally, when jobs fail with the relevant error, they shouldn’t be retried (other failures still should)
  - Recent low load on Grid
    - Partially alleviated e.g. DC14 digi+reco; overlay samples (3.7M in system; more coming); call from PC also received some response – more simulation to be submitted over next couple of days
    - Should be OK for a couple of weeks at least; hopefully more requests will come
- Memory Usage for 8-core digi+reco DC14 Run-2 (mc14-13TeV) reconstruction
  - VMEM usage of single-core reco job, just over 4.1 GB for 500 events (very little memory leak)
  - The validation of 19.1.1.5 digi+reco done on highmemory queues as normal panda queues failed
  - digi+reco for DC14 Run-2 launched in MCORE mode only
  - Memory usage (as measured on cgroup site):
    - ~20GB of RSS per 8-core job
    - Individual processes: ~2.8GB RSS, ~4.3GB VMEM
  - Several sites are failing:
    - Previously, 8-core simulation used less than 3GB of RSS in total, 1.1GB/process, so high gain in shared memory – sites not tuned for digi+reco mcore jobs with 20GB RSS
      - Problems in configuration were hidden till now
    - Job kills on sites with low memory settings in AGIS
    - Memory kills on sites for exceeded per-process vmem (3GB,4GB)
    - Other failures, also software problems obscuring the above issues
  - Current production: ~15M events/day – about 1/4-1/3 of expected
    - MCORE queues missing on half of T2 sites
    - Failing sites
  - Total 8-core job requirements:
    - 24GB RSS (physical memory limit)
      - With cgroups, 20GB might be enough but not tested
    - 48GB VMEM limit, generous
      - Maybe 36GB would be enough
    - Per process requirements: 3GB RSS, 6GB VMEM
    - This is a very rough estimate by inspecting jobs on the nodes; still discussing with SW team on actual memory profiling and requirements
  - Solutions depend on the site/batch configuration
    - Some possibilities for “quick” action:
    - A) 24GB RSS limit, no VMEM limit (cgroups)
    - B) 48GB VMEM limit for sites with no RSS limit, but with overcommitment of memory in the batch
    - C) 24GB memory reservation, 48GB VMEM (torque)
    - D) no memory limits: try with 16GB allocation and see how it goes (might cause excessive swapping)
  - Need to investigate with the sites on the configuration options (T1s, big T2s), and provide recommendations to the others
  - Conclusion
    - RSS for digi+reco is higher than expected (3GB)
    - On mcore queues, the actual usage per process is lower due to shared memory (2.2-2.5GB per core)
    - More accurate memory profiling is needed to be more precise – it is expected that the memory consumption is much lower most of the time
    - Due to sharing resources with other jobs (simulation, analysis...) we do not expect issues with overall core occupancy
    - digi+reco represents about 20% of MC resources, but can fill at least half the grid slots at times
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift