Tier1 and Tier-2 Weekly Report 2014-07-15

Tier1 and Tier-2 Weekly Report 2014-07-15

General News

ADC and ATLAS Computing News [20140715]
- ADC Weekly meeting
- Rucio migration
  - Started testing 1M files/day last week - uncovered some issues
    - FTS prod and pilot share same DB - ask to separate these
    - Keeping up with deletions from all the tests combined with other catch-up data deletions caused T1 overload
    - A bottleneck in the polling of FTS jobs has been identified and being fixed
  - Decrease back to 100k files/day for the meantime
    - Start testing some T2's as well in chain
    - Test more sites with deletions through https
    - DDM and PanDA discussions on integrating Rucio
      - New testing rucio-aware DQ2 client (rc6) being released this week
      - Start validating
- ProdSys2 - Validation and Testing
  - evgen: running for most use cases
  - sim: container with more than one dataset will be addressed by automated registering (secondaries) the evgen datasets in the clouds
  - merging and pileup: working but need more testing
  - post-production: work being done on retries
  - fix tasks not starting until previous task is finished
  - Group productions - about half way through validations (event by event comparisons)
  - Part of express stream reprocessing is running on ProdSys2
- Deletion Service status
  - Some file deletions affected by gfal library bug in migration from LFC to Rucio (lacking Scope) - caused massive slowdown
  - Regular transient data cleanup was not running since April till June (usually runs monthly)
  - Catmore policy (cleanup of data untouched for 6 months) was re-applied again in July
  - Huge backlog being worked through now - added two more nodes (5 total) - expect a week to catch up?
    - e.g. BNL, RAL - ~90k datasets, TRIUMF 45k
  - MWT2 reports 6 million dark files, 120TB
  - Rucio dumps are now available, we need to have a dark data check on all the sites.
- MC Production
  - Run 1: DC14 8TeV digi+reco submitted last week - 450M events, single core, half-way done (20M / day)
    - round of 19.0.3.X validation ongoing in parallel to production, including multicore validation (sign off today?)
  - Run 2: DC14 13TeV submitted last week - 275M events, mcore, 5M/day (hope for 7/8M/day)
    - digi+reco: still to be validated - 19.1.1 to be built this week - tight schedule as need to start in September
  - DC14 utilizing a fair fraction of compute resources now
  - Expect 150-200M events/week (sim+digi/reco)
  - + AODs, correspond to around 180-220 TB per week, or 0.7-0.9 PB/month (includes merged HITS and AODS)
- ADC Shifts
  - Merge ADCOS + Comp@P1 shifts
  - ADCoS very busy already
  - Setting up checklists now and organizing tasks
  - More monitoring tools and automation for some of the tasks (ease shift summaries etc)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift