Tier1 and Tier-2 Weekly Report 2014-08-05

Tier1 and Tier-2 Weekly Report 2014-08-05

General News

ADC and ATLAS Computing News [20140805]
- ADC Weekly meeting
- Rucio Commissioning
  - Reported issues last week with PandDA lookup delays - timeouts when PanDA tries to resolve many (thousands) of files in Rucio file catalog
    - Rucio team made some query and cache optimizations
    - PanDA servers updated to latest Rucio clients
    - More servers added to Rucio production server pool
    - The timeouts errors were not reported anymore
  - Last Saturday grid started to drain
    - Almost no simul jobs, but only pileup
    - Jobs were not brokered because of slow resolving of big bulks of files having replicas on many sites
    - On Monday Tadashi changed brokering algorithm to put less pressure on Rucio
    - The reason of slow response is not fully understood (DB load is OK, CPU load on Rucio servers is only slightly higher than usual)
    - The issue will be followed up further with Rucio devs
  - Rucio - PanDA Integration
    - PanDA developers are in process of validating latest (2.6.0) DQ2 client
      - Many things changed inside DQ2 client to support Rucio as a backend
    - Issues reported to Rucio during the validation are under investigation of Rucio devs
    - Plan to ask HC to use the latest DQ2 client (and register part of the produced output datasets) when these issues are resolved and DQ2 client validated by PanDA team
- MC Production
  - (last month): 513M events (111M in past week, 16M/day)
  - (last month): 999M events (127M in past week, ave 18M/day)
  - Rate has been reduced in past two weeks, mainly due to Rucio issues
  - About 150M events full sim (55M mcore), 175M reco in system to be processed, 140M full sim/230M AFII to be submitted
  - DC-14 status
    - Digi+Reco 8TeV: Most of the Release 19.0.X full campaign is finished (360M in single core)
    - Simulation 13TeV: New geometry and using mcore
      - 190M out of 380M finished, lots of the remaining launch yesterday
    - Digi+Reco 13TeV: Release 19.1.X built but not yet validated. First validation tests showed a memory leak (being discussed now) that will be fixed fast and new cache built.
      - mcore not yet validated and needed for production: also in discussion now
  - May need an increase of mcore resources as early as next week (380M to do, don't know rate yet)
  - Sim@P1 available from 15August (2.1k jobs slots)
- Reprocessing
  - DC-14 full express stream: reconstruction and HIST merging done; replicated to CERN. ESD/AOD merging ongoing, problems with memory usage and preservation of lumiblock boundaries
    - Require 500TB for xAOD and ESD - no replication policy defined yet for xAOD (still labelled AOD in dataset name)
  - 2010 HI peripheral events: output merged NTUP. Problems with skimming blocking jobs and whole tasks (if the 10 scout jobs fail)
  - 2012 DRAW_ZEE: test with one run done, waiting for feedback from egamma
  - Bulk reprocessing - hope to start end of this week
- JEDI
  - Agreed to switch user analysis to JEDI Aug 12 16:00 CET
  - Will be announced by email
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift