Tier1 and Tier-2 Weekly Report 2015-05-19

Tier1 and Tier-2 Weekly Report 2015-05-19

General News

ADC and ATLAS Computing News [201500519]
- ADC Weekly meeting
- CRC Report
  - Was not a calm week - exacerbated by Ascension day (Thu) in France
  - Most issues resolved by Friday - fair amount of work on shifters and experts over weekend
    - Running full steam since then
    - New record: got up to 200k job slots used in last few days
  - Follow the daily activities https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/ADCOperationsDailyReports2015
  - Mostly FTS/DDM Transfer issues from the week
    - Tue - bringonline daemon crashing sometimes with GFAL2 bug introduced during FTS server upgrade 10 days ago
    - FTS servers deployed with many frontend suffered much less
      - Discussion ongoing with FTS service managers
    - Rucio update issue
      - Unfortunate: part of code pushed in production while it should have not
      - More defined procedures are being discussed now between Rucio team and ADC Ops to avoid this happening in the future
    - Rucio: DB high load
      - ADRC load issue caused by convolution of:
        
        Jumbo datasets locking too many rows in DB
        
        New massive Pile workflow, 2 datasets with 11k files each with many replicas
  - SCORE vs MCORE
    - Last week: about 2:1
      - Somewhat unusual as finish up productions for mc12 Run 1 anaylses
      - More EVGEN than expected
    - Still we think we can run more MCORE at sites
    - Discussion ongoing how to chase sites not providing enough
    - How to better optimize: 4 cores instead of 8? 2.5k events in pile (now 1k)?
    - ADC: we need Global Shares to be sure that we properly organize the work for the various 'players': need to throttle e.g. MC prod vs DerivationFramework vs HLT repro, this will imply also SCORE vs MCORE
  - DDM activities: default and user sub
    - Default: discovered that Panda use one method (register location) which don't pass the activity:
    - Following this up with Panda
    - No problem with Users Sub, just bursty activity, shares seems to be working OK
  - Final thought:
    - Difficult to find balance between fixing things and avoid updates to minimize the risk to break other things
    - More testing needed before updating services
    - Caution for sites about downtimes around June 1 when 50ns physics collisions expected as expect demands on ADC will be high
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150519]
  - April 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift