Tier1 and Tier-2 Weekly Report 2015-07-21

Tier1 and Tier-2 Weekly Report 2015-07-21

General News

ADC and ATLAS Computing News [20150721]
- ADC Weekly meeting
- ADC News
  - Decommisioning old DaTRI interface July 27
    - No major issues reported for new DaTRI
  - Lost files report
    - Due to an issue between Rucio and FTS (solved end of June)
    - Double FTS jobs causing entries in Rucio but not on disk
      - About 2M double jobs, 15k files completely lost, ~30k recoverable
  - Storage Consistency Checks
    - DDM OPs working on automating the checks for dark and lost files
    - Started with T1s last week, follow up next week
    - Overview of storage:
      - T1 pledged disk: Total/Used SRM 46.4/39.2PB, Used Rucio: 37.5PB
      - Difference Total SRM vs Used SRM: 7.2 PB (15%)
        
        2PB new unused space in RRC-KI-T1
        
        3.9PB 10% space buffer on DATADISK
        
        0.6PB 50% space buffer on SCRATCHDISK
        
        0.7PB unused BNL SCRATCHDISK (0.5 PB was moved after we made this table to DATADISK), PRODDISK and GROUPDISK
      - Difference SRM Used vs Rucio: 1.7PB
        
        1PB temporary datasets
        
        0.7PB dark data (0.5PB DATADISK)
      - T2 pledged disk: Total/Used SRM 57.8/41.3PB, Used Rucio: 39.3PB
      - Difference Total SRM vs Used SRM: 16.5 PB (28% of total)
        
        4.4PB space buffer on DATADISK
        
        1.2PB space buffer on SCRATCHDISK
        
        0.9PB space buffer on PRODDISK
        
        10.2PB unused space
        
        N.B.: Some is space added for new pledges
      - Difference Used SRM vs Rucio: 2PB
        
        0.8PB temporary datasets on PRODDISK
        
        0.4PB temporary datasets on DATADISK
        
        0.8PB dark data
  - Storage Space optimization:
    - Clean up of remnants of DQ2 to Rucio migration: files not belonging to any datasets and without tombstone: 30.2 M files (~800TB logical, ~600TB physical) were deleted last week
    - Not belonging to any dataset but with tombstone: 20.2M files - not deleted yet until space needed (tagged for deletion)
    - Some discussion about secondarizing DC14 data (4PB disk, 1.5PB tape) - approved by CREM and announced but some physics groups still vacilating * Lifetime model: June list datasets secondarized 20th July (0.6PB disk, 0.4 PB tape)
    - Rebalance percentage of primary vs secondary. Should be "similar" for T1 and T2
      - Constructing dataset list for move now
    - Need to review the amount of secondary we can replicate on Tier2s, in particular we need to review PD2P: this will be done in the coming weeks
    - PRODDISK is being merged with DATADISK to more effectively use the space: 1st site done last week, no problems
- Derivation Production
  - Have been running mostly successfully with open-ended technical containers
  - Couple of issues with calibrations for TRT with new gas mixture and getting this into the conditionsDB - could not process last couple of runs
    - Too late anyway for EPS conference
  - MC Production requests so far have come through Jira tickets.
    - Physics groups would like to be able to run with open-ended technical containers similar to data - may be possible for some
    - In the meantime Prodsys2 providing new DEFT interface to speed up submission of MC derivations, bypassing the Jira
- T0 Report
  - Two small runs still in pipeline (36h calibration loop ends today)
  - Remaining runs taken until Thursday (up to Run 271744) fully processed
    - calibration loop successfully shortened from 48h -> 36h -> 24h
    - Timely completion of ES-1 and calibration processing
  - Working on understanding tails
    - Target: time for reco and AOD merging (until start of derivations) < 24h
      - Reached for majority of runs, up to 28-30h for some
    - Speedup: AOD merging strategy
    - Most slow/stuck jobs seem to be on 32-core machines - investigating with IT
  - Spill-over validation exercise in progress
    - Complicated by large number of small outputs
  - Will implement continuous spill-over of small stream (ZeroBias) to check readiness of system
- Cloud Support + Operations Discussion
  - Motivation for discussion
    - Need for good liaison and support for every site in ATLAS
    - Many new sites and new countries
    - The "cloud" hierarchy is no longer very meaningful in ATLAS (no local LFC, FTS, automated switcher etc)
  - Review where we are
  - No decision has been made to “drop” cloud squads
  - Thanks to those who responded with feedback
  - See slides for history and feedback
  - Cloud Support (in theory)
    - A liaison between sites and ATLAS
      - Site admins are not generally ATLAS members
      - ADC has no direct contact with sites
    - Help for new sites to join ATLAS
      - Initial help to get started and building a network of support
    - Triage of ATLAS related problems
      - Cloud support translates ATLAS problems into something a site can understand and fix
      - Handling of notifications (HammerCloud, switcher, DDM space)
    - Site support is accounted differently
      - We should not mix up cloud and site support
  - Feedback - well-supported clouds feel current structure works well
  - What works well
    - Clouds with central coordination
    - Clouds in one geographical area
      - Also to some extent sharing linguistic/cultural bonds
    - Clouds where people are paid to do cloud support
  - What doesn’t work
    - Clouds with no geographical or linguistic connection
    - Fragmented support within the cloud
    - Support of small sites outside the T1 country
      - "We’re not sure where to put new site X, let’s just put it in cloud Y"
    - Asking cloud support for non-site related problems
  - OTP Credits
    - ~17 FTEs in 2014
    - Credits do not reflect what we in ADC define as "cloud support"
      - They tend to be awarded for anything that touches ATLAS computing
        
        Non-ATLAS specific development projects
        
        Bureaucratic/management roles
  - Possible solutions
    - A support contact per site will not work
      - Not scaleable and most sysadmins are not ATLAS people
    - Support centres per country (funding agency)
    - A catch-all support for small/disconnected sites
      - Create a forum where all the Squad can share their issues?
      - dedicated people for catch-all
    - Identify one clear coordinator for each Squad (plus one deputy)
      - regular meeting (O(monthly)) between ADC and Squads to identify the actions to be done and the activity done
  - Short report from German Cloud Support
  - Lots of discussion and some suggestions - notes will be sent by email
  - Will take this up again in September at ATLAS S&C week
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150707]
  - June 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift