Tier1 and Tier-2 Weekly Report 2016-07-12

Tier1 and Tier-2 Weekly Report 2016-07-12

General News

ADC Technical Coordination Meeting [20160711]
- Discussions about AMI tags
- Network-based Brokering and Global Shares
  - Network-based weights turned on during May
  - Calculate weights based on last 24-hours
  - Don't assign new tasks to a nucleus with stuck output file transfers > 2000 from satellites or to a satellite with > 150 files queued
  - Thinking about applying weights for queued input files as well - need some monitoring tools to visualize these
  - Global Shares discussed at TIM
  - Working on defining shares and workqueue normalization
  - Need tools to apply these at job dispatch level
  - Longer term replace site capability by maxIOIntensity and make job brokerage share aware
  - Could do task brokerage as well but likely less impact
ADC Weekly Meeting [20160712]
- MC Production
  - Kept the grid mostly busy this week (goal 230k slots full)
  - Still waiting on valiation for 800M Sherpa request
  - mu=40 for current MC will be kept. Might reconsider in Fall, depending on improvements in LHC luminosity
- Group Production
  - Expect some production with 20.7.6.6
  - Rebase with 20.7.7.x for next cache
  - Open-ended derivation for data16 still with 20.7.6.5. Wait for 20.7.7.x to switch
  - Bulk MC15c production with 20.7.6.4 finished in 2 weeks
  - 3.4 PB of older production now marked for deletion. Wait for 48 hour grace period (tomorrow?)
- Reprocessing
- CRC/ADCoS report
  - 13 fb^-1 data delivered
  - Tier-0 full - need spillover (lost of manual intervention)
    - Run on largest runs (38 hours) at BNL SCORE
    - Issues with slowness of RAW data transport to BNL with EOS and other production activity
    - Slow export of output from satellites to BNL
      - Some sites without dedicated DDM endpoints still get jobs pushed by brokerage even if link to nucleus is full
    - EOS issues identified with gridFTP doors - added more doors and three new top of rack switches yesterday
    - One user managed to generate 1M output file requests to BNL LOCALGROUPDISK
      - Cancelled jobs and also added global cap on User Subscriptions in Rucio (5k?)
        
        Needs to be examined if can be applied per user
        
        Also see if possible not to copy llogs when --destSE is used
    - T1's getting full - rebalanced three of them
    - FZK DATATAPE failing due to full buffers - tape writing not keeping up
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift