Tier1 and Tier-2 Weekly Report 2016-05-03

Tier1 and Tier-2 Weekly Report 2016-05-03

General News

ADC and ATLAS Computing News [20160503]
- ADC Weekly meeting
- No ADC Technical Coordination Meeting this week
- MC Production
  - Mostly sim and digi+reco this week
  - MC15c reprocessing about 85% done after 7 weeks (1-2%/day) (3.9B/4.5B done)
  - Failure rate 0.1-0.5% (35M events so far)
  - Pushing to make sure tasks don't land up with very long tails
  - Some investigation of using Event Index to identify duplicate events
    - Fairly manual intensive at the moment - hope to automate after characterizing
- Derivations
  - Open-ended production on data16 with 20.7.5.4 continuing (8 runs so far)
  - mc15c - ramped up priority of CP samples
  - Jobs are about 3h length (0.8 input events/second) - would like to tune a bit
    - 95% of jobs use 1.4 sec/event or less which implies 20k events for 8h job
    - Currently running with nGB per job = 10 (consistent with 20k if AOD size ~0.5Mb)
    - Still see outliers that would violate this though
- HI Reprocessing
  - All HardProbes and MinBias submitted/running
  - All Express streams submitted yesterday
  - HI - UPC submitted, MinBiasOverlays coming
- CRC/ADCoS report
  - 200k slots used consistently last week
  - ATLAS-RAL-Frontier and ATLAS-T0-Frontier degraded
  - T1 DATADISKs filling up - BNL, SARA, INFN, FZK - being rebalanced
  - Transfer timeouts to BU - removed from list of nuclei
  - Some transfer issues between BNL and SARA and CERN - solved
- Commissioning new HC Blacklisting
  - Rely on AGIS for information on topology and Panda resource status
  - Site definitions in
  - Start using for blacklisting
  - Check AGIS fields and blacklisting rules:
    - https://its.cern.ch/jira/secure/attachment/80851/20160317-AGIS-in-HammerCloud.v2.pdf
  - Slides describe the relevant AGIS parameters and HC tests and blacklisting criteria
    - Need to check hc_param, capability, type and is_default for sites
    - Need one "is_default" for production and one for analysis
    - Been tested with CERN, ND and DE clouds. Volunteered/told that can start with CA cloud
    - Updates for AFT/PFT templates presented next week
- ADCoS Status (plea for shifters)
  - Have not fully recovered from the drop of shifters during LS1
  - Three shift levels: Expert, Senior, Trainee
  - Daily shift period (CERN time): AP(0-8), EU(8-16), US(16-24)
  - Daily shift slots - Expert:1, Senior:3, Trainee:3
  - Shift duration: 9 hours for Expert, 8 hours for Senior/Trainee
  - Can be done remotely
  - Counted as Class 2 service in OTP
  - Over last few months, had about 1/2 coverage for Expert shifters - only 6 expert shifters
  - about 3/4 coverage for Senior shifts
  - 18 Trainees, 6 finishing their training
- DAST Status Review
  - 8 shifters - 4 NA (incl. Yun-Ha) / 4 EU
  - Need management to prosletize for more volunteers
- Computing Run Coordinator Shift Status
  - Onsite. 2/3 to 3/4 coverage. 14 actve shifters (4 most active do > 1/2 shifts)
  - ADC experts or experienced Cloud Support members
  - Worked on Live Page - global monitoring, updated daily
  - Improve tracking of long-standing issues
    - Got rid of white board, twiki seems better
  - Still some uncovered shifts
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift