Tier1 and Tier-2 Weekly Report 2014-03-11

Tier1 and Tier-2 Weekly Report 2014-03-11

General News

ADC and ATLAS Computing News [20140311]
- ADC Weekly meeting
- GDP Status
  - Disk space crisis
    - Prevents task brokerage
    - Massive transfer timeouts failures if not in time (FR)
  - Some issues with missing HITS dataset
    - Fastsim HITS not written to tape and deleted after digireco ran
    - Problems in logic in Prodsys
    - Work on rectifying http://savannah.cern.ch/support/?142419
  - High RW prevents task brokerage
    - US and DE had many tasks with expntotal>>ntotal=ndone
    - expntotal incorrect - fixed by hand for these tasks but could bite again
  - MCORE queues draining
    - INFN-T1 downtime drained all MCORE
    - Single Bamboo queue for all clouds
    - Fix: was to pause IT task, kill assigned jobs and let task from other T1 run
    - Better fix: Asked for more MCORE bamboo queues – or at least to split into 2 sets of clouds
  - Factories
    - CondorG for cream/arc
      - Many canceled pilots for many sites – could be load related – load scales with CEs
      - All factories submit all Ces: split again?
    - Vanilla Condor for P1, CERN AI cloud
      - Using retired physical machine voatlas217; only one submitting to P1 MCORE so drained
      - 18k cores with little oversight - not direct GDP responsibility
    - MCP settings are patchy and random
      - Cannot set all clouds for all sites – networkwise/brokerage
      - Was not set for some large _SL6 US sites
      - Should be cloud squad responsibility - choose clouds and monitor transfers
    - No global oversight of resource usage efficiency
      - Are all sites full? Low failure rates?
      - Most often site admin asks “Why not full”?
      - Cloud support would be better (eg.rule out lack of tasks)
- Rucio Migration
  - /cvmfs/atlas.cern.ch/repo/sw/ddm/latest changed last week to point to 2.5.0
  - No problems seen
  - Nothing special needs to be done on APF side for Rucio FC migration
  - Two sites being migrated right now
    - Frascati (IT), LRZ (DE)
  - If all goes well migrate a T1 on Wed/Thurs (FR or DE)
  - Some US cloud specific questions need to be resolved (registrations, deletions, etc)
- Data Placement draft proposal
  - James Catmore reviewing current data at T1s and policies
  - Critical issue as many T1 sites with < 100TB and need ~300TB for ongoing operations - not able to run NTUP_COMMON v2 production
  - Brief Summary:
    - Large productions done in the last few months
      - 22% of the primary data was created in the last 90 days
    - Replication policy has too much emphasis on keeping T1 primary copies
    - Old primary datasets sitting around on T1 not being used
    - About 15 pages of details of what the 21PB of data consists of and its usage
      - No clear single dataset causing issues - largest are about 50TB
    - Some Conclusions:
      - New production is not a big part of the primary data (20%) - so not really the issue
      - Need to make headroom!
      - Old data which has been accessed recently is a much bigger piece
        
        But this data is clearly useful so not a problem
      - Old data which is not being used is the thing we should attack - roughly 40% of the total
    - Some Possible Solutions
      - Emergency fix: any primary data not touched for more than three months should be set to secondary, provided they are found elsewhere (e.g. on tape) - affects about 5PB
      - Could send a list of datasets on the “hit list” to data prep, physics coordination and upgrade coordination and ask for them to identify any datasets that need to be kept
      - Longer term: revise the data placement policy - various discussions this week (also addressing T2 policies)
      - Reduction of replicas does not mean that the data are not accessible or not usable
      - For production single custodial tape copy is often used
      - For analysis we have DaTRi just for this: users can make the requests
        
        We don't have statistics today on how often it happens that data are recalled from tape by users, it is believed to be very low activity
        
        Moreover today the tape systems are not heavily used so even if we get a bit more requests, we can sustain the load
        
        On the other hand, deleting data from tape is a complex procedure, and tape isn’t an infinite resource
- AMOD/ADCOS Report
  - All shifts covered this week
    - Pilot factory voatlas273 decommissioned (ELOG:48290)
    - CVMFS_Stratum1_atlas degraded (ELOG:48331)
    - Minor issues at sites
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift