Tier1 and Tier-2 Weekly Report 2015-09-15

Tier1 and Tier-2 Weekly Report 2015-09-15

General News

ADC and ATLAS Computing News [20150915]
- ADC Weekly meeting
- From last week's meeting - summary of DDM activities
- MC Production Status
  - Expect more score jobs for a while and maybe dip in production
- Derivation Production
  - Submitted 144 tasks on Sept 10 to process new data after technical stop, using the same technical containers as before since no change at Tier0 for reco:
    - data15_13TeV.physics_Main.OpenEnded.AOD.t0prod19
    - data15_13TeV.physics_ZeroBias.OpenEnded.AOD.t0prod19
    - Some tasks stuck in merging in DE cloud - geing followed up by GDP
  - MC derivations production should be opened up to group this week
- Tier-0 report
  - Smooth production - no problem with clearing backlog
  - Have seen issues with pushing out grid jobs due to too long MCORE jobs (12 hours)
  - Working on performance and efficiency optimizations - have a dedicated small sub-cluster
    - Optimize workflows inside athena and transformations to avoid unnecessary IO (causing stress on spinning disks at Wigner)
- ADC News/Report
  - Up to 250k job slots being used
  - Some issues with RAL FTS3 (backend database) causing drop off on job completions in UK, NL and IT clouds
  - FTS software updates done at BNL, scheduled for CERN and RAL - fix problems with message handling, allows Rucio to give up on active polling
  - Last Thursday 3 of 4 transatlantic 100G links down (only NY-LON up). Boston-Amsterdam repaired today
  - Wed, Sep 9 - Rucio overloaded by 3M FTS requests, mostly at RAL, BNL. Caused by distribution of phys-generevgen input to Tier1s
    - Considering change in the policy for submitting transfer requests to FTS, to keep their number to reasonable value
  - Fri, Sep 11 - Large increase in the number of transferring jobs in UK, IT, NL (served by RAL FTS). Situation levelling off now. But it seems that it started having impact on the production level at some of these clouds
  - Fixed issue with stuck transfers when exporting Tier0 to TAPE
  - CERN-PROD DATADISK: deployed additional 300 TB, was low on disk
  - Issue on CERN-PROD queue where HC whitelisting can be slow due to priority competition on public resource
  - HC 32-bit test switching on/off large sites, having problems with long mounted cvmfs. To be replaced by new mc15 test, in development
- Derivation Heatmaps - see slides for lots of detail
  - Run 2 code allows collection of samples of user activity (grid and non-grid jobs) - being logged in Hadoop and mined
  - Look at August - grid-only user jobs
    - AODs 60% of jobs, Derivations (DAODs) 26%, HITS popular (beginning of Run so understandable)
    - Lots of jobs use only select branches - DAOD_HIGG1D1 use about 70% but many are < 50%
    - Question being considered - are the DAODs too "fat"? Should they be split more?
    - Lots of derivations would work well via remote access as relatively little data accessed
    - Lots more to be gleaned from this meta-analysis and to track over time
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150707]
  - June 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift