Tier1 and Tier-2 Weekly Report 2015-06-02

Tier1 and Tier-2 Weekly Report 2015-06-02

General News

ADC and ATLAS Computing News [20150602]
- ADC Weekly meeting
- Stable 13 TeV expected to start tomorrow
- Rucio DaTRi replacement
  - Asked various people to test - not too many comments back and most implemented already
  - Users can create rules within their quota:
    - Datasets/containers can be selected based on patterns or lists
    - Storage endpoints can be selected by name or RSE expression
    - A summary page shows dataset sizes and RSE quotas
    - Warnings if the quota is exceeded or write permission denied
  - Currently under development:
    - Possibility to request a sample of a dataset/container
    - Approval system for over-quota requests
  - The system is in a pre-production state now and can be opened to more users
    - Caveat: TAPE endpoints currently excluded
- Reprocessing
  - 2015 Cosmics Reprocessing
    - Full details can be found in the DATREP-12 JIRA ticket.
    - Inputs: 44 runs of physics_IDCosmic stream from early 2015
    - Output formats: merged-AOD, ESD, supermerged-HIST
    - Software release: 20.1.5.3(4)
    - Conditions tag: CONDBR2-BLKPA-2015-05; Geo-tag: ATLAS-R2-2015-03-01-00
    - AOD outputs are small and complete LBs
    - Slice tests (5 runs) submitted last Thursday in release 20.1.5.3
    - Full production submitted Saturday in release 20.1.5.4 (includes Tile update), one problematic run
      - 20/769 broken jobs, task is set to finished. JIRA: ATLASRECTS-2168
      - Campaign should be completed by end of today after some staging issues
  - New "exhausted" (sub-)state in JEDI seen in this production, works perfectly:
    - Delay in task going to finished allows decisions to be made without child steps being affected
    - Would be useful if reprocessing coords get a mail when a task goes to this state
    - Some discussion if "exhausted" can be given to DEFT from JEDI to display somehow in prodtask-dev.cern.ch (as well as / instead of "running")?
  - Started to do run 1 slice tests in release 20.1.5.X, beginning with 2010 and debugging problems
- Derivation Production
  - Post-mortem of 13 TeV reco and derivation productions from last week pointed out number of issues to be aware of, especially tails of T0 reco
    - Whole cycle took about 6 days; could have been 4 with automatic systems set correctly to start as previous steps completed
  - Debugging and building cache now for stable beam data this week
  - Train configurations are finalized: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/DerivationProductionTeam#Train_Configuration
- CRC Report
  - Beam expected to start tomorrow
  - Detector commissioning mostly done, all magnets on since last week
  - Trigger menu for week 1 validated and HLT algorithms validated
  - ADC reprocessing of tile slice test from May 28 started in bulk May 30
    - ~80% done; first "exhausted" tasks seen
  - New record for Rucio transfers - 2.5-3PB per day (good week for Run 1 was 6PB/week)
  - Prodsys2 doing its jobs: HLT, T0 processing, Event Index being tested
  - Bunch spacing meta-info (+ campaigns, sub-campaigns) have been agreed between AMI, Rucio, MC, GP and ProdSys2
    - Ready for the implementation (waiting confirmation from AMI)
  - No major issues from the week
- New SONAR for Run 2
  - https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/DDMSonarTests
  - Adapt to Rucio interface
  - Single file size (small, medium, large in Run 1)
    - FTS3 gives SRM overhead info
    - Less load to the mesh and easier to publish
  - Full mesh is needed, including T3s
  - Testing a single link
  - Can compare to other tests (e.g. perfsonar)
  - Results published in SSB "network measurements" view
    - https://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteview#aaSorting[0][]=7&aaSorting[0][]=desc&currentView=Network+measurements&highlight=false
    - Shows 7 day average bandwidth throughput with and without overhead and failure rate
  - All sites but two have test files
  - Full mesh transfers just started
  - To do:
    - tool for occasional transfers (testing specific channel)
    - filtering for publishing (all files, all files with same size, sonar only)
  - Discussions around only testing links which have low statistics; adding and comparing with perfSonar and xrd cp
    - Maybe measure packet loss rather than throughput?
- Managing MultiCloud Processing
  - Reminder: Multi Cloud Processing mechanism (MCP) allows resources from sites in one cloud to get jobs from tasks running in another cloud.
  - Important point is that the job output is returned back to the Tier1 of the cloud where the task belongs
  - Ideal case would have MCP turned on for all sites - allow anyone to pick up workload that has availability
  - Reality is moving data across cloud boundaries can cause issues
    - The increase of transfer load sometimes does not compensate the distribution of the load
    - Rucio transfers with higher priority or importance may get stuck
      - This may be mitigated by introducing shares.
      - Have seen times where one cloud "steals" load from other clouds and regional resources get unused, or what is worse, regional resources start getting jobs from another clouds via MCP
  - Testing PanDA metric to balance MCP load:
    - Remaining Work (RW) = (Total Events - Events Done) * Expected walltime per event / 24 / 3600
    - https://twiki.cern.ch/twiki/bin/view/PanDA/PandaJEDI#Task_brokerage
    - Calculate serial and mcore jobs independently
    - RW values are shown in BigPanda
      - score - http://bigpanda.cern.ch/dash/production/?mode=task&corecount=1
      - mcore - http://bigpanda.cern.ch/dash/production/?mode=task&corecount=8
    - MCP is enabled in resources for clouds with RW below average to clouds with RW above average
    - Exception: High memory resources have always MCP enabled by default.
    - Not observed any issues since MCP is managed based on RW
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150519]
  - April 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift