Tier1 and Tier-2 Weekly Report 2015-04-28

Tier1 and Tier-2 Weekly Report 2015-04-28

General News

ADC and ATLAS Computing News [20150428]
- ADC Weekly meeting
- Rucio DaTRi Replacement
  - Web Interface for Rucio Replication Rules
  - Currently under active development
  - Only for selected users - Not open for public use yet (if interested please contact rucio-dev@cern.ch)
  - Provides similar functionality as the CLI but in a multi step web interface
    - Requesting new rules
    - Listing of already created rules and their replication status
    - (in development) approval mechanism for special cases
  - Comparison of old/new interfaces
    - Before: provide data pattern and data type
    - Now: provide full dataset/container name or pattern and get a list of all matching datasets/container
    - For each datasets/container show detailed metadata
    - User can decide to continue with all matching datasets/containers or select a subset
    - See slides for details of web interface
  - Comparison old/new interface DDM endpoint selection
    - Before: provide one specific endpoint from a list of all endpoints
    - Now: either select endpoint (RSE) from list of all endpoints
    - Or provide an RSE expression where the system picks an endpoint matching the expression
  - Quotas and Approval
    - Currently working only when quota is available
      - If a rule is requested the system checks if the requesting account has enough quota on the selected RSE
      - If no quota is defined or all quota is taken up the rule is rejected
    - Approval mechanism in future versions:
      - If the account has no quota he can asks explicitly for approval of the rule
      - The rule will be in a waiting state and an approval request will be sent
    - Current Quota Set Up
      - All accounts have quota on SCRATCHDISK
      - Based on the country, accounts have quota on the LOCALGROUPDISKs of the country
      - Physics group accounts have quota on their GROUPDISKs
      - only special accounts (panda, tzero) have quota on DATADISK
- Using Optimised Software Environment Setup - see slides for details
  - Arose from discussions at S&C week in October 2013 that environment setup load on the file system could be an issue for HPC sites
  - Optimizations and modifications in CMT configurations
    - Deployed in CVMFS last year in rel 17.7.3.12
      - Showed 20x reduction in file system access calls for setup commands
    - Bug in grid installation procedure for patches resulted in delayed deployment until Nov 2014
    - Been tested at some HPC sites
      - Request to be rolled out via modifications to setup in the pilot
- Reprocessing
  - Two muon productions in release 20.1.4.6 done in a week
    - 2012 DRAW_ZMUU (286 runs) and 2012 Period B Muons stream (72 runs)
    - Very few crashes, containers now done "repro17_v02"
    - Still an issue possibly remaining to check: replication of HISTs to CERN:
      - To be done by hand dq2-ls data12_8TeV.*.physics_Muons.merge.HIST.r6614_p2320_p2320_p2320/
  - 2015 cosmic runs from Period C2 in release 20.1.4.6
    - Underway since Monday 20th April
      - Input: 23 runs, physics_CosmicMuons stream
      - Output AOD, ESD, HIST, NTUP_MUONCALIB, all to be merged (HIST x3)
    - Status today: 17/23 reco tasks done, the others are struggling
    - Lots of manual intervention due to bad_alloc crashes
    - Remaining jobs are needing 8-10 GB to get through, perhaps still not enough
      - Also ramcount=0 scouts issue limiting number of available HIGHMEM sites - now fixed?
    - Some lost files from reco, causing incomplete merges
  - Issues with workflows with tasks going to finished not done due to last job failures due to not enough memory
    - Causes issue for HIST merging
- ADCoS Report
  - Do not have full coverage for shifts in upcoming weeks - being discussed at management level
  - Issues solved
    - Jira 4963/GGUS 113179: Many transfers failing with "Protocol not supported or path/url invalid"
      - File system on one FTS server failed
    - SARA fixed broken squid
    - NDGF-t1 DATATAPE space token quota increased
  - Issues open
    - JIRA 4939: Concurrent FTS server transferring a single file - one tried to delete while other registers in rucio
      - Difficult to identify
    - Taiwan has backlog of transfers from data tape
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift