Tier1 and Tier-2 Weekly Report 2014-02-18

Tier1 and Tier-2 Weekly Report 2014-02-18

General News

ADC and ATLAS Computing News [20140218]
- ADC Weekly meeting
- Rucio migration - new stable DDM Clients 2.5.0
  - DQ2CLients 2.5.0 - transparently support Rucio as a file catalog
  - Validated with DQ2 services, PanDA test server/pilot, CVMFS
  - PanDA jobs with Rucio - can retrieve output via WebDAV
  - Central services must be upgraded to 2.5.0
  - Old DQ2Clients won't be able to access data via Rucio
  - 2.5.0 and rucio clients 0.1.12 are deployed on the grid cvmfs area (not yet the default)
  - LAPP will be the first site migrated from LFC to Rucio and tested
- CNAF-FNAL WAN access
  - Feb 3-5 saturated LHCOPN link (max 20G currently, going to 40G)
  - CMS user analysis (XRootD) reading data from CNAF
  - Misconfiguration of XRootD at FNAL
  - General concern that single analysis can saturate the networks and difficult to debug as comes from WNs and not from e.g. FTS so can't tune
  - Would like VOs to provide middleware that tunes/smears XRootD, http etc transfers - orchestration
  - Not put burden on T1 monitoring and balance between VOs
  - KIT: Saw saturation of firewall by ATLAS XRootD
- NTUP_COMMON Production Plans
  - v1 output used by all except b-tagging which needs v2
  - Resource estimates based on AtlasPhysics-17.2.11.1.8
  - v1 production used 1.6 PB on data12 and 3.6 PB on MC
    - T1: one primary, one tape; T2: two primaries (made secondary before Xmas)
  - v2 should be 25% less space
    - Agreed to 1 primary on T2 and one copy on T1 TAPE
    - Production take about 3 weeks
    - Some long running jobs (want to preserve lumiblocks)
  - New cache built this week for more detailed egamma truth for Upgrade studies
- ATLAS SAM tests in FR Cloud
  - Investigating discrepancies between availability/reliability reports and site activities in FR cloud
  - Tests timeout when sites working well and full leading to bad availability not reflecting site health
    - Mostly due to timeouts at the batch level at site
  - Downtime not accounted correctly in reliability
  - Compared ops tests with atlas test
  - Note: Recent ATLAS studies shows timeout problems mainly at WMS level
    - Proposed solution was to increase timeout time at this level
  - Investigation shows OR-ing of statuses does not seem to account for downtimes on occassion
  - WLCG/SAM notified of this in January
    - Claim this is a known problem, that has minor impact on the overall monthly A/R numbers
    - They will try to deploy the fix for it as soon as possible
    - For January reports, they will perform a general re-computation for sites, which have been impacted by this before publishing the final reports.
  - FR Cloud conclusion: Situation still not 100% clear
    - Sites need to check downtime are well taken into account and complain if not so that reliability takes downtime well into account (should be done automatically for final January report)
    - Note: Sites need to be careful, going to VO tests could have some consequences (ex : some sites used to drain the queue in advance for downtime => ops jobs stay ok VO ones not…)
- AMOD/ADCoS report
  - Stable week
  - BNL-OSG2 (Sat-Sun)
    - dcache namespace manager suffering high-load as database queries intefering with auto-vacuum operations
  - INFN-T1 (Sat-Sun)
    - Storm server high load causing 60% production job failures. Installation of a new storm version underway in order to solve this problem.
- DDMOps T1 DATADISK
  - Current: 2.2 PB free, 560 TB secondary
  - James Catmore joined to do “Data placement coordination”
    - Coordinate data placement policy and space management with groups, physics coordination, data preparation, ADC coordination and CREM
  - February activity for T1 DATADISKS
    - removed 981 TB of intermediate mc HITS and AODs
    - set secondary 312 TB of unused data AODs
    - reduced number of primary replicas of EVNTs to 3
  - Ongoing
    - test merging old logs (770TB) with JEDI to put on tape
- Pilot error
  - Want to release bug fix version
- FTS3 down at RAL for hour tomorrow (0900-1000 CERN time)
  - ~1 day intervention needed in ~2 weeks for infrastructure upgrade
  - Switch to CERN server during this time (test beforehand)
- Rucio renaming
  - Rename HOTDISK for now (remove eventually!)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift