Tier1 and Tier-2 Weekly Report 2014-07-22

Tier1 and Tier-2 Weekly Report 2014-07-22

General News

ADC and ATLAS Computing News [20140722]
- ADC Weekly meeting
- Rucio migration
  - Won't increase throughput for this week (100k/day)
  - Will set up new stream of testing datasets (in current folders) distributed with 3 replicas with "weight" based on MoU shares.
  - Still testing new DQ2 clients ( that will allow to register datasets in Rucio transparently) before exposing to end-users
- Dark Data Deletion
  - Deletion service stable for the last week (+3 nodes; 8 total)
  - Majority of backlog done (160k datasets in US sites to go; down from 727k total last week) - couple of days
  - Dark data in USERDISKs understood - caused by PanDA bug (fixed in Jan)
  - Files registered in LFC and unregistered from DQ2 (but not LFC/Rucio)
  - Running consistency check between Rucio and files known to DQ2 (per site) now
    - Preliminary results:
    - MWT2 / BNL / AGLT2 : 4.4M (25TB) / 6.3M (120TB) / 3.7M (55TB)
  - DDM ops will declare them to Rucio and the files will be deleted by Reaper (Rucio deletion agent)
  - Good opportunity to try deletion through HTTP/WebDAV doors
    - US will look into this
  - Next steps: Rucio will take care of inconsistencies between dataset definition and file catalog when migrated
  - For inconsistencies between file catalog and storage elements:
    - DDM ops is adapting scripts described in DDM ops twiki to work with Rucio dumps
    - Consistency check tool (CCC) developed in MWT2 could be useful, sites and cloud squads could discuss with MWT2 experts if the tool can be used for their sites)
    - Discuss any deletion of dark data in atlas-dq2-ops e-group or in ATLDDMOPS JIRA
- MC Production
  - ProdSys2 commissioning
    - sim: : Discrepancy between number of events requested and processed (needs JIRA)
    - pile: Issue with memory problems some parameters do not seem to be passed to the transform
    - post-prod: retry implemented and seems to work but need to test behavior for rest of the chain in JEDI
  - DC14
    - 8TeV (pile): still queuing in the system with 55Mevents
    - 13 TeV (simul): still queuing in the system with 120 Mevents (80M mcore)
    - MCore: present throughput is enough
- AMOD/ADCoS
  - Central services degraded - ATLAS_DDM_VOBOX lyon, cnaf
  - FTS3 Update: short failure period in all clouds
  - Twiki pages down for a few hours on Sunday
  - Taiwan network problems, solved Monday
  - SARA MCTAPE full - dCache not deleting files although space is reclaimable - bug reported
  - Grid not completely full during last week
    - Jul 15-16 about 25-30% of resources underutilized
    - Causes still being investigated. Were some DDM transfers stuck/delayed due to staging SS node with credential errors or restarts
    - Large number of DDM lookup error caused many jobs stuck in WAITING and not going to ASSIGNED
- AOB
  - Staging issues caused when same tape holds different sets of data. Repeated back and forth of the tape causes delay and stress to systems. More on this to be discussed at future meetings
  - DAST: 2 of North America shifters will be lost in the next weeks, putting us in crisis mode again.
  - New draft for data placement strategy being circulated to ADC and CREM mailing list.
  - 2 main points:
    - All data will have a lifetime, also the ones on tape.
    - There should be very few things pinned on disk. The data on disks should be organized based on priority, priority still to be defined, based on frequency of access in the past and forecast of usage for the future.
    - More traffic most probably from/to tape
    - The draft currently discusses strategy, the implementations will be done incrementally.
  - Need time to switch from classic panda to JEDI (also bigpandamon). Maybe next week?
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift