Tier1 and Tier-2 Weekly Report 2014-12-16

Tier1 and Tier-2 Weekly Report 2014-12-16

General News

ADC and ATLAS Computing News [20141216]
- ADC Weekly meeting
- Rucio
  - Slow transfers - Problem at BNL FTS3 over weekend affected whole system
  - Staging from tape was broken until yesterday
  - Message consumer for FTS transfers is turned off due to database contention issues caused by first issue
  - Deletion is on for all endpoints except TAPE and LOCALGROUPDISK
    - We urgently need space on T1s before Xmas break
  - Problems with webdav on many dCache sites (suspect v2.10+). Deletion was switched to SRM
  - Blacklisting/downtimes: A new collector was written for AGIS for automatic blacklisting
  - Recovery of lost files is being tested
  - Accounting is getting there
    - http://dashb-atlas-ddm-acc.cern.ch/dashboard/request.py/ddmaccounting
    - Old DQ2 info at http://dashb-atlas-ddm-acc-dq2.cern.ch/dashboard/request.py/ddmaccounting
  - Fixes in DQ2 clients for panda/DaTRI
  - Message for cloud squads
    - Declaring lost files or deleting data can only be done by central operations at the moment, due to permission issues
    - Please fill a JIRA ticket at https://its.cern.ch/jira/browse/ATLDDMOPS
  - Everything related to DQ2 will be switched off TOMORROW
    - No bourricot monitoring
    - No DQ2 clients < 2.6 will work
    - The central catalog database will be kept for data mining etc.
- Group Production
  - 13 TeV Derivation Production
    - 8 broken derivations from the first round submitted last week with new cache AtlasDerivation-19.1.4.4 (about 2/3 done)
    - Tail of the first round of production to be taken care of - still need to determine which tasks should be retried among finished/failed/aborted
  - 8 TeV Derivation Production
    - 2nd round of production is over, tail is to be taken care of once determine which tasks should be retried among finished/failed/aborted
    - Waiting for activeMQ messages to be turned on by Rucio team, current dataset counts in AMI
  - Production Plans Over Holidays
    - No new production planned
    - All production requests have been submitted and in the system already
    - Would be great to finish up the remaining jobs/tasks by the end of this week
- MC Production
  - Production System : Main issues
    - Duplicated events : using output from failed jobs in the next steps of the chain.
      - Need to know the samples affected.
    - Pileup distribution wrong in retries due to wrong random number seed - fixed by using SEQNUMBER for the random seed and jobNumber
    - Extending Samples : Discovery of previous files processed with the same configuration such that we do not reprocess same things again - PRODSYS-328
    - Several datasets in container : Now we can not process a task (digi+reco) which has to use two datasets - PRODSYS-327
  - Production System : Xmas break and Storage
    - About 100-150 Mevts in the system being processed now (~150 TB)
    - Pending requests so far : 160 Mevts Full Sim Mevts (~200TB)
    - Expected : requests will arrive up to Thursday, the optimistic scenario would include the min bias samples for MC15 during Xmas
    - If need to reprocess everything since Rucio started : 30 Mevts Simulation, 140 Mevts digi+reco (no additional storgage)
    - Adding all together ~ 300 Mevts simulation and 400 digi+reco.
      - This would keep the grid busy mostly for the Xmas break, almost all is multicore.
      - Probably add a 20-40% safety margin to the numbers due to last minute requests
- ADCoS Report
  - Data transfer failures to several sites with full space tokens. Part of the reason recent pause in deletion service. Several Jira tickets opened on full space tokens.
  - Some job failures due to full spacetokens
  - Some open tickets about the absence of auto-blacklisting, while it is decommissioned as part of dq2 and reimplemented in Rucio
  - FZK-LCG2, (GGUS: 110693) job failures with stage-in error, caused by dCache bug, when gridFTP doors get unresponsive.
    - Waiting for a fix from dCache developers. Also significant level of deletion errors, looks like also related to new dCache version.
  - High job failure rate yesterday at all multicore queues, with could not register location error (eLog:52258). Due to conditions when RSE was considered invalid and replication rule failed.
  - Shifters finding it a little difficult to monitor deletion services and need some guidance from experts
- Next meeting: Jan 6, 2015
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift