Tier1 and Tier-2 Weekly Report 2016-01-12

Tier1 and Tier-2 Weekly Report 2016-01-12

General News

ADC and ATLAS Computing News [20160112]
- Compute Canada Usage Stats - 2015 - see slides (pdf)
- ADC Weekly meeting
- MC Production
  - Still have about 600M events in queue
  - No bulk requests from PMG, only from Combined Performance/Physics WG
  - Expect MC15c with rel 20.7 will start from the Feb 1
  - Before that, several requests from CP is expected with rel 20.7. This is for checking performance etc of rel 20.7
- Derivation Production and Reprocessing
  - All done except for min-bias by yesterday
  - Some problems discovered though
    - Period G-J: IBL conditions not correctly loaded - waiting for sign off from DQ
    - Fixed the trigger slimming and doing final checks, about to build AtlasDerivation-20.7.3.2
    - Possible all periods affected by vertexing issue - likely need to redo all
  - Were going to rerun all RAW again to produce DESD so could reproduce the derivations at the same time
    - Likely a week or two from a fix and new cache plus validation
    - Will need to discuss with ADC about obsoleting the produced AODs to make space
  - Pixels and V.der Meer need new calibration which likely only available in 21.x and not 20.7
- CRC/ADCoS report
  - Decent usage last week (up to 250k job slots)
  - Network issue between EU and CA sites on 7-11 January
  - Disk space running out at a few Tier-1s and some Tier-2s
  - MadGraph jobs create enormous tarball of log files (incorrect settings) that may crash the jobs (no memory) or result in lost heartbeat and produce dark data (see below)
  - Pinkpanda monitoring now disabled
  - Found corrupted files at RAL. Understood as interference between the job recovery mechanism and re-running of failed jobs by Condor. Now fixed.
  - Very few ADCoS senior shifters this week
- Dark Data
  - Seen about 3PB of dark data produced at sites in last couple of weeks
  - Caused by the madgraph event generator tasks producing huge (15GB) log tarballs over Xmas
  - Logs of successful jobs have already been cleaned
  - Dark data is produced by 'lost heartbeat' failures
    - Worker node grinds to a halt processing the tarball so pilot gets stuck
    - Panda times out the job and retries
    - Pilot finally gets unstuck and uploads log tarball to SE
    - But since panda already cancelled the job it is not registered in Rucio
    - One example job: http://bigpanda.cern.ch/job?pandaid=2720131446
    - Some improvements could be done in the pilot to clean up after cancelled jobs
  - A consistency check is the only way to detect dark data
  - Thanks to all the sites who responded to the tickets to create regular storage dumps
  - Will contact the worst-affected sites soon to make manual dumps in order to clean as soon as possible
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20151001]
  - September 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift