Tier1 and Tier-2 Weekly Report 2015-06-09

Tier1 and Tier-2 Weekly Report 2015-06-09

General News

ADC and ATLAS Computing News [20150609]
- ADC Weekly meeting
- MC Production
  - Grid running at max capacity last week
    - 120M simulated events, 440M of digi+reco
    - Currently close to 250M of events are already in tasks waiting to be simulated and more waiting to be submitted
    - New minbias pileup with the truth jet fix under way. Once they are distributed they will be use for standard 25ns and 50ns reconstruction
    - Fix for jobs causing huge log files already in 20.1.4.14.
    - Pileup efficiency seems about 70%
    - Some issues with some sites not setting wallclock in AGIS. Might need 2 days for some jobs
- Derivation Production
  - Open-ended derivation production on stable beam data
    - First AOD dataset appeared at Tier0 on Saturday night, open-ended production request was placed in ProdSys2
    - First tasks were defined automatically (after a small fix) on Sunday
    - Submitted manually on physics_MinBias stream. Will set up open-ended production on physics_ZeroBias
    - All 13 TeV derivation processing can be monitored at:
      - https://bigpanda.cern.ch/tasks/?tasktype=prod&user=atlas-dpd-production&project=data15_13TeV
      - https://bigpanda.cern.ch/tasks/?tasktype=prod&user=atlas-dpd-production&project=mc15_13TeV
    - Train configurations defined (16 physics, 1 min-bias) with sub-carriages
      - https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/DerivationProductionTeam#Train_Configuration
    - Production Status
      - Turnaround from first 2 runs were pretty good
        
        2 days - calibration loop (beamspot, hot/dead channel identification)
        
        28 hours - reco/merge at T0
        
        24 hours - most (11/14) derivations finished, many in 6 hours
      - 4 days achievable but takes lot of supervision at the moment, probably won't achieve this in steady states
      - Third/fourth rus had some issues (900 Hz vs 200Hz, smaller RAW event size)
        
        many individual jobs took over 12 hours + retries
        
        more typical of Run 1 experience
      - Still tuning the system
      - Around 80 derivations in ~15 trains
        
        Sizes are mostly under 1% (few under 10%)
        
        CPU times 40-400ms/evt (jet1 in particular very heavy algorithm)
- Event Index Status (see slides for details)
  - Get EI from permanent EVNT, AOD and DAOD for all data (real and simulated) based on AMI
  - ProdSys2 recognizes new datasets as they come in and defines tasks to run the EI producer
    - short jobs (30 min for 50 AOD files)
    - High T1 weights
    - Most tests OK - all jobs sent from CNAF to TRIUMF due to MCP failed
- Tape Usage at SARA-MATRIX (see slides for details)
  - Brief overview of ATLAS use of tapes at SARA-MATRIX.
  - No issues.
  - 4 of 6PB used.
  - Can likely increase usage/bandwidth without issue
- CRC Report
  - Not calm week, but not major issues
  - Daily activities at https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/ADCOperationsDailyReports2015
  - Some issues with FTS bringonline fixed but still issues at INFN-T1
    - Likely will update all FTS servers again this week
  - Been transferring about 40Hz peak, 10GB/s
  - Close 20 200k running slots used
  - HLT Reprocessing went well
    - Target was <24h, DONE within 14h20m !
    - Following up on open issues
      - Some files not being added to dataset correctly
    - Open-ended derivation ran well - one task out of 27 still running this morning
  - Timeline for EPS conference starting July 22
    - Concentrate on high-mass strong production searches using 50 ns data
    - Candidates: dijet resonance & angular, multijet & lepton+jets BH, gamma+jets
    - Push as much as possible to include all data before machine studies
      - Goal - try to reduce calbration loop from 48h to 24h and reprocessing to 24-48h
      - Expect derivations for critical physics streams can be done in 12h
      - Try to have data for users within ~3days
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150519]
  - April 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift