Tier1 and Tier-2 Weekly Report 2016-05-10

Tier1 and Tier-2 Weekly Report 2016-05-10

General News

ADC and ATLAS Computing News [20160510]
ADC Technical Coordination Meeting [20160509]
- Building Network Awareness into PanDA brokering
  - Network Weather Service
    - For each src-dst link measure nqueded, ntransferred in last 1 & 6 hours, MB/s in last 1h, 1d, 1w according to FTS, dashboard, perfSONAR and FAX
    - https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/DDMNetworkMetrics
  - AGIS closeness
    - Monthly recalculation, based on max 1h throughput bin over last month from dashboard
    - Set value 0-10 (-1 blacklisted, 11 = not known)
  - Task brokerage cap
    - Don't assign any task to a nucleus with output file transfers from the satellites piling up
    - 3k output file queue limit (could be reduced to 1k)
  - Don't assign any job to a satellite that has a long transfer queue t the task nucleus
    - 300 output file queue limit (could be reduced to 100)
  - Besides the caps, want to weight towards nuclei & satellites with good throughput and low queues
  - Add network weights to the existing job brokerage (data availability, #jobs in different states etc)
  - Network weight (currently running in passive mode)
    - Seek simple, dynamic link classification
    - NW_weight = 0.5 x Queued_weight + 0.5 x Throughput_weight
      - Queued_weight linear function; 2.0 for 0 queued to 1.0 for 300 queued
      - Throughput_weight based on first FTS measurements available (1h, 1d, 1w) - if none available, use normalized AGIS closeness
        
        typically ~1.0-1.5 for < 0-50MB/s, 2.0 for > 200 MB/s
    - Urgent jobs have 1.7 NW_weight cap
  - Other weights could be added (e.g. input file based NW weight)
- Discussions about AMI
ADC Weekly Meeting [20160510]
- MC Production
  - 4.4/4.7B events done, 270M to go for MC15c
  - Uptick in sim jobs in last day
  - Some improved monitoring now for tracking segmentation faults
  - Need some coordination to clean up recon.AOD datasets
  - Work on EVGEN datasets first as part of cleaning up duplicate event issues for MC16
- Derivations
  - MC15c - 80% done of 1200 new tasks requested last week; 2500 still submitting
  - Increase nGBperJob from 10 to 12 to increase the running time. Should have safety margin of 0.3GB
  - Longish discussion about issues seen at INFN on GPFS for derivations
    - Errors such as
      - SysError in <TFile::ReadBuffer>: error reading from file AODxxx.pool.root.1 (Stale File handle)
      - This is a ROOT error but Athena still thinks events processed and reports as such - only seen in log files
      - Need to investigate how pervasive this problem might be (might be seen in analysis jobs too?)
      - ASG-ticket: ATLASG-448, ROOT-ticket: ROOT-8169
- HI Reprocessing
  - HardProbes, MinBias, Express, CosmicCalo: All done
    - AOD merging is now requested: files are typically 1.1GB (HP) and 1.3GB (MB)
  - MinBiasOverlay launched yesterday
  - UPC - one run done, still in validation
  - HI pp data15_13TeV ESD reprocessing
    - For HI jet cross-calibration
    - In rel 20.7.5.5, validated
    - AOD output go to DATADISK
      - 2x30M/stream events, ESD size: 2-3MB/event, AOD: 0.5MB. Total AOD 32TB, 2.5s/event, 1750 cpu days
- CRC/ADCoS report
  - No report today
- New AFT/PFT templates
  - Derivation Production (Reco_tf 20.1.5.7)
    - Template 731(functional) works (~10 sites)
    - Template 751 (rc)/752(rcm) working for few weeks
    - Template 763(stress) still needs minor tweaks
    - Template 783,784(stress, both rc, 500/5000 events) ready and already in use
      - e.g. for 784: 3.5GB input, 0.35GB output, single core, 20 min WallTime. Total 400 files -> 1.4TB input size (1 file per job)
  - Reco HITStoRDO (no pileup) (Reco_tf 20.7.5.1)
    - Template 781/779/782(stress) 1/8/12-core
      - e.g. for 782: 4.2GB input, 13GB output, 20 min WallTime. Total 500 files -> 2.1TB input size (1 file per job)
    - To come: functional test template (also rc,rcm)
  - AFT Status
    - New AFTs since March: 729 (AltasDerivation 20.1.5.7)
    - Candidate AFTs
      - 775: Athena-based: AthAnalysis 2.3.48 - NEW, under development, will replace 506.
        
        Pathena job OK, ganga job fails - being debugged
      - 722: Root-based: RootAnalysis 2.3.14
        
        Running for a while and ready to replace 505 now
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20160216]
  - January 2016 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift