Tier1 and Tier-2 Weekly Report 2015-11-10

Tier1 and Tier-2 Weekly Report 2015-11-10

General News

ADC and ATLAS Computing News [20151110]
- ADC Weekly meeting
- MC Production
  - MC15b digi+reco started last weekend
    - Submitted 120M / 40M done already
    - Expect additional 500M
    - Expect to process about 25M a day; should have less evgen and simulation and more digit+reco in the system soon ~ 2 weeks
  - Duplication issue
    - Delay a week before doing bulk obsoletion of MC samples for checking. Need to be careful as there are duplications in both evgen and simul level
  - 500M events HPC simulation
    - Waiting on some decisions to define tasks (100M diphoton, 400M V+jets) and which sites to send to
    - Number of events per job? Special project mode settings?
    - Variable number of event per job could be an issue if required
  - WORLD cloud
    - Rod has requested a complete request with all steps to be pre-assigned to the WORLD cloud as this has not been tested yet. Identify suitable request today or tomorrow
- Derivation Production
  - No report - things seem to be stable
- Tier-0 report
  - Smooth processing of last pp runs
    - pp data taking stopped on November 4 (morning) - in machine studies now
    - Tier-0 processing finished yesterday morning, as foreseen
  - Reprocessing and testing activities
    - CAF: processing of a couple of older runs for L1Calo group
    - Test processing of calibration_DataScouting_05_Jets stream
    - Preparations for HI: reprocessing of few selected data11_hi RAW datasets with 20.1.8.5
  - Spill-over exercise:
    - Successfully running since one week in parallel with T0 ~ 25 k slots
    - Sent to T1s at begining of week
    - Processed all available DQM HIST at Tier-0 (5 runs so far, 400M events?)
    - Issues encountered
      - Merging of data failing when jobs with no output files
        
        Solved by modifying pmerge and pilots
      - Spill-over started slowly (~ 1k slots)
        
        Boosted setting job type from reprocessing to urgent
      - Memory requirement is large and typically runs on the high memory queues
      - Induced quite a high level of transfer (a lot of files)
      - Non merged datasets were registered to AMI
        
        Some discussion about merging in general (on node)
    - For future:
      - Spill-over needs to be properly labelled for monitoring
      - AMI registration of produced datasets to be checked
      - Discuss how to handle priorities
- CRC/ADCoS report
  - Up to 250k job slots in use at end of week (mostly opportunistic resources - CERN trigger farm, NERSC, ORNL)
  - ~ 1PB/day being moved, 50k-75k files
  - PRODDISK removal done except for few US sites
  - TAIWAN tape fully decommissioned - 250k files lost
  - Different pilots users (DN) to allow dynamic sharing of mcore/score - needed for INFN-T1 using LFS
  - FTS bring-online issue at BNL affecting staging - fixed by developers
  - Tapes are getting heavy usage and disk buffers filling up
    - Reduced weight in PanDA for brokering jobs to sites with tape replica
      - Seems to have helped
    - Improved monitoring for different stages of data subscription has helped to find this
  - Only about 1/2 senior shifts are covered in next few weeks
- Characterizing Analysis Activity on the Grid
  - Use Kibana data to look at things like is derivation production succesful, are people doing private filtering on AODs, how much processing do analysis people need, etc, etc
  - Advantages of Kibana (MWT2, Ilija)
    - Jobs/tasks info from panda DB (available with 3-day delay)
    - Allows for more complex query/more information/faster than ATLAS dashboard
    - Manipulation of variables is limited - more complex analysis would require scripting in Pig
    - Some discussion about how memory is measured and reported and if correct in all cases
    - Few slides on error codes and distribution
    - Some metrics from last 30 days:
      - Efficiency: 80%
      - 9.7M jobs, 745 users, average (max) walltime = 1 (160) hour, Average queue time 1.8 hrs, avg input: 5 files, 4.6GB, avg output: 1.7 files, 750MB
      - Failures from transformation (75%) and pilot (25%)
        
        Pilot failures mostly by too long jobs/too high memory requirements
      - Many (more than 200!) users still running on AODs
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20151001]
  - September 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift