Tier1 and Tier-2 Weekly Report 2015-08-25

Tier1 and Tier-2 Weekly Report 2015-08-25

General News

ADC and ATLAS Computing News [20150825]
- ADC Weekly meeting
- MC Production Status
  - Grid fairly busy with more simul and dig+reco last week
  - Still many jobs in pipeline
  - Have about 100M fullsim and digi+reco to be submitted in the next week
  - Also expect ~500M evgen (single core) coming soon
- Group Production
  - Concern about fairshare and time taken to start some tasks being discussed
- Reprocessing
  - Bulk reprocessing started last Friday and essentiall completed today
    - https://twiki.cern.ch/twiki/bin/view/Atlas/August2015Run2Reprocessing
    - https://its.cern.ch/jira/browse/DATREP-25
  - Repro post-mortem
    - Still two outstanding problems:
      - Missing conditions crashes (DCS/magnet), ATLASRECTS-2221, still happening for new runs
      - Too large ESD problem, ATEAM-148
      - both mainly in express/CosmicCalo streams,
    - Many jobs timed out in transfer at TW... we should stop using those sites?
    - Need some sort of quota, have to decide if reprocessing is a "above all else" priority
      - 60% for a few days (instead of 40%) would have probably done it (instead of being part of all)
      - Really slow to get merges done - all of those should run in < 1 day, but it's taken 3-5 days
    - Still something funny going on when a merge takes 5 days / 6 sites:
      - One example succeeded in 15 mins after manual submission to ATLAS_BNL_2 after 5 JEDI reassignments in 5 days
    - Some problems at CERN-PROD_CLOUD_MCORE, due to "bad nodes"?
      - last minBias reco jobs successful on 25th attempt!
    - TAG merges still running, is long time 24h...?
    - Panda numbers in monitoring wrong again, don't know why anything has changed - will open JIRA ticket
- Tier-0 report
  - Tier-0 kept fully busy with processing of 25ns runs
  - Some manual interventions to speed things up
    - Killing and redirection of slow jobs
    - Killing of Grid jobs to free slots faster, on request by DataPrep coordinator
  - Van-der-Meer scans finish today
    - Processing of calibration streams will keep Tier-0 busy the rest of the week
- PanDA Pilot Update - v63.0
  - Memory monitor utility update - restart in case of crash
  - Support for multiple clone of jobs
    - Meant for high-priority tasks - either "runonce" or "storeonce" modes
  - Monitor and report of payload work directory size
    - Need to add handles for sites on shared file systems to disable expensive du checks
  - Now setting job status to failed when stage-out stuck (so not stuck in transfer state until later timeout)
  - FAX updates to support T3 access and to correctly handle file paths for derived datasets
  - Some updates to Event Service
- ADC News/Report
  - Few minor issues at sites
  - Record amount of analysis is causing problems to MC productions
    - Asked for share of analysis on T2s reduced (50% -> 25%)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20150707]
  - June 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift