T2WeeklyReport20150317 < AtlasCanada

TWiki>

AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20150317 (2015-03-17, LeslieGroer)

EditAttach

--+ Tier1 and Tier-2 Weekly Report 2015-03-17

General News

ADC and ATLAS Computing News [20150317]
- ADC Weekly meeting * Rucio Status
  - Transfers - Dominated by production (due to panda brokering changes) ~500TB/day
  - Transfer Shares
    - FTS3 shares are used by Rucio to separate activities (e.g. T0 Export, Production, User Subscriptions)
    - Was tested and verified but still issues
    - Transfer of input data for production jobs was blocked by other activities - Under investigation
    - Missing ~20% of messages from FTS, also under investigation
  - Deletion
    - Up to 200Hz and 2PB/day - but still not aggressive enough
    - Log files at the front of the queue can lead to site filling up
      - Manual intervention to push big files to the front of the queue
    - Rucio configuration is also being balanced/optimised
    - Bulk SRM deletion will help (especially with Castor/dCache)
      - Upcoming version of gfal2 will support it
  - Blacklisting
    - Blacklisting DDM endpoints during downtime has been enabled
      - https://atlas-agis.cern.ch/agis/ddmblacklisting/list/
    - Cannot make rules for new data on blacklisted endpoints (including DATRI), unless ignore_availability flag is on (eg for Panda)
    - Deletion and transfer to blacklisted endpoints stops
    - But can still be used as a source
    - Blacklisting is done by AGIS and a collector propagates info to Rucio RSEs
- Derivation Production
  - Release 19 derivation production is fully submitted now with 3 caches
  - Full production with 19.1.4.8 just launched today (below is sub-set of tasks, more tasks will be defined):
  - Full production with 19.1.4.6+7 (4k+2k tasks) close to completion, need to take care of the tails now.
- Distributed Analysis Analysis!
  - Looking at issues with user analysis jobs
  - Cancelled Jobs
    - Waste a significant fraction of wall time.
      - Significant increase in walltime wasted since introduction of JEDI (but not number of jobs)
    - Caused by users cancelling jobs, especially when long user tasks are getting stuck!
    - An example http://bigpanda.cern.ch/task/4854180/
      - Users ask: Should I cancel and resubmit it?
      - Task has been running for a month even though it has reasonably high priority.
      - A significant fraction of failures caused by obvious site problem(s). Possibly confuses JEDI?!
  - Failed jobs
    - Lost heartbeats
    - Get / Put errors still major source of failures
    - Lots of failures because of batch limit exceeded
    - Jobs running out of memory is a growing problem
  - DDM server errors
    - There were 885 000 DDM server errors in March!!
    - Vast majority caused by incorrect characters in dataset names
      - Users were trying wildcards (*) and colons (scopes) when they weren't needed.
    - Tadashi added a check when the task is submitted: https://its.cern.ch/jira/browse/ATLASPANDA-171
    - Documentation was not clear enough.
    - Small number of HC jobs with errors that should be investigated.
  - Documentation
  - We have two Panda twikis:
    - 1. https://twiki.cern.ch/twiki/bin/view/PanDA/PandaAthena
    - 2. https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/PandaAthena
    - Tadashi is updating (1) but (2) is linked from most other ATLAS twiki pages.
    - Need to consolidate twiki pages.
  - MadGraph (Event)
    - The MadGraph generator is a frequent offender when it comes to analysis jobs using multiple CPUs
    - MadGraph uses multi-core mode by default:
      - To disable the multi-core mode the user needs to use
      - "./bin/generate_events --nb_core=1.
    - Some tests run at Glasgo show Cgroups limits this without affecting job length:
    - Some discussion about user education needed
  - Memory usage
    - Increasing number of user analysis jobs requiring more resources
    - Shouldn't be unexpected as users are frequently running small scale production for validation
    - Need to think about HIMEM (and MCORE) analysis queues
      - Setup Hospital queues in T1s?
  - Pilot improvements
    - Pilot not picking up cgroups correctly yet which would help (in future) with memory usage problems
    - Some analysis job errors because MAXWDIR and MAXINPUTSIZE are not set correctly in AGIS.
    - Twiki states:
      - MAXINPUTSIZE - 14336 MB is the default (14 GB), but can be higher. Maximum copy-to-scratch input data total. Used in Panda Brokerage. To be obsolete once MAXWDIR is used in job brokerage in
      - MAXWDIR - Maximum available workdir space per job (MB). The initial value has been set as MAXINPUTSIZE + 2GB. Not used in Panda Brokerage yet. (APF-used)
    - JEDI now uses schedconfig.maxwdir-2GB as input size.
  - Suspicious files
    - JIRA ticket tracking file recovery and suspicious file reporting: https://its.cern.ch/jira/browse/RUCIO-953
    - Can now declare lost files, but still not find suspicious files yet.
    - Some (smaller) sites that have experienced data loss continue to cause problems for users.
      - Can we improve the clean up procedure?
  - Rucio Download
    - User trying to use Rucio download as part of his prun job.
    - This should NOT be allowed.
      - If users need access to data they should use panda which will ensure there job is sent to the data
      - Following up with user to see if there is a missing use case in prun
- ATLAS Central Services issues
  - Failed network switch in CERN Geneva data center 2015-03-16, 00:15 to 16:00 CET
  - Problems with complete communication chain from CERN IT to ADC
  - Affected many services (AI, Ceph)
  - All IT services should be back as of Fri 16:00 CET
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift