Tier1 and Tier-2 Weekly Report 2014-08-19

Tier1 and Tier-2 Weekly Report 2014-08-19

General News

ADC and ATLAS Computing News [20140819]
- ADC Weekly meeting
- Rucio Status
  - New version (0.1.31) released Monday
  - File replica lookup stabilized over last two weeks
  - The stress test load will be increased back to 1M files per day
  - Stress test uses FTS3 pilot instance in CERN
  - It is now configured with a DB backend separate from the production FTS3 instance
  - Waiting for the deletion backlog to be consumed before actually increasing the load (mostly US USERDISKS)
  - HC able to submit to rucio
  - Plan to run HC tests processing input from DQ2 and registering output to Rucio
  - Next step is to run validation tasks on data cloned in Rucio
    - To test PanDA workflow for creating input (_dis) datasets and output (_sub) datasets when running many jobs for one task
- DDM Deletion Status
  - In DQ2, deletion service made of Resolver, CatalogCleaner, StorageCleaner
  - Independent services, only supported SRM deletion
  - In Rucio, the daemon responsible for deletion is called the reaper.
    - Takes the file replicas with a "tombstone" and mark them as being deleted.
    - Sends a deletion request to the Storage Element
    - Multiple protocols can be used (SRM, WebDAV...)
    - When the replicas has been physically deleted, it is removed
    - Reduces the probability to close to zero of a file still being registered in catalog and not being on the storage (or vice versa)
    - Use of other protocols like WebDAV allows to have multiple Storage Cleaner for each endpoints
    - Can also be used for SRMless sites (the so-called gridFTP sites)
    - Rucio now send deletion events to the DDM Dashboard
    - Tested SRMless deletion via WebDAV at large number of sites (right now deletion limited to datasets from Rucio stress-tests and deletion of Dark Data)
    - As soon as the production system starts writing some new datasets in Rucio (coming soon), the potential number of datasets deletable via WebDAV will grow
    - Rates of ~50Hz achievable over a day (tested on US sites during dark data deletions with 20 parallel cleaners per site) without significant extra load on resources
    - Still about 30 sites without WebDAV
- JEDI Status
  - Users informed prior in various ways
  - Migration last week hampered initially by stratum 0 update (could not update cvmfs)
  - Bit hectic for support
  - Gone from v0.5.7 to v0.5.13 in a week resolving a number of issues
  - Memory consumption of JEDI is large. Optimization or more nodes (currently 2 nodes) will be required
  - http://bigpanda.cern.ch/ had no problems handling load
  - Lots of monitor changes made, primarily adding new features (requested by users) rather than fixing problems
  - Did not prepare users enough for the switch to 'task'
    - It's not about jobs, it's about the fraction of the input datasetprocessed
  - A separate 'task' tutorial is needed to complement the in-monitor documentation
  - Most ennquiries handled by DAST OK (5-10 / day)
  - Main problems:
    - Transient problems. (First submission under classic panda, resubmission under JEDI)
    - Naming of output datasets. (Ongoing discussion about how to improve this)
    - Understanding new task states. (Lots of users had 'broken' tasks, this is also being worked on)
  - Expect a few more weeks before back to normal load
- Tape experience at TRIUMF - Reda Tafirout
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift