Tier1 and Tier-2 Weekly Report 2015-01-06

Tier1 and Tier-2 Weekly Report 2015-01-06

General News

ADC and ATLAS Computing News [20150106]
- ADC Weekly meeting
- Rucio
  - No major issues during winter break
    - Rucio servers and daemons running OK
    - Free space in T1+T2 DATADISKs remained under the critical threshold
  - Many small issues
    - Deletion requests from LOCALGROUPDISK
    - Missing users from Rucio (but present in VOMS added manually)
  - Current high priority issues
    - Staging from TZERO broken
      - Files are staged but PanDA does not get the proper information about the success
    - Ownership of replicas and permission configuration
      - Group managers do not own the data in group disks and so they are not allowed to delete them
        
        Fixed for old data
        
        Will be fixed for newly subscribed data
      - LOCALGROUPDISK replicas can be removed only by their owners
        
        Correct policy is to allow management of the LOCALGROUPDISK to users with production role within given voms group (per country)
        
        Will be addressed in the next release of Rucio
        
        For now, managers are asked to create JIRA ticket in ATLDDMOPS project
- MC Production
  - Week of Dec 22
    - ProdSys2 requests were aborted/obsoleted.
      - Files in the containers produced in those requests were removed
      - A couple of chains were submitted to check the differences (fluctuations seen) in validation
      - Parameters for randomSeed and jobNumber were modified to use the proper JEDI variable
        
        Not as straight forward as expected; needed some iterations
      - Some requests were submitted once the above was confirmed.
  - Week of Dec 29
    - More requests submitted
    - Issue around Dec 31 - Tasks would stay in registered state and not proceed. Fixed now.
    - Did not lead to increase in running jobs as expected due to issues with problems with larger requests
- ProdSys2 Remaining Issues
  - Fixes in to allow sample extensions and working
  - Several datasets in container
    - This needs to be fixed soon - high priority
    - For large (> 10M simulation chains) hitting the limit of the 100k files.
      - Need to confirm that manually splitting the chains would work as expected (inputs will not be used twice)
      - Splitting to be done automatically.
    - Dealing with very large numbers of slices e.g. pMSSM request with 1000s of datasets
      - Being looked at now
    - Change of the randomSeed still not completely solved ("Failures from "randomSeed='' is not of type int..)
      - Needs to be fixed in DEfT - Tadashi is looking at this today
  - Plan to submit as much as possible but need help getting some of the tasks through
  - Physics validation tasks with identical s-tags are running - will be checked as soon as jobs are done
- ADCoS Report
  - Most shifts covered in AP+US during break, about half of EU
  - Few problems resolved during the break
    - PilotFactories degraded (ELOG:52348, 52363)
    - Global CVMFS degraded (ELOG:52378, 52395)
  - Still some unresolved issues with transfers from CERN-PROD failing with "No such file or directory" (GGUS:111013)
  - Also, job failures possibly caused by DDM issues - Unknown RSE (ADCSUPPORT-4103), Setupper._subscribeDistpatchDB() could not register location (ADCSUPPORT-4105), and Could not add files to DDM (ADCSUPPORT-4108)
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift