Tier1 and Tier-2 Weekly Report 2014-11-04

Tier1 and Tier-2 Weekly Report 2014-11-04

General News

ADC and ATLAS Computing News [20141104]
- ADC Weekly meeting
- Rucio Status
  - Almost ready for final stage of commissioning - having all datasets in and managed by Rucio
  - A lot of effort has been done recently to get to this point - working on final small details
    - How to map DQ2 subscriptions to Rucio rules/locks
    - Mapping of DQ2 replica classes primary, secondary etc
    - Mapping of DQ2 dataset owner to Rucio account
    - Protection for custodial data
  - Migrating 6M datasets will take a long time
    - Done by DB procedure run by Rucio dev, but most (unused by running jobs) can be done in background
    - Reminder: New DQ2 client can use datasets in DQ2 and Rucio
  - ~50k datasets are "active" in panda/DDM subscriptions at any one point
  - To move those have to pause panda and stop DQ2 site services
  - Any open subscriptions will be picked up by Rucio
  - A date has not yet been set but likely mid-November
    - First the procedure needs to be thoroughly tested
    - Still a few small Rucio bugs to fix
- Derivation Production
  - Full production on DC14 8TeV MC and period B samples was launched with AtlasDerivation-19.1.4.1 on October 20th
    - Data processing essentially done
      - Input data sets 60-80TB, output datasets about 70,77% for Muon and Egamma and 24% for JetTauEtmiss
    - MC with skimming done, with no skimming about half done
      - Output size about 32% for skimming
  - Validation tasks on DC14 13TeV MC submitted on Monday with AtlasDerivation-19.1.4.2
    - Good progress so far except the merging ratio. Requested files to be merged to 1GB instead of the default 4GB, however files are around ~100MB and less, being investigated
    - Full production to be launched earliest possible (today?)
      - Total input volume 222TB - can make rough estimates on output size but some groups are using truth thinning and trigger skimming so this will tend to push these sizes down
      - There are a lot of no-skim SUSY requests in 13 TeV so this might be a bigger portion.
- MC Production
  - Grid not too busy in recent weeks, and not much in system right now
    - 140M fullsim and 180M AF-II to be submitted (from requests in last couple of days plus some special configurations which are currently being set up)
    - Also will ask PC about more requests
  - ProdSys II production
    - Currently most DC14 requests being sent to ProdSysII
    - Amounting to only ~150k FullSim so far (still learning the system)
    - More to be sent in the next week, ~20M.
    - Will start with MC14 standard requests - Validation sample A ready - should get feedback soon.
  - Some work needed on notifications, features needed in JIRA ticket request - needed within a couple of weeks
- Experience on setting up xrootd and webDAV at IN2P3-LAPP
  - Main issues with monitoring in a multi-VO environment
- PanDA Pilot Update
  - Next pilot version 59d
  - Numerous minor fixes/updates
    - includes glExec update from Edward Karavakis to resolve issues at two Canadian sites that do not have the $GLEXEC_LOCATION environment variable set
    - Also need to skip core count check to facilitate installation jobs on MCORE sites
  - PanDA Pilot now available in GitHub - https://github.com/PanDAWMS/pilot
    - Discussing now how to update the tarball scripts
    - Idea: wrappers will primarily cp the pilot from cvmfs, and secondary/alternatively using https from server
    - Pilot source in cvmfs updated every 4 hours, immediately upon request for emergencies
  - Logs to BNL object store for prod/user jobs
    - Not replacing standard SE transfers
    - Starting with a couple of US queues (at SWT2)
    - Tests pending release of new pilot
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift