Tier1 and Tier-2 Weekly Report 2015-01-20

Tier1 and Tier-2 Weekly Report 2015-01-20

General News

ADC and ATLAS Computing News [20150120]
- ADC Weekly meeting
- Rucio Status
  - Lost file recovery is now working - loss at SARA of 500k files last week was good stress test (see later)
  - Backlog of tickets has been treated by DDM ops
    - Reminder: please create JIRA tickets for lost files/LOCALGROUPDISK for the next few weeks
  - Developers are working on monitoring and an improved permissions model
  - Recovery activity can be monitored on the DDM dashboard
- MC Production
  - ProdSys2 Status
    - Sample extension : Fix implemented and working, some more use cases being checked - PRODSYS-328
    - Several datasets in container : Fix implemented and running. Checks ongoing - PRODSYS-327
    - . Circular buffer for pileup files : Pileup files were not being re-used so large digi+reco tasks crash - PRODSYS-351 - Fix implemented need to be deployed and can be done on-the-fly
    - Group contacts : Need final implementations to open the ProdSys2 interface to group contacts . PRODSYS-354
  - Almost all the requests have now been placed in ProdSys2. Still some issues to follow but system is getting more stable.
  - Simulation (last week): ran 10 Mevents/day
    - mc12_8TeV: 22M, mc14_13TeV: 41M
  - Monitor what has been submitted at link below (preliminary numbers using DEfT API) :
  - http://atlas-project-mc-production.web.cern.ch/atlas-project-mc-production/requests/MCGraphs
  - To be processed in system: 37M Full Sim, 79M Reco, 13M AFII Sim, 16M AFII
  - 50ns request (mostly digi+reco) will be submitted shortly if the validation is successful. This has been pending until configuration validated
  - Initial list of MC15 simulation should be ready shortly. Once we get the go ahead from validation
  - Need to standardize some requests to 5k for evgen so can add to old metadata prodsys1 files
- Physics Validation Report
  - Remaining tests before ProdSys2 could be validated:
    - Test results from tracking comparing a number of physics traits:
      - Primary and seconday impact parameters
      - Position of the hard-scatter and secondary interactions with materials
    - Not seeing any significant differences beyond statistics from the two samples, so looks good for tracking overall
    - Comparing running twice the same simulation in Prodsys2 - no striking difference
    - Compare two samples, one done with a large random offset - differences seem to be understood
    - See slides for plots and other details of the validation.
- SARA Data Loss Incident
  - Friday, SARA reported lost ~500k files (RAID0 failure made of 3xRAID6 systems; SAS expander backplane failed)
  - As part of file recovery, files were declared lost to Rucio by DDM ops:
    - Non-log DATADISK, SCRATCHDISK, logs DATADISK
    - DATADISK Lost/Recoverable/Total: 327857/98288/426145
    - SCRATCHDISK Lost/Recoverable/Total: 39764/6210/45974
  - About 1/4 of data files deemed recoverable
  - Processing of the bad files by Rucio was very fast, and most data recovered that can be
    - Workflow is much simpler than DQ2
    - Some lost files will be used to test prodsys2 recovery
- GDP
  - Ensure that all operational procedures are available with Prodsys2/JEDI which used to use with Prodsys1/Panda
  - Are back at ~100k grid job slots utilized, but still only about half for production
  - Still many ongoing issues, some minor, some major
  - Main outstanding issues in production:
    - The dominant error rate recently is due to the submission of large simulation samples where the secondary minbias pileup or cavern background inputs could not be reused. This happens only to big tasks, each job uses few secondary inputs but when they are all used, the next jobs should start reusing the inputs.
    - Distribution of secondary inputs - The tasks cannot be assigned to the clouds because the primary and the secondary input datasets are not located in the same cloud. This need manual intervention to create rules ad-hoc.
    - Data file loss recoveries - In principle we can now recover missing outputs resubmitting the corresponding jobs from finished or completed tasks. With the opportunity of 500 k files lost at SARA we will start developing the operational procedure.
    - Derivation production merging - Unable to 100% complete the derivation production submitted last December. Suspect loopers in some of the internal merging jobs
- New minor pilot release (tomorrow?) for event service
- T2 Sites with no MCORE queues should update the ticket https://its.cern.ch/jira/browse/ADCSUPPORT-4117
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Sites should be providing the 2014 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift