Tier1 and Tier-2 Weekly Report 2015-12-08

Tier1 and Tier-2 Weekly Report 2015-12-08

General News

ADC and ATLAS Computing News [20151208]
- ADC Weekly meeting
- MC Production
  - Status of MC15b digit+reco
    - Submitted “~760M(bulk)+new requests” events and ~95% done (30M queued)
    - Due to the file losses at Taiwan, some specific samples do not have sufficient events (e.g. Powheg Z to tautau)
    - Will rerun samples for these. Discussing with Prodsys2 machinery to automate rerunning of lost jobs/files
    - Not much in the system at the moment, but some large requests in the pipeline
    - Expect the 500M fullsim in about 10 days
    - Also will run new Sherpa V+jets rather than rerun duplicated events samples
  - Set cpuTimeUnit=HS06sPerEvent and leaveLogs=yes for all tasks by default (from today 16:00).
    - Logs will be left on the DATADISK of the site where they are produced (i.e. not moved to T1)
  - Long standing AMI tag issue with processing type - merge tag has label "recon" instead of merge.AOD
- Derivation Production
  - Should be 100% complete but some tasks went into finished state without completing all the jobs
    - Changed the brokerage today to send pmerge jobs to any other site after 24h (when the site went offline after finishing the reco jobs the pmerge jobs were expiring after 24h)
    - Task completion goal will be set to 100% for data derivation tasks so that the tasks will go to “exhausted” state instead of “finished” for us to act on them
    - AMI markes the datasets made from “finished” tasks as “ALL EVENTS AVAILABLE” in the same way as for “done” tasks. This should change soon after discussions with Prodsys2 and AMI teams.
    - A set of “finished” tasks have been identified from data derivations and reran to have complete stats, PC was notified yesterday to give the message in the meeting. We will provide the list of affected datasets in the dpd twiki
  - AtlasDerivation cache based on 20.7 will be built for reprocessing, deadline to collect the tags from groups is Monday December 14th
  - One more AtlasDerivation cache based on 20.1 is also requested for EXOT2 and HIGG4D* derivations, same deadline as above will apply
- Reprocessing
  - Express stream - essentially done, using release 20.7.3.2
    - Three r-tags in the end, after periods G,H,J had to be resubmitted due to missed IBL bowing fix conditions folder
    - Question: Can empty containers be deleted as causes confusion for analyzers?
    - Some jobs with too large ESD, format really not useable anymore
    - One new crash so far, looks like problem with RAW input, see ADCSUPPORT-4407.
    - DCS/magnet crash "cured", 'all:svcMgr.IOVDbSvc.FoldersToMetaData.remove("/EXT/DCS/MAGNETS/SENSORDATA");'
    - No software crashes
    - CosmicCalo about 75% done as well.
    - DQ sign off using 10 runs planned for tomorrow or Friday.
  - Intermediate bulk reprocessing
    - Period F (7 runs) of DRAW_ZMUMU -> DAOD_ZMUMU submitted using 20.7.3.3, 78 more runs to come (rest of 25ns data)
    - Three physics main runs also submitted
    - Had to resubmit as 50ns is default and these are 25ns - third time this has happened
      - Beamspot is unaffected by this - request the 7 25ns runs at super high priority so DQ signoff can be done Friday
  - Bulk reprocessing: Target launch date: Dec 14th
- Verbal report on T0 HI processing
  - Have to limit jobs on nodes as need 25-30GB memory for 8-cores (1 job per 16-core node at Wigner)
  - Hard collisions take ~200s per event
  - Don't expect to process real time other than Express stream
- CRC/ADCoS report
  - No shifter this week
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20151001]
  - September 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift