Tier1 and Tier-2 Weekly Report 2016-02-02

Tier1 and Tier-2 Weekly Report 2016-02-02

General News

ADC and ATLAS Computing News [20160202]
- ADC Weekly meeting
- MC Production
  - Running steadily this week - not many evgen at moment
  - Retention of Log files set by CREM policy a while back (1 year, 3 months for broken/aborted tasks)
- Derivations
  - Won't use open-ended mode for reprocessing right now as some groups are updating their derivations
  - Based on the progress and reprocessing status, plan to launch the validation run in ~2 weeks and full production in ~3 weeks
  - Data processing requests on full 2015 data with AtlasDerivation-20.1.9.5 launched on 2 trains:
    - HIGG3D1,HIGG4D1,HIGG4D2,HIGG4D3,HIGG4D4,HIGG5D1,HIGG5D2,HIGG5D3
    - EXOT5, JETM1, TOPQ1, TAUP1, TAUP3
  - Data processing requests on full 2015 data with AtlasDerivation-20.1.9.6 will launch on 3 trains:
    - SUSY4, FTAG1,HIGG6D1,HIGG6D2, JETM4, JETM8
    - BPHY1,HIGG2D5
    - EXOT3, EXOT4, EXOT8, EXOT11, EXOT12, EXOT13, SUSY10
  - Data deletion policy for "obsoleted" and "old and good" derivations is being discussed at CREM and PC meetings. Technicalities for a possible change for not deleting "old and good" at LOCALGROUPDISKs will be discussed at the ProdSys2 meeting next Tuesday, Feb. 9th
- Reprocessing
  - 2012 Jet-MET - 1/3 done last week. Went smoothly. Launching second 1/3 now
  - No news on 2015 reprocessing
- CRC/ADCoS report
  - Normal production activity - mainly simulation, pile-up, and analysis
  - T0 activity - Ongoing tests of new release 20.7.X to measure performances and optimise CPU efficiency and workflows
  - WORLD cloud - Tasks assigned, tests ongoing (mainly T1 as nucleus, few T2s)
  - Lot of pending analysis jobs (Moriond around the corner...) but lower level of activity than usual
    - Seems due to ATLAS pilot factory inducing CREAM-CE instability
      - Impacting mostly NL (no jobs), DE and RU
      - Pilot factories that tried to create job leases with lease creation time in the past inducing high load in CREAM database and pilots failing
      - Experts working on solution...
    - RAL also blacklisted because of problem on disk server
    - WORLD cloud - Tasks stuck because sent to a nucleus with data but not enough memory to run the task
      - Request sent to get panda monitoring with sites chosen as nucleus/satellites
  - Fair amount of data consolidation causing some T1 full issues + Data staging for 2012 reprocessing
    - FZK and PIC datadisks full (almost no secondary) - rebalanced to NDGF
  - CERN SCRATCHDISK filled up during the weekend
    - Caused by tasks from one user using the entire 2012 AOD data as input (valid use case? - there was some discussion on this point), which is only on CERN tape. Input data were staged from tape and put on scratchdisk
  - Some DPD tasks failing because AODtoDAOD got a SIGABRT signal (exit code 134)
  - Some tasks failing at BOINC
  - Occasional issues with CVMFS Stratum 1 and PanDA servers
  - ADCoS senior shifter coverage ok but no ADC experts this week
CA Cloud Squad Actions
- User DaTRI requests approved / processed (last week)
Open Issues and Ongoing Discussions
- Availability & Reliability Report [20151001]
  - September 2015 - All Sites Report ATLAS
- Sites should be providing the 2015 storage pledge.
  - REBUS Site Capacities
  - REBUS Federation Resource Pledges
- ATLAS Canada HammerCloud Tests
WLCG and General Computing News
- Security News
- gLite/EMI News
ROC_Canada News
- Current shift