Tier1 and Tier-2 Weekly Report 2016-12-06

General News

• ADC Weekly Meeting [20161206]
  • MC Production
    • Running about 150k slots most days
    • 330M in production queue, about 10 requests in approval step
    • MC16 preproduction wil start before break. Expect 260M events over Xmas
    • For Xmas break, stop task submission Dec 21 and resume Jan 5. Hope to have 600M in queue by then?
  • Group Derivation Production
    • Validation with 20.7.8.5 tasks submitted
    • Cleaning up tails of long running derivations jobs
    • Should have new obsolescence list for cleanup before break
  • Reprocessing
    • Running full reprocessing workflow in 21.0.11, one run from 2015, one from 2016
    • Small processing of 5 runs from DRAW_EGZ and DRAW_ZMUMU in 21.0.11 to check alignment
    • 250TB express stream ESD identified for deletion, waiting on decision with 750TB minBias ESD
    • Plans for break should be in place next week, after this week's DPC
      • Preliminary plan was to run some express stream, say 20 runs, AOD+HIST
      • Release 21 is very very late, but want to run something
  • CRC/ADCoS report
    • No major problems from week
    • Some issues with fairshares to allow derivation and upgrade tasks to complete rapidly, even with global shares
      • Merging of derivation tasks is done on 1 core and takes time
      • Some jobs of urgent upgrade task need high memory => only few sites available to run such tasks
      • Lack of monitoring to evaluate total of jobs/event still to be processed
    • Fair amount of user subscriptions resulting in a lot of transfers
    • Some issues with CVMFS stratum 1 and CONDB being behind on latest release being discussed
    • One problematic Powheg task reappeared where should use single core but spawns over all cores
    • Wanting to clean up open GGUS tickets before break (~40)
    • Issues with data transfers to CA-MCGILL-CLUMEQ-T2 mentioned
  • Site Monitoring
    • Followup on sites with low availability due to monitoring issues
    • Discussed queue selection for SAM tests (currently random)
    • From previous meeting, select pq_is_default=1&pq_capability=score
      • Implemented but ~10% of CE's don't have this defined
      • Suggest adding new flag etf_default and use this if true or pick randomly otherwise
    • Some discussion about VOfeed
  • Pilot Update
    • Hotfix release of v67.2 today (wrong event type sent with trace)
    • v67.3 release this week once finish testing
      • Relax checksum verification in new lsm mover
      • Added https protocol to list of supported schemes in new GFAL mover
      • Changes for nightlies
      • Add teskID to tracing report * CRC/ADCoS report
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2016-12-06%

Comments