Tier1 and Tier-2 Weekly Report 2016-11-08

General News

• ADC Technical Coordination Meeting [20161101]
  • pyAMI 5.0.6 migration tests
• ADC Weekly Meeting [20161108]
  • MC Production
    • Running about 150k slots most of the week (mostly sim)
    • ~570M events in prod queue and several requests awaiting approval
    • Fair number of mc15c requeuest but lower priority than mc16a
    • Some discussion about possible playing with relative priorities as having trouble getting some sim tasks through if there are many reconstruction tasks ahead
  • Group Derivation Production
    • All derivations skimming on offline muons affected by TRT issue
      • Build 20.7.8.2 in next couple of days to address this
      • PC decided to rerun all data and MC for derivations used for Moriond and requiring multiple muons (~60% of all derivations)
      • Estimate 5-6 weeks to run
    • Long running derivation jobs due to too large files after switching to AthenaMP (up to 100GB inputs per job)
      • Suggest changing to 80GB (data) and 60GB (MC)
      • Modify the split levels and retries (needs DB access - would be useful to be able to set these parameters in the Web UI)
    • Jobs silently skipping input files investigated in ICHEP cache (20.7.6.4 through 20.7.7.9)
      • 234 tasks of 7.5k affected
      • TRF is now fixed and will abort the job if an input file is missing
      • Discussing whether to rerun derivations in current cache (lots of datasets get deleted in the process though)or wait until new cache
    • With AthenaMP validated will do test with fast merge (20.7.7.9)
  • Reprocessing
  • Analysis Task Duration
    • Investigating user complaints that despite average job walltime of 0.7 hours, average queue time is 2.5 hours but max > 300 hours (12 days)
    • Jedi uses weighting algorithm to broker jobs, mainly driven by number of jobs in queue with higher priority
      • But Jedi does not know how long the jobs ahead in the queue will actually take
      • If jobs are still in assigned or starting (new) state after 24 hours, Jedi kills them and creates new ones at a different site based on input availability, site status, etc
      • Can we develop better algorithm for rebrokering etc
      • 85% (95%) of user tasks complete within 1 (3) days, and a tail of 1% taking more than a week
      • 20% of jobs in last 3 months got reasigned by Jedi (and some mulitple times)
        • Reducing to 6 hours meant 15%/6%/2% of jobs reassigned in 6/12/18 hours
      • (D)AODs tend to be shorter but are in the same queue as longer jobs - can these be separated somehow (different fairshare)
      • At some sites >50% of jobs get reassigned. A few slow sites can affect a cloud (ND,US,ES,DE notably)
      • Allowing reassignment for jobs in starting state too in ND seemed to help
  • CRC/ADCoS report
    • Most shifts covered this week
    • Couple of CVMFS_Stratum1 and Frontier issues
    • Taiwan-LCG2 - 135k files lost in hardware failure
  • BigPanda Monitoring
    • working on SSO access
  • Update on SAM and ASAP
    • Couple of issues with US sites where not being monitored due to BDII or wrong flags set
    • Issue with CERN-PROD SRM-DEL premission error being followed up on
    • Tracking down causes of very low availability reports for some sites
  • Pilot News (66.5)
    • Various site mover related fixes
    • CA and UK clouds switched to new movers last week
    • Also done for ES,RU,TW and IT
    • DE and FR are in progress
    • Site experts are kindly asked to verify sitemovers configuration (copytools priorities, root protocols endpoints) in AGIS and report any issues or suspicious behaviour
  • CRC/ADCoS report
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2016-11-08%

Comments