Tier1 and Tier-2 Weekly Report 2017-01-24

General News

• ADC Weekly Meeting [20170124]
  • MC Production
    • Current usage dominated by derivations. About 70k cores for MC
    • Still have 740M events in production queue, 30 pending requests
    • Many evgen tasks are paused
  • Group Derivation Production
    • Built a number of caches
      • 20.7.8.8: full processing for STDM7
      • 20.7.8.9: full data processing for HIGG2D5
      • 20.7.8.10 built today: full data processing for FTAG1
      • 21.0.11.1 built today: first test derivations for rel 21
    • Launched production of data16 debug stream for EXOT/SUSY
      • Will launch data15 debug stream soon
    • December processing (data15/16) still running
      • Working on exhausted tasks
      • About 50% (2300 tasks) of the data processing for formats with normal priority still remain after 4.5 weeks
      • Not a lot of MC derivation in system
    • Started serious work of size reduction of the DAODs
  • Reprocessing
  • CRC/ADCoS Report
    • Weird monitoring issue on Thursday last week were all jobs in system counted as running
    • Frontier servers limits increased at CERN on Monday but then restored original values
      • Two servers now with IPv6 preferred
      • Not able to run with maximum number of threads while trying to contact frontier.cern.ch over IPv6
      • Clients fallback to IN2P3, which got overloaded and needed restarts
    • Missing input file for HC caused some site blacklistings on Thursday
      • Were trying to cleanup old rel 17 PFTs but issue so older datasets restored
    • Staging tests TAPE to DISK finished last week at IN2P3 and FZK
    • Deletion failures last week due to a variety of issues at sites
    • Sites with dCache < 2.13.51, which was released last week, will continue to have issues due to old JGlobus library with bug that Di, Asoka and others have been exploring at TRIUMF in https://ggus.eu/index.php?mode=ticket_info&ticket_id=124650.
      • Looks like certificates issued by Canadian and Russian CA's are hit by this.
      • Similar issue for BestMan
    • Apache mod_ssl update broke ADC livepage at CERN
    • Changed limit of 10k jobs to 5k per user due to excessive usage of resources
      • Mostly AOD files for analysis and large scale MC production
      • Will be monitoring usage
    • INFN-T1 was offline for production due to jobs stuck reading /afs on WN's (in rel 20.7)
    • Sites identified with HTTPS and CC7 incompatible SSL ciphers
    • dq2 clients removed from setupATLAS
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2017-01-24%

Comments