Tier1 and Tier-2 Weekly Report 2016-11-01

General News

• ADC Technical Coordination Meeting [20161031]
  • HammerCloud infrastructure/testing upgrade
  • BigPanda monitoring updates
  • Pilot Mover migration status
• ADC Weekly Meeting [20161101]
  • MC Production
    • Steady increase in MC jobs over last week - competition from data-reprocessing?
    • ~400M events in production queue and several requests (>100M) in approval step
    • pre-MC16 production has been started but only with simulation
    • Wil need r-tag and mu-profile defined (after Charmonix workshop?)
    • Very large evgen log files still an issue - moving to tape for now with no reduction. CREM still wants reduction in size
  • Group Derivation Production
    • 20.7.8.1 built over weekend
      • Only for STDM4 MC
    • 20.7.8.2 will be for general use
    • Longer discussion about problem that production system (pilot) is sometimes silently skipping input files
      • Still investigating how pervasive this is and what datasets affected
      • Tough to identify as skimming so can't rely on event counts
      • Not all log files available to check for issues
      • https://its.cern.ch/jira/browse/ATLASPANDA-310
  • Reprocessing
    • Bunch of special processing runs completed.
    • Special 2015/16 ALFA runs in preparation (rel 20.7.8.6)
    • Release 21 validation
      • Waiting for 21.0.9, one crash fixed since 21.0.8(21.0.7)
      • Some jobs crashing jobs again with WARNING FileSize > domMaxFileSize for tmp.ESD
        • Use postExec for now 'svcMgr.AthenaPoolCnvSvc.MaxFileSizes = [ "15000000000" ]' in 21.0.9 (15GB)
      • Other crashes with "StreamESD FATAL commitOutput failed" looks like hitting memory limit - need to turn this retry killer off, will contact ADC
  • CRC/ADCoS report
    • Reduced site capacity for 1-2 days because of kernel upgrades
    • Frontier overload when processing sparse data mitigated (but not solved) by preventing reads of unused DCS data into job memory
      • Found in this process that all conditions data are read in (Ncore+1) times in athenaMP jobs
      • For sequential processing (normal RAW data files) the IoV is the same for all events, so the squid cache works and we never noticed
      • For sparse (selected) RAW data files, there are many IoVs and correspondingly more conditions data read in
    • Data transfers dominated by production inputs
    • Ongoing discussions:
      • Reduction of job input transfers
      • Save unused data in group space to tape and free disk space
      • Time-out for large files in FTS is too short, especially as the number of parallel transfers was increased
      • Never-ending discussion on optimal brokering for high and low priority tasks
  • CRC/ADCoS report
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2016-11-01%

Comments