Tier1 and Tier-2 Weekly Report 2017-02-14

General News

• ADC Tech Coord Meeting [20170213]
  • Distributed Production and Analysis Overview (ex-GDP)
    • Team, Meetings, shifts, operations, monitoring
• ADC Weekly Meeting [20170214]
  • MC Production
    • Down to about 290M events in production queue
    • Present fairshare for MC prod is 75%
      • Two global fairshares: MC16 ~1%, MC default ~ 73%
      • Will push these to 50/50 after monitoring
    • Start MC16 sim tasks asap. Optimized for a month's production ~ 500M events
    • Expect to start MC16 digi+reco end of the month
  • Group Derivation Production
    • Both data15+16 nearly finished, apart from few stragglers
    • debug streams also complete
    • Build 20.7.8.12 cache yesterday
    • Still working on DAOD size reduction survey
  • Reprocessing
    • Planning being discussed - want about 75k slots for ~2 months for 2016 data and ~1 month for 2015 data. Been delayed by 6+ months.
  • CRC/ADCoS Report
    • Got up to 300k slots with CERN P1 (50k cores) and Titan
    • Some tape tests going on with T1's last week
      • Stage 150TB from each T1 tape (300TB at BNL)
      • Ran into issue with timeouts set in FTS (48 hours) and defaults on sites
      • Full report next week
    • DDM functional tests have not been running since 18 Jan due to failure to upload source files - not sure this is necessary?
    • FTS race condition resurfaced leading to lost files
    • Overflow jobs have been stopped because new pilot mover requires data to be local
    • Jobs requiring 1000 tiny EVNT files:
      • Tasks are forced to nucleus to avoid data transfer
      • Still causes problems in NDGF since ARC CE has to copy them
      • EVNT merging is currently under test
    • CERN P1 and SuperMUCboth running event service stressed CERN S3 object store - turned out there was one bad node causing problems
    • Live page updates for shifters been improved and CC7 compatible
    • No AFS day TOMORROW
      • Reminder: CERN AFS will be blocked outside CERN for 24h starting 9am tomorrow
      • Some SW releases are known to try and access libraries on afsfirst before cvmfs
      • If CERN afs is mounted it is likely those jobs will hang
      • Recommendation: unmount CERN AFS if you can
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2017-02-14%

Comments