Tier1 and Tier-2 Weekly Report 2017-01-31

General News

• ADC Tech Coord Meeting [20170130]
  • Storage Space Reporting with SRM (JSON)
  • Some issues with current task brokering as based on absolute distance from goal rather than relative
    • validation tasks not running as prod tasks always get ahead
• ADC Weekly Meeting [20170131]
  • MC Production
    • Getting back to about 150k slots this week for MC
    • Still have 650M events in prod queue, with about 40 pending requests
    • Want some more insight whether reattempts for jobs with Athena failures makes sense (e.g. segmentation faults)
    • Discussion about Throttled tasks and philosophy and cleaning up tails of tasks
  • Group Derivation Production
    • Almost all data15/16 tasks done
    • Debug stream data15 70% complete and data16 97% complete
    • Working on DAOD size reduction via surveying physics groups
  • Reprocessing
  • Pilot v67.5
    • Added job status check after job download (jobs could already be running on Nordugrid due to a batch system bug)
    • remove hard-coded PanDA URLs
    • Few updates for Memory monitoring, BOINC client, time-outs, mover updates
    • Expect to release next couple of days
  • CRC/ADCoS Report
    • Still limit overlay tasks 2-3k to avoid stressing frontier servers - were some restarts needed last week
    • Spike in data rebalancing on Friday and Monday due to some site decommissioning - deleted untouched ESD/DAOD older than 6 months in process
    • Sites alerted with issue with CC7 SSL ciphers
    • Contacting users with >500TB or >500Msec/day usage (1-2 a week)
    • Occassional issue with sites with persistent transfer problems
      • DAST suggestion to have Hammercloud like test to blacklist sites with persistent transfer problems
      • Sites should declare an unscheduled downtime, but if not done, DAST can ask ask DDM to blacklist the site by hand
      • ADC will work on a solution in the meantime
    • Number of sites with squid issues this week
  • Update on SAM Site Monitoring
    • Plan to use AGIS flag to select default queue for SAM tests - waiting for this to be implemented
    • Test all DDM protocols
    • Need AGIS flag to know which is used for ATLAS_CRITICAL
    • Also look at creating metrics from delivred cpu vs pledge and job efficiency
  • PanDA Controller Migration
    • Still using PINKPANDA to change status of Panda queue and resources, usually via curl
    • Want to do this via AGIS and populate to schedconfig etc
    • Plan - turn off old tool next Monday
    • Encourage people to use AGIS centralized blacklisting:
      • http://atlas-agis.cern.ch/agis/pandablacklisting/list/
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20160216]
    • January 2016 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ATLAS Site Availability for last two weeks - CA Cloud
• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- %USERSIG{LeslieGroer - 2017-01-31%

Comments