Tier1 and Tier-2 Weekly Report 2013-10-08

General News

• ADC and ATLAS Computing News [20131008]
  • ADC Weekly meeting
  • Message to T1's not to change Panda Resources or PandaSiteIDs without conferring with ADC due to knock on effects in Cloudconfig: https://pandamon.cern.ch/taskBuffer?method=getCloudConfig
  • GDP
    • Small chance that WN can produce two output files with same GUID.
      • Not a big issue: the 2nd job is marked failed due to DQ2 registration issue, but the LFC catalogs need a manual cleanup
    • In some cases, more than one version of a file is registered in the datasets
    • Panda queue parameter mintime activated in job brokering on request by sites (FR T1)
      • Jobs which are shorter (maxCpuCount) than mintime are not sent to the panda queue
      • Helps the sites to optimize the resource usage and better backfill of short jobs
  • AMOD
    • BNL VOMS server had to be switched off because of the new CA (/DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=vo.racf.bnl.gov) which is now providing the host certificate, resulting in a different DN of the VOMS server.
      • This DN is hardcoded in a file for the EGI sites. Sites should update the file.
  • Monitoring PanDA usage of FAX
    • The first production use case for FAX is so called FAX failover.
    • In case pilot fails to get stage input files from 2 tries, it will try to get them from FAX.
    • To failover or not is controlled by a AGIS queue variable “failoverToFAX”: https://twiki.cern.ch/twiki/bin/view/AtlasComputing/FAXconfigAGIS#Enabling_FAX_failover
    • As it is always beneficial, we would suggest all the sites and all the queues to have it on.
      • Currently all US sites, large part of UK sites and a few continental EU site
      • There is no need to contribute to FAX in order to use it.
    • We need to verify that service runs and what it contributes so comes monitoring
      • Pilot for each job collects information on transfers and curl-ed back to PandaLogger
      • Not human friendly logging: http://pandamon.atlascloud.org/logsummary
      • Dedicated Panda Monitor pages from Valeri Fine
    • Other possible Use Cases for FAX
      • Overflow - when Panda decides that site is too busy and decides to send job elsewhere while reading from "busy" site via FAX. Should start in roughly one month
      • User directly gives FAX filenames in input dataset. Already possible using prun jobs.
        • Requires changes in pilot to separate use cases and collect information
    • Very long discussion about stress testing and how to get this into production without adverse effects

• New Panda Monitor pages

• CA Cloud Squad Actions [20120626]
  • User DaTRI requests approved / processed (last week)
  • Declared Downtimes

• • HammerCloud autoexclusions

• • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
• Open Issues and Ongoing Discussions
  • Sites should be providing the 2013 storage pledge.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/ http://gstat-wlcg.cern.ch/apps/pledges/resources
  • HammerCloud [20120214]
    • [20120214] Ongoing tests with McGill https://twiki.atlas-canada.ca/bin/view/AtlasCanada/McGill_Feb2012
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20120801]

• gLite/EMI News [20121127]
  • [20121029] UMD v.2.2.1 has been released
  • [20120924] Retirement calendar of gLite 3.1 and currently unsupported gLite 3.2 products
  • [20120911] EMI-2 WN & UI rpms (SL5/x86_64) beta testing version available

• ROC_Canada News [20130625]
  • Current shift - Erming Pei June 24-28, 2013
  • Next Week - Leslie Groer

• WLCG and General Computing News [20120911]

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

• site service status via Gridview - Please check your site

• April 2012 Detailed Site Information (valid until 20120430):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-MCGILL-CLUMEQ-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2013-02-01%2020:59:59&start-date=2013-02-07%2020:00:00

-- LeslieGroer - 2013-10-08