Tier1 and Tier-2 Weekly Report 2011-07-05

General News

• ADC and ATLAS Computing News [20110704]
  • ADC Weekly meeting
  • WLCG-DESY meeting going on next week (July 11-13) https://indico.desy.de/conferenceTimeTable.py?confId=4019#20110711
  • Try to increase PD2P placement of data at Tier-2's this summer (ovecome the dearth of data at T2's)
    • Several ideas (extra copy using fairshare whenever a T2 copy occurs; or when a T1 copy occurs; make 2 preplaced fairshare copies at T2 of all group produced NTUPLEs; use DQ2 "closeness" along with brokerage to improve subscription performance)
    • Approved by CREM last week and implement in a few days. Evaluate the results in 2 months
    • Expect a lot more copies - at least 2-4 times more data at T2's
  • Squid Cache for Generator job options - prevent flooding of CERN services (DOS) getting MCopts.tar.gz
    • Detect site squid cache(s) (e.g. frontier caches). Define ATLAS_PROXIES, but respect http_proxy settings in job transform
    • May be security settings issues if securing squid caches based on destination rather than source
    • Check this with

   $ export http_proxy=my.cache:3128
   $ wget http://atlas-computing.web.cern.ch/atlas-computing/links/kitsDirectory/EvgenJobOpts/MC11JobOpts-00-00-19.tar.gz

• • • Implement in pilots eventually?
  • Couple of incidents of users issuing huge numbers of dq2-get requests (47k in one day from a big T1)
    • For now sites should report these "abuses" to AMOD if this is seen and they will follow up with user (contact, then blacklist if necessary)
    • Longer term - implement some limitation on dq2_get.
    • Some recipe needed for T3's without grid storage (no FTS) - encouraged to provide srm storage and be included in ToA.
  • Some issues with FTS channels being stuck caused some sites to not get the latest DBRelease file and ANA queues were set brokeroff at 4 T1's and some T2's
    • Implement new policy for "DBReleases latest" where if more than 90% of sites and all online T1's have the new DBRelease then this new version will be used by all analysis jobs
      • Shifters need to be able to track which release is meant to the "latest"
    • Release 17 in validation (17.0.1.2) and deriving conditions data for runs up to the technical stop
      • Build 17.0.2.1 this week. Deadline for conditions data July 11.
      • Start processing initial express and CosmicCalo streams July 14? - derive final conditions
      • Release 17 on Tier-0 ~ August 1. Bulk production about 2 weeks later.
        • Some experts away on vacation so schedule is optimistic!
    • Florentin Bufor (Wisconsin) taking over SLS monitoring for Squid/Frontier
      • Create SSB views
      • Some errors for TRIUMF and KIT being fixed now

• CA Cloud Squad Actions [20110628]
  • DaTRI requests approved / processed (last week)

NOTE: Lot's of failures for subscriptions to user.cloh.RPVLL2011

• • Declared Downtimes
    • CA-ALBERTA-WESTGRID-T2/SCHEDULED downtime (OUTAGE) [06-07-2011 15:00 to 06-07-2011 23:00 UTC ] - dCache Upgrade from 1.9.3 to 1.9.5
  • ATLAS Site Exclusion Savannah entries [20110621]
    • HammerCloud autoexclusions
      • None
  • other cloud squad related savannah and / or ROC ggus tickets [20110705]
    • !PD2P (dynamic data placement) in CA cloud [20110607]
    • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)

• Open Issues and Ongoing Discussions
  • Sites should be providing the April 2011 storage pledge. [see 20110315 report.
    • http://gstat-wlcg.cern.ch/apps/capacities/sites/
  • CREAM CE deployment preparations
    • TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20110426]
  • HammerCloud [20110531]
    • [20110531] Ongoing tests with Victoria https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CA_Cloud_Apr2011#Additional_Victoria_WG1_testing
    • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • https://twiki.atlas-canada.ca/bin/view/AtlasCanada/TRIUMF_Aug2010
      • follow-up required to determine which modifications can be added to Panda schedconfig so as to enable the more efficient FileStager method of data input
    • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • TWiki Link: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CACloudOct2010

• Security News [20110621]
  • [20110524] UVic and Toronto participated in SSC5 (23 May - 03 June 2011) - final reports to be sent to SSC5
  • [20110616] EGI CSIRT ADVISORY [EGI-ADV-20110615-02] Vulnerability for LRMS (details to be posted later)

• gLite News [20110705]
  • [20110705] [https://operations-portal.egi.eu/broadcast/archive/id/420][EGI Trust Anchor release 1.40-1] is now final - please install.
  • [20110504] [https://operations-portal.egi.eu/broadcast/archive/id/378 ][gLite 3.2 WN release] has an important known issue detailed here: http://glite.cern.ch/glite-WN/known_issues
  • [20110627] New IGTF distribution (certificates) and version 3 of the CRL retrieval tool Fetch-crl https://dist.eugridpma.info/distribution/util/fetch-crl3/. Version 1.39 was updated to 1.40 on 2011-06-28.
  • [20110628] [https://operations-portal.egi.eu/broadcast/archive/id/410][gLite 3.2 update 30 release] The highlights of the update are:
    • New version of glite-BDII_top
    • New version of glite-CREAM (to support changes for SGE)
    • New version of glite-LB
    • New version of glite-SGE_utils

• ROC_Canada News [20110621]
  • Current shift - Erming Pei July 4-8
  • Next Week - Leslie Groer

• Squid Deployment [20110329]
  • CA cloud Frontier and Squid deployment information (versions and network information)
    • Frontier-Squid Monitoring http://frontier.cern.ch/squidstats/indexatlas.html
  • SAM tests: (CA Link), (Aus Link):

• General Computing News

• SONAR Tests [20110426]
  • Performance monitored at http://bourricot.cern.ch/dq2/ftsmon/sonar_view/cached/ with historical views at http://bourricot.cern.ch/dq2/ftsmon/ . Canadian cloud now tested on Fridays. All CA Tier-2's are tested with small (20MB) and medium (200MB) files, but only TRIUMF, CA-SCINET-T2 and CA-VICTORIA-WESTGRID-T2 are currently tested with large (2GB) files outside of the cloud.

SAM Results

• site service status via Gridview - Please check your site

• June 29-July 5, 2011

• July 2011 Detailed Site Information (valid until 20110731):
  • TRIUMF-LCG2
    • OPS (this month)
    • ATLAS (this month)
  • Australia-ATLAS
    • OPS (this month)
    • ATLAS (this month)
  • CA-ALBERTA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-SCINET-T2
    • OPS (this month)
    • ATLAS (this month)
  • CA-VICTORIA-WESTGRID-T2
    • OPS (this month)
    • ATLAS (this month)
  • SFU-LCG2
    • OPS (this month)
    • ATLAS (this month)

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

• • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

• http://goc-accounting.grid-support.ac.uk/rss/TRIUMF-LCG2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/Australia-ATLAS_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-ALBERTA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-VICTORIA-WESTGRID-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/CA-SCINET-T2_Sync.html
• http://goc-accounting.grid-support.ac.uk/rss/SFU-LCG2_Sync.html

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

McGill

Recent Issues

Outstanding Issues

TRIUMF

Recent Issues

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Production

Summary

• NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

• Within the CA Cloud and Worldwide (Panglia View):

• Alternate Views (Source link for last 7 days):
• http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/summary?cloud=TRIUMF&grouping=site&grouping=cloud&end-date=2011-05-24%2020:59:59&start-date=2011-05-31%2020:00:00

World-wide weekly ATLAS ADCoS summary (Jun.28-Jul.4,2011)
________________________________________________________________________

I. General summary:
   ---------------

A)  During the past week (Jun.28-Jul.4,2011) Panda production service
   - completed 1,554,865 managed group, MC production, validation
     and reprocessing jobs
   - average ~221,124 jobs per day
   - failed 88,607 jobs
   - average efficiency:
      -- jobs     ~94.6%
   - active tasks from Panda-monitor: 1,476
      -- distribution by cloud:
CA:33 CERN:77 DE:186 ES:41 FR:133 IT:25 ND:28 NL:168 TW:11 UK:164 US:610 

B) - CERN/CERN-PROD downtime (SE,SRM OUTAGE:CASTORATLAS),Jun.29,Jul.4. 

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
    -------------------------------------------------

3) Thu Jun.30 
+ CERN/DBRelease.v150700 subscriptions got stuck since Jun.21.Elog 26932,
  experts informed.Seems fixed. 

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
    (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+ 15.6.14.3 mc10_7TeV evgen task 434243 failures: ATH_FAILURE - Athena
 non-zero exit.Error: Can't find tan beta in BLOCK MINPAR or BLOCK EXTPAR.
 Savannah 83699,83769.MC10 production-SUSY.All jobs failed 9 attempts each
 at different sites in CA cloud.The same problem with t.434191,434215 in
 NL cloud.Tasks are in "failed"/ABORTED state.Config issue.

</verbatim>

• Stats:
  • See Key for what the headers mean