TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100309 (2010-03-09, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-03-09

General News

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100309]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2 - upgrade to SL5 of WNs scheduled for weekend of March 21 (to follow scheduled power maintenance)
      • SFU-LCG2 (snowpatch) - current downtime
      • VICTORIA-LCG2

  • CA Cloud Squad Actions [20100309]
    • marked several CA Cloud sites offline in Panda during 2 hour period of TRIUMF LFC migration from SL4 to SL5 [20100208]
      • restored to online at the end of the successful migration
    • will shortly be confirming to DDM default policy of no ESD distribution to CA-based T2s in CA cloud will be applied
      • awaiting confirmation from Melbourne before submitting overall cloud response
    • Ongoing Discussion:
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Actions to Do:
      • request update to ToA to include local squid cache installation details for CA-SCINET-T2
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • new update released for SiteTests from Asoka [20100304]
    • SL5 sites should login via lxplus5 to run SiteTests from now on

  • Squid deployment [20100309]
    • brief problems last week reported [200100303]
      • SAM Frontier tests for all CA sites failing with TRIUMF Frontier server; fixed that day at the T1 (cause?)
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100309]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: WARN (local squid server online but not tested in SAM yet)
      • ACTION ITEM: cloud contacts to request ToA update for site info to reference local Squid cache as primary now that the local client tests passes
      • ACTION ITEM: ALBERTA to confirm configuration of Squid ACLs to accept backup connections from CA-SCINET-T2
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid which is OK)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

  • General Information: WLCG office will hold off publishing T1 and T2 accounting info for another 1 to 2 weeks [20100308]
    • Sites please confirm their historical APEL data that has been published looks correct
    • CA-ALBERTA-WESTGRID-T2 and CA-SCINET-T2 please work to complete APEL publishing this week

APEL Publishing Status

  • apel-status-aus-20100309.png:
    apel-status-aus-20100309.png

  • apel-status-ca-20100309.png:
    apel-status-ca-20100309.png

APEL Accounting Results

  • apel-statistics-aus-20100309.png:
    apel-statistics-aus-20100309.png

  • apel-statistics-ca-20100309.png:
    apel-statistics-ca-20100309.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100315
      • to be extended to 20100405

  • CA-ALBERTA-WESTGRID-T2 site status [20100302]
    • unscheduled downtime registered during period of TRIUMF LFC migration [20100308]
    • scheduled downtime booked into GOCDB for main campus computing centre power upgrades [starting 20100319]
      • https://goc.gridops.org/downtime?id=70455462
      • network cabling maintenance planned for lighpath connection during this time
      • WN migration to SL5 will also be performed
        • additionally cluster will be migrated from SL4 to SL5 following the power upgrade completion on 20100321
        • verified test configuration of SL54 x86_64 with ATLAS kits; few nodes to be upgraded prior to Mar 21 shutdown with majority of nodes on Mar 22
        • additional CE will not be created but rather lcg-CE + BDII will be updated with SL5 WN information
      • torque qsub wrapper activated 20100305 to remove email notification errors associated with grid jobs coming from CE to torque/moab
      • default walltime and cputime for queues reduced to 36 hours from 72 hours based upon the historical job information since January 2010

    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • ACTION ITEM: To confirm ACL setup on site squid (once restored) to accept backup connections from CA-SCINET-T2

Outstanding Issues

  • APEL accounting information publishing (see above ticket) is the highest priority now
    • APEL server updated to accept site information; now debugging local mysql access errors for APEL service
  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • Confirmed ACL setup on site squid to accept backup connections from CA-SCINET-T2 [20100302]

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • CA-SCINET-T2
    • admin network switch problems [20100308]
      • rebooted 20100309 morning and appears to have successfully recovered
      • jobs during overnight period died

    • pilot job errors [20100305]
      • error loading shared libraries libvomsapi.so.0 no such file or directory is reported but underlying python path problems
      • problem resulted from default environment change on Scinet compute nodes which overwrote some path elements needed by gLite
        • configuration of legacy version of libgfortran now excluded from HEP image provisioning and corrupt images have been flushed

    • installed local Squid server and validated through the usual client tests after correcting the reference TRIUMF Frontier URL [20100304]
      • ACTION ITEM: cloud contact to request ToA update for CA-SCINET-T2 entry so as to reference the local squid cache

    • CA-SCINET-T2 not included in dashboard (production) and ATLAS SRMv2 SAM tests

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

  • correction to T1 Frontier setup to restore successful SAM Frontier functional tests from CA sites [20100304]

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • scheduled downtime during TRIUMF LFC migration to perform site torque server upgrade [20100309]
    • site Panda queues in test awaiting confirmation of job success

Outstanding Issues

  • Tim indicated that the dashboard discrepancies reported last week are a side effect of being part of the cream-ce testing [20100305]

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

Reference Link for the Last Week

  • Within the CA Cloud:
  • production-ca-20100309.png:
    production-ca-20100309.png

  • How we fared to other ATLAS sites during the past week:
  • production-all-20100310.png:
    production-all-20100310.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard:

TRIUMF name,rank,value
TRFERROR,1,21
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,17
EXEPANDA_JOBKILL_BYPILOT,3,17
EXEPANDA_DQ2_STAGEIN,4,10
TRF_UNKNOWN ,5,10
EXEPANDA_Release_MISSING,6,9
EXEPANDA_DQ2PUT_FILECOPYERROR,7,2
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,8,2
EXEPANDA_DQ2_FILEADD,9,1
TRF_SVRINIT ,10,1
EXEPANDA_DQ2GET_CANTCALL,11,1
EXEPANDA_PUT_GLOBUSSYSTEMERROR,12,1

Alberta name,rank,value
TRF_UNKNOWN ,1,8
EXEPANDA_DQ2PUT_FILECOPYERROR,2,6
EXEPANDA_JOBKILL_BYPILOT,3,5
EXEPANDA_DQ2_FILEADD,4,1
TRFERROR,5,1
EXEPANDA_JOBKILL_NOLOCALSPACE,6,1
TRF_OUTFILE ,7,1

SFU name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,5
EXEPANDA_DQ2PUT_FILECOPYERROR,2,2
TRF_UNKNOWN ,3,2
TRFERROR,4,1

Australia name,rank,value
TRFERROR,1,6
TRF_UNKNOWN ,2,5
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,4
EXEPANDA_JOBKILL_BYPILOT,4,3
EXEPANDA_DQ2PUT_FILECOPYERROR,5,2
EXEPANDA_DQ2_FILEADD,6,1
TRF_OUTFILE ,7,1

Victoria name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,1
EXEPANDA_JOBKILL_BYPILOT,2,1
EXEPANDA_DQ2GET_INFILE,3,1

You can find the meanings of the transform error codes on this link.

  • Yuri's notes:
    • active tasks from PandaDB: 99
    • distribution by cloud:
      • CA:16 CERN:1 DE:17 ES:2 FR:5 IT:6 ND:14 NL:3 TW:1 UK:6 US:28

    • Tue Mar 2: CA/TRIUMF<-TAIWAN FT f./GRIDFTP_ER/server er.500.GGUS 56137 verified: reduced streams number 20->10 for AGSC-TRIUMF,AUATLAS-TRIUMF and TRIUMF-AUATLAS channels on FTS server.This helped.

-- BryanCaron - 09 Mar 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100309.png r1 manage 868.4 K 2010-03-09 - 18:30 BryanCaron  
PNGpng apel-statistics-ca-20100309.png r1 manage 1115.6 K 2010-03-09 - 18:30 BryanCaron  
PNGpng apel-status-aus-20100309.png r1 manage 15.2 K 2010-03-09 - 18:31 BryanCaron  
PNGpng apel-status-ca-20100309.png r1 manage 22.3 K 2010-03-09 - 18:31 BryanCaron  
PNGpng production-all-20100310.png r1 manage 925.4 K 2010-03-09 - 19:56 BryanCaron  
PNGpng production-ca-20100309.png r1 manage 512.5 K 2010-03-09 - 19:47 BryanCaron  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2010-03-09 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback