TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20090901 (2009-09-01, AsokaDeSilva) EditAttach

Tier1 and Tier-2 Weekly Report 2009-09-01

General News

  • REMINDER #1: reported SRM subsystem vulnerability in dCache [2009-07-27]
    • see dcache user-forum archives or forwarded email from Reda
    • all production versions affected and require advertised patches [2009-08-04]
      • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
    • I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions
      • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
      • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7

  • REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Data Subscription Requests

  • This past week: None

Site Reports

Alberta

Recent Issues

  • stale movers associated with dcap doors briefly (~ 2 hours) following reboot of NFS servers on WestGrid-UofA hosting 2 x 2 pools on current dCache Alberta setup [20090819]
    • restart of pool services corrected the problem

  • issues with connectivity to TRIUMF via end-to-end lightpath from Alberta (and other sites) [20090824]
    • network outage affecting Toronto and Alberta lightpaths to TRIUMF for approx. 2 minutes
    • routing issue with the BCnet primary DNS server was the cause

  • installed lcg-CE on SL48 node of WestGrid-Alberta cluster [2009-08-14]
    • successful setup of site-BDII and submissions of jobs via gatekeeper to WNs (including ATLAS Canada VOMS roles)
    • job submissions from CE to WNs successful (stderr/stdout delivery) following tested fix to pbs_mom config (thanks Sergey for suggested fix)
      • pbs_mom config being applied to remaining WNs by WestGrid-UofA admins
    • baseline dCache admin and pool services installation performed on three nodes (admin + 2 pools) [20090824]
      • clean 1.9.1-11 installation with chimera ; in preparation for update to 1.9.3-3 on system before further validation
    • APEL-MON services installation targetted for this week (Wed/Thurs)

Outstanding Issues

  • few minor issues pending correction on existing T2 facility:
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • required production releases installed on bugaboo (SL5 WNs) and tags published [20090824]
    • missing x86_64 and i386 dependencies for root to be corrected

  • unscheduled downtime announced for 20090826 to 20090831 for various upgrades (lustre, kernel, torque/moab) [20090820]

  • analysis errors observed in regards to missing files (see Outstanding Issues entry below)

  • apel publishing [20090825]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)

Outstanding Issues

  • savannah ticket for approximately 20k missing files at SFU dCache that are still registered in DDM [2009-08-14]
    • https://savannah.cern.ch/bugs/?54369
    • clean-up of LFC done by DDM Ops and completed [20090825]
      • files being transferred again to MCDISK for the whole dataset that was deleted since significant fraction was lost

  • recurrent problems with missing files expected to exist on SE but unavailable during analysis and production jobs [2009-07-22]
    • see previous item

Victoria

Recent Issues

  • unscheduled downtime for approx. 4 hours [20090821]
    • unscheduled power outage to the University brought down site services and nodes

  • analysis job failures with timeouts in file copy from SE (27) [20090819]
    • network bandwidth to SE (which nodes?) saturated during HammerCloud
    • maximum job slots for atlas pilots user (atlaspt) misconfigured to accept too high (100) number of concurrently running jobs ; reduced to 40

Outstanding Issues

Toronto

Recent Issues

  • SE issues [20090819]
    • failing SiteTests, production and analysis jobs, HammerCloud tests observed [20090819]
    • restart of srm services corrected the problem and was confirmed with successful SiteTests run by site admin [20090819]
    • GGUS ticket: https://gus.fzk.de/ws/ticket_info.php?ticket=51019 [solved]

  • migration of contents from faulty dCache pool (approx 800 GB) to other pools [20090824]

  • issues with connectivity to TRIUMF via end-to-end lightpath from Toronto (and other sites) [20090824]
    • network outage affecting Toronto and Alberta lightpaths to TRIUMF for approx. 2 minutes
    • routing issue with the BCnet primary DNS server was the cause

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

  • attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
    • pilots (without payload) successfully running
    • next target is successful SAM tests
    • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

  • brief 1 hour TRIUMF scheduled downtime for oracle cpu and linux package update [20090820]

Outstanding Issues

  • scheduled downtime for dCache patch application [2009-07-30]
    • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 17104 169 239934093 1169241 99% 99.5%
TORONTO-LCG2 2377 31 99563900 471118 98.7% 99.5%
VICTORIA-LCG2 1132 8 41746679 207984 99.3% 99.5%
ALBERTA-LCG2 943 14 20770687 65632 98.5% 99.7%
SFU-LCG2 21 0 757448 0 100% 100%
total 21577 222 402772807 1913975 99% 99.5%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

TRIUMF:
name,rank,value

TRF_SVRINIT ,1,100
TRF_SEGFAULT ,2,41
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,13
EXEPANDA_JOBKILL_SIGTERM,4,10
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,5,3
EXEPANDA_JOBKILL_BYPILOT,6,2


Alberta:

name,rank,value
TRF_SVRINIT ,1,14


SFU:

Toronto:

name,rank,value
TRF_SVRINIT ,1,26
EXEPANDA_JOBKILL_BYPILOT,2,3
TRF_SEGFAULT ,3,1
EXEPANDA_DQ2_STAGEIN,4,1


Victoria:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,6
EXEPANDA_DQ2PUT_FILECOPYERROR,2,1
EXEPANDA_JOBKILL_BYPILOT,3,1

Notes

SFU offline for UPS upgrade (27-31 Aug) but still offline today (1 Sep).

Toronto online on Aug 28.

-- BryanCaron - 01 Sep 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 92.7 K 2009-09-01 - 16:24 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 45.3 K 2009-09-01 - 16:24 AsokaDeSilva  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2009-09-01 - AsokaDeSilva
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback