Tier1 and Tier-2 Weekly Report 2009-07-21

General News

* STEP09 Site Follow-up Homework

• Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14]
  • ATLAS CA STEP09 Site Reports: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Step09SiteReports
  • STEP09 Summary Results http://gangarobot.cern.ch/st/step09summary.html

• APEL reporting for June looks good (Victoria still missing) - thanks to the sites for checking the status of your APEL parsing and publishing

• upcoming request this week to sites re: new accounts for pilot and poweruser0 roles within /atlas/ca group
  • the handling of new pilots associated with the new roles of 'pilot' and 'poweruser0' have been tested at TRIUMF and now at ALBERTA
    • sites will be requested to add a few needed local user pool-style accounts to accommodate these roles
    • See instructions for changes to site CE and SE to support the new roles

• REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena
  • Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
    • So far only ALBERTA and TORONTO with some reporting of SiteTests results (see for templates for report summaries)

• REMINDER #3: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090721.png:
  

• ops-reliability-t2-20090721.png:
  

ATLAS VO

• atlas-availability-t2-20090721.png:
  

• atlas-reliability-t2-20090721.png:
  

APEL Accounting

APEL Publishing Status

• apel-status-20090721.png:
  

APEL Accounting Results

• apel-statistics-20090721.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week:

Site Reports

Alberta

Recent Issues

• updated CE and SE so as to accommodate new poweruser0 and pilots roles (new pool accounts required).
  • for SE added dcache.kpwd as fallback authorization mechanism to the primary based on VO-roles.
    • required adding several packages and a missing grid-mapfile2dcache-kpwd script.

Outstanding Issues

• installation of grid services on new Westgrid-UA T2 site to be done

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

Outstanding Issues

Victoria

Recent Issues

Outstanding Issues

Toronto

Recent Issues

Outstanding Issues

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	12670	108	347979729	2263369	99.2%	99.4%
SFU-LCG2	2476	140	82099612	1350429	94.6%	98.4%
TORONTO-LCG2	2325	30	194867991	3392904	98.7%	98.3%
VICTORIA-LCG2	1268	191	83366515	2859419	86.9%	96.7%
ALBERTA-LCG2	853	120	32530856	1692948	87.7%	95.1%
total	19592	589	740844703	11559069	97.1%	98.5%

DashboardSummary.pdf

Failed Jobs

(from Panda Monitor)

FailedJobs.pdf

(from dashboard)

TRIUMF:

name,rank,value
TRF_EXC ,1,47
TRF_OUTFILE ,2,21
EXEPANDA_JOBKILL_BYPILOT,3,11
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,4,5
EXEPANDA_RUNJOBEXCEPTION,5,5
EXEPANDA_JOBKILL_NOLOCALSPACE,6,4
EXEPANDA_JOBEXPIRED_SIXDAYS,7,4
TRF_ARG ,8,4
EXEPANDA_JOBDISPATCHER_HEARTBEAT,9,3
EXEPANDA_JOBKILL_SIGTERM,10,2
TRF_SEGFAULT ,11,2

name,rank,value
EXEPANDA_GET_REPLICANOTFOUND,1,54
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,39
EXEPANDA_GET_NOSUCHFILE,3,12
EXEPANDA_JOBKILL_BYEXTIF,4,9
TRF_EXC ,5,4
TRF_UNKNOWN ,6,1
EXEPANDA_JOBKILL_BYSERVER,7,1

name,rank,value
TRF_OUTFILE ,1,55
EXEPANDA_JOBKILL_SIGTERM,2,44
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,38
EXEPANDA_DQ2_STAGEIN,4,2
EXEPANDA_PUT_LFCMODULEIMPORT,5,1

name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,16
TRF_SEGVIO ,2,6
EXEPANDA_DQ2GET_INFILE,3,4
TRF_UNKNOWN ,4,2
TRF_OUTFILE ,5,1
TRF_SEGFAULT ,6,1

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,153
TRF_OUTFILE ,2,10
EXEPANDA_JOBKILL_BYPILOT,3,10
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,8
EXEPANDA_DQ2GET_INFILE,5,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,3
EXEPANDA_JOBKILL_SIGTERM,7,1

Notes

-- BryanCaron - 21 Jul 2009