Tier1 and Tier-2 Weekly Report 2009-08-11

General News

* draft APEL reports for WLCG for July released [2009-08-11]

       T2 CA East =  148% of MoU pledge
              TORONTO-LCG2 --->  132489 kSI2K-Hrs
       T2 CA West = 140% of MoU pledge
              ALBERTA-LCG2 --->  62509 kSI2K-Hrs
                  SFU-LCG2 --->  23373 kSI2K-Hrs
             VICTORIA-LCG2 --->   101555 kSI2K-Hrs

* APEL reports for WLCG updated (Version 2) for June to include missing statistics [re-published 2009-08-11]

       T2 CA East =  140% of MoU pledge (old was 105%)
              TORONTO-LCG2 --->  120549 kSI2K-Hrs
       T2 CA West = 97% of MoU pledge (previous was 90%; original was 74%)
              ALBERTA-LCG2 --->  44244 kSI2K-Hrs
                  SFU-LCG2 --->  15593 kSI2K-Hrs
             VICTORIA-LCG2 --->   65399 kSI2K-Hrs

* ADC Operations News [2009-08-06] (notes)

• ongoing discussion re: transition from SL(C)4 to SL(C)5
  • WLCG sites will be requested to upgrade WNs to SL(C)5 by WLCG MB once confirmation received from ATLAS management
    • so still some fact-finding and details being worked out in atlas
• ATLAS Tier-1/2/3 Jamboree @ CERN Tuesday October 13, 2009 Agenda Link
  • purpose is to discuss planning, setup and operations in advance of the LHC restart
  • meeting will be available via EVO

* reported SRM subsystem vulnerability in dCache [2009-07-27]

• see dcache user-forum archives or forwarded email from Reda
• all production versions affected and require advertised patches [2009-08-04]
  • 1.9.1-11, 1.9.2-10, 1.9.3-3, 1.9.4-2
• I would like to see us take a detailed inventory of our current and planned site dCache configurations (hardware and service distribution) and deployed release versions

* REMINDER #1: STEP09 Site Follow-up Homework [since 2009-07-14]

• Sites please look at your individual reports which have been updated to include "Site Tasks for Follow-up" [added 2009-07-14] and report on progress during meeting
  • Alberta Homework
  • SFU Homework
  • Toronto Homework
  • Victoria Homework

* REMINDER #2: SiteTests package should be installed by all site admins and exercised regularly (daily) to confirm site functions for ganga and pathena

• Results Page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/SiteTestResults
  • helpful to publish daily result summaries into a simple table template as done already for ALBERTA and TORONTO

* REMINDER #3: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

• Tier-1 Source URL: http://lcg-sam.cern.ch:8080/reports/site_avail.xsql
• Tier-2 Source URL: http://lcg-sam.cern.ch:8080/reports/t2/site_avail.xsql

OPS VO

• ops-availability-t2-20090811.png:
  

• ops-reliability-t2-20090811.png:
  

ATLAS VO

• atlas-availability-t2-20090811.png:
  

• atlas-reliability-t2-20090811.png:
  

APEL Accounting

APEL Publishing Status

• apel-gantt-20090811.png:
  

APEL Accounting Results

• apel-statistics-20090811.png:
  

Data Subscription Requests

• See: http://panda.cern.ch:25880/server/pandamon/query?mode=listPatterns&filter=&r_cloud=CANADASITES&tier=ALL&swversion=any&action=Continue

• This past week: None

Site Reports

Alberta

Recent Issues

• 28 production job failures (Athena errors, non-site related) [2009-08-06]
  • http://panda.cern.ch:25980/server/pandamon/query?job=1018318891

• 4 production job failures (Athena completed, but connectivity to panda server failed from WN [2009-08-03]
  • http://panda.cern.ch:25980/server/pandamon/query?job=1017968349

Outstanding Issues

• software release 15.1.0 issues [2009-07-21]
  • failing validation test which was previous successful in May and week ago
  • Joe to perform clean-up of 15.1.0 and reinstall the base release [2009-07-31]
  • further tests of KV of 15.1.0 at TRIUMF and SFU show identical failures so problem is not site related [2009-08-10]
    • installation of release tags forced
    • apparently same problem known to exist for 15.3.0 for which there will be a centrally provided fix released soon but may not be done for 15.1.0 [!AdSalvo]

• installation of grid services on new Westgrid-UA T2 site in progress
  • started preparations for new LCG-CE installation with dedicated set of a couple WNs for debugging purposes

• few minor issues pending correction on existing T2 facility:
  • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
  • pnfsd.log to be added to log-rotate [2009-03-31]
  • few WNs to be debugged with respect to missing stderr/stdout
  • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

• RGMA problem with snowpatch
  • GGUS ticket https://gus.fzk.de/ws/ticket_info.php?ticket=50855&from=search
• unscheduled downtime for approx. 1 hour to resolve issue with CE host certificate renewal [2009-08-07]

• debugging installation of ATLAS software releases via SL5 WNs [2009-07-29]
  • broader discussion ongoing within ATLAS re: SL5 for WNs
  • options available to force tags given that compilation during Kitvalidation can be ignored for SL5
    • force tags under debug

Outstanding Issues

• recurrent problems with missing files expected to exist on SE but unavailable during analysis and production jobs [2009-07-22]
• list of files expected to exist at SFU SE extracted from dump of TRIUMF LFC
  • http://trshare.triumf.ca/~steele/CA/datasets_in_CA/files_registered_at_SFU.txt
• list of files expected to exist at SFU SE but not found in pnfs: lost_atlasmcdisk_pnfs.files
• Sergey creating list of files that exist in pnfs but are not on any pools
• Sergey additionally creating list of files that exist in pnfs but are not on any pools
  • Oroginal reference ticket (7 missing files): https://savannah.cern.ch/bugs/?53395

Victoria

Recent Issues

Outstanding Issues

Toronto

Recent Issues

• production failures with errors on transfers from SE to WNs [2009-08-06]
  • heavy network traffic observed; one storage pool rebooted and dcache pools started thereafter [2009-08-06]

Outstanding Issues

• Still have two open GGUS tickets to resolve GIP information for the dCache 1.9
  installation. Not movement on this in over a week.
  https://gus.fzk.de/ws/ticket_info.php?ticket=48236
  https://gus.fzk.de/ws/ticket_info.php?ticket=48235

McGill

Recent Issues

• status?

Outstanding Issues

• attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
  • pilots (without payload) successfully running
  • next target is successful SAM tests
  • SE functionality still to be tested on current dCache instance with two pools

TRIUMF

Recent Issues

Outstanding Issues

• scheduled downtime for dCache patch application [2009-07-30]
  • postponed: TRIUMF planned to upgrade to 1.9.1-10, but a bug warning was issued, we need to reevaluate which upgrade path is best.

Production

Summary

site	success	failure	success (walltime)	failure (walltime)	efficiency	efficiency (walltime)
TRIUMF-LCG2	5177	1617	86621276	8375980	76.2%	91.2%
VICTORIA-LCG2	488	203	15127070	8940560	70.6%	62.9%
SFU-LCG2	133	15	4251446	559382	89.9%	88.4%
TORONTO-LCG2	988	184	110699384	3982327	84.3%	96.5%
ALBERTA-LCG2	311	28	11109691	48985	91.7%	99.6%
total	7097	2047	227808867	21907234	77.6%	91.2%

• Dashboard Summary (Link), Dashboard Summary (PDF)

Failed Jobs

(from Panda Monitor)

(from dashboard)

TRIUMF

name,rank,value 
TRFERROR,1,1218 
TRF_GEO_MISMATCH ,2,84 
TRF_INFILE_TOOFEW ,3,70 
TRF_SEGFAULT ,4,65 
TRF_OUTFILE ,5,58 
TRF_UNKNOWN ,6,52 
TRF_ATHENACRASH ,7,30 
EXEPANDA_JOBKILL_TOOMUCHSTDOUT,8,29 
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,9,4 
EXEPANDA_ATHENA_RAN-OUT-OF-MEMORY,10,3 
EXEPANDA_JOBKILL_SIGTERM,11,2 
EXEPANDA_JOBKILL_BYPILOT,12,1 
EXEPANDA_DQ2PUT_FILECOPYERROR,13,1

SFU

name,rank,value 
EXEPANDA_JOBKILL_SIGTERM,1,8 
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,5 
EXEPANDA_DQ2GET_CANTCALL,3,1 
TRF_SEGFAULT ,4,1

TORONTO

name,rank,value 
TRFERROR,1,135 
EXEPANDA_DQ2GET_INFILE,2,33 
EXEPANDA_JOBKILL_SIGTERM,3,7 
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,4 
TRF_SEGFAULT ,5,3 
TRF_UNKNOWN ,6,1 
EXEPANDA_DQ2PUT_FILECOPYERROR,7,1

VICTORIA

name,rank,value 
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,106 
EXEPANDA_JOBKILL_SIGTERM,2,74 
TRFERROR,3,14 TRF_OUTFILE_TOOFEW ,4,6 
EXEPANDA_DQ2PUT_FILECOPYERROR,5,1 
EXEPANDA_JOBKILL_BYPILOT,6,1 TRF_ATHENACRASH ,7,1

ALBERTA

name,rank,value
TRFERROR,1,28

Notes

-- BryanCaron - 11 Aug 2009