TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100330 (2010-03-30, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-03-30

General News

  • ADC News [20100330]
    • Past Week
      • ADC Weekly Meeting [20100329] (agenda)
    • Currently Ongoing this Week
      • BNL ATLAS Distributed Computing workshop [20100329] (agenda)

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100330]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2 - upgrade of WNs to SL5 completed 20100326. Now in the process of validating new gLite WN install and CE configuration [20100321]
      • VICTORIA-LCG2 - ATLAS software release installation and validation in progress

  • CA Cloud Squad Actions [20100330]
    • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
    • Open Issues and Ongoing Discussions:
      • need to follow-up with the schedconfig team as to why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • gLite Updates Announced [20100329]
    • Release of UPDATE 09 to gLite 3.2 Priority: Normal (link)

  • Squid Deployment [20100330]
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100323]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: OK (local squid server online and active)
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (local squid up but upgrade to SL5 WNs requires the new PoolCondPFC to be installed to pickup the Frontier configuration)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

  • General Information: WLCG office announced 20100308 that it will hold off publishing T1 and T2 accounting info for another 1 to 2 weeks from that time
    • Sites please confirm their historical APEL data that has been published looks correct

APEL Publishing Status

  • apel-status-aus-20100330.png:
    apel-status-aus-20100330.png

  • apel-status-ca-20100330.png:
    apel-status-ca-20100330.png

APEL Accounting Results

  • apel-statistics-aus-20100330.png:
    apel-statistics-aus-20100330.png

  • apel-statistics-ca-20100330.png:
    apel-statistics-ca-20100330.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100329

  • CA-ALBERTA-WESTGRID-T2 site status [20100330]
    • scheduled downtime in progress starting with the main campus computing centre power upgrades [starting 20100319]
      • https://goc.gridops.org/downtime?id=70455462 plus additional extension downtimes
      • network cabling maintenance planned for lighpath connection during this time - postponed until later in downtime period
      • the migration of the cluster WNs from SL4 to SL5 started morning of 20100322 and completed 20100326 with handover to ATLAS for software installation and CE reconfiguration that is now in progress since 20100328
      • temporary dCache pool mounted from previous T2 showing access problems due to iscsi errors; currently offline during the outage so no real impact yet [20100329]
    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

Outstanding Issues

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

  • CA-SCINET-T2 (and CA-ALBERTA-WESTGRID-T2) not included in ATLAS SRMv2 SAM tests (20100304) (in progress)

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

Reference Link for the Last Week

  • Within the CA Cloud:
    production-statistics-ca-20100330.png

  • How we fared to other ATLAS sites during the past week:
    production-statistics-all-20100330.png

  • Alternate Views (See link for last 3 days only):
    production-panda-statistics-summary-20100330.png

  • CA Cloud (last 3 days):
    production-panda-statistics-ca-20100330.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard:

Australia
name,rank,value
TRF_SEGFAULT ,1,106
TRFERROR,2,29
EXEPANDA_DQ2PUT_FILECOPYERROR,3,2
EXEPANDA_JOBKILL_BYPILOT,4,2

SFU
name,rank,value
TRFERROR,1,43
EXEPANDA_JOBKILL_BYPILOT,2,23
EXEPANDA_DQ2PUT_FILECOPYERROR,3,1

TRIUMF
name,rank,value
TRF_SEGFAULT ,1,422
EXEPANDA_Release_MISSING,2,416
TRFERROR,3,343
EXEPANDA_DQ2_SERVERERROR,4,153
EXEPANDA_JOBKILL_BYPILOT,5,66
EXEPANDA_DQ2_STAGEIN,6,16
TRF_UNKNOWN ,7,7

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • sites offline for SL4 to SL5 migration: VICTORIA-LCG2 and CA-ALBERTA-WESTGRID-T2
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes:
    • active tasks from PandaDB: 182
      • distribution by cloud: CA:10 CERN:3 DE:52 ES:9 FR:24 IT:18 ND:8 NL:29 TW:1 UK:4 US:24
    • Tue Mar.23
      • CA/TRIUMF-LCG2 job f./dccp failed.GGUS 56621 solved:storage pools crashed due to power supply problem,1 node was also running dcap door nodes were brought up again quickly.
    • Sat Mar.27
      • CA/CA-SCINET-T2 t.122378-122383 job f./missing 15.6.3.13 inst.GGUS 56776 solved:installation took a while,but completed.Scouts finished.

-- BryanCaron - 30 Mar 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100330.png r1 manage 820.8 K 2010-03-30 - 15:43 BryanCaron  
PNGpng apel-statistics-ca-20100330.png r1 manage 1259.7 K 2010-03-30 - 15:43 BryanCaron  
PNGpng apel-status-aus-20100330.png r1 manage 15.2 K 2010-03-30 - 15:44 BryanCaron  
PNGpng apel-status-ca-20100330.png r1 manage 24.7 K 2010-03-30 - 15:44 BryanCaron  
PNGpng production-panda-statistics-ca-20100330.png r1 manage 478.9 K 2010-03-30 - 16:24 BryanCaron  
PNGpng production-panda-statistics-summary-20100330.png r1 manage 1816.8 K 2010-03-30 - 16:06 BryanCaron  
PNGpng production-statistics-all-20100330.png r1 manage 944.8 K 2010-03-30 - 16:31 BryanCaron  
PNGpng production-statistics-ca-20100330.png r1 manage 385.8 K 2010-03-30 - 16:06 BryanCaron  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2010-03-30 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback