TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100323 (2010-03-23, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-03-23

General News

  • ADC News [20100323]
    • Past Week
      • ADC Weekly Meeting [20100322] (agenda)
      • ADC February 2010 re-processing post-mortem [20100318] (agenda)
    • Upcoming Week
      • BNL ATLAS Distributed Computing workshop [20100329] (agenda)

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100323]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2 - ongoing upgrade to SL5 of WNs scheduled following downtime for power maintenance [20100321]
      • VICTORIA-LCG2 - scheduled downtime completed for SL4 to SL5 migration - software release installation and validation in progress

  • CA Cloud Squad Actions [20100323]
    • set ANALY_VICTORIA queue in Panda to 'test' status for follow-up testing with SiteTests and release 15.5.1 that was successfully installed [20100323]
    • marked CA-ALBERTA-WESTGRID-T2 analysis and production queues offline in Panda for weekend shutdown and migration from SL4 to SL5 [started 20100319]
    • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
    • Open Issues and Ongoing Discussions:
      • need to follow-up with the schedconfig team as to why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • Squid Deployment [20100323]
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100323]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: OK (local squid server online and active)
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (local squid up but upgrade to SL5 WNs requires the new PoolCondPFC to be installed to pickup the Frontier configuration)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

  • General Information: WLCG office will hold off publishing T1 and T2 accounting info for another 1 to 2 weeks [announced 20100308]
    • Sites please confirm their historical APEL data that has been published looks correct
    • CA-ALBERTA-WESTGRID-T2 accounting records received by APEL [20100316] and began appearing in the portal and synchronization tests a couple days later

APEL Publishing Status

  • apel-status-aus-20100322.png:
    apel-status-aus-20100322.png

  • apel-status-ca-20100322.png:
    apel-status-ca-20100322.png

APEL Accounting Results

  • apel-statistics-aus-20100323.png:
    apel-statistics-aus-20100323.png

  • apel-statistics-ca-20100323.png:
    apel-statistics-ca-20100323.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100329

  • CA-ALBERTA-WESTGRID-T2 site status [20100323]
    • last week corrected configuration of apel and accounting log file parsing. Greater than 300k accounting records published to APEL which appeared appear a couple days later on the CESGA portal [20100319]
    • scheduled downtime in progress starting with the main campus computing centre power upgrades [starting 20100319]
      • https://goc.gridops.org/downtime?id=70455462
      • network cabling maintenance planned for lighpath connection during this time - postponed until later in downtime period
      • the migration of the cluster WNs from SL4 to SL5 started morning of 20100322 with access to WNs for ATLAS software installation and CE reconfiguration hopefully today
    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

Outstanding Issues

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

  • CA-SCINET-T2 (and CA-ALBERTA-WESTGRID-T2) not included in ATLAS SRMv2 SAM tests (20100304) (in progress)

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

  • observed files written to tape system under atlasmctape/groupmc09 that do not follow convention and with missing HITS directory, associated with group production activity being run specifically on the SciNet site [20100322]
    • problem induced by the production path not being completely clarified or validated prior to the production start
    • ADC decision is to abort current tasks, rename and resubmit such that no files to be written to tape [20100323]

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

Reference Link for the Last Week

  • Within the CA Cloud:
    dashboard-production-ca-201023.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-production-all-201023.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard:

AUSTRALIA-ATLAS
name,rank,value
TRFERROR,1,8
EXEPANDA_DQ2PUT_FILECOPYERROR,2,1

SFU-LCG2
name,rank,value
TRFERROR,1,26
TRF_UNKNOWN ,2,2
TRF_SEGFAULT ,3,1

TRIUMF-LCG2
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,797
EXEPANDA_DQ2_STAGEIN,2,169
TRF_SEGFAULT ,3,128
TRFERROR,4,37
TRF_UNKNOWN ,5,17
EXEPANDA_DQ2PUT_FILECOPYERROR,6,3
EXEPANDA_UNKNOWN_CATCHALL,7,2
EXEPANDA_GET_SIZEMISMATCH,8,1
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,9,1
EXEPANDA_JOBKILL_BYPILOT,10,1

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • sites offline for SL4 to SL5 migration: VICTORIA-LCG2 and CA-ALBERTA-WESTGRID-T2
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes:
    • active tasks from PandaDB: 141
      • distribution by cloud: CA:17 CERN:2 DE:17 ES:2 FR:18 IT:16 ND:17 NL:6 TW:1 UK:8 US:37
    • 15.6.3.10 mc09_7TeV simu task 121016 failures: Error details:
        • taskBuffer: expired after waiting for input data for 2 day.Savannah 64432.All jobs failed/came to waiting in CA.Missing input evgen/EVNT dataset registered/located in US,FR,NDGF,NL T1-s,not in CA. Fixed: evgen subscribed,scouts completed OK next attempt. Task is in "running" state.

-- BryanCaron - 22 Mar 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100323.png r1 manage 80.5 K 2010-03-23 - 05:55 BryanCaron  
PNGpng apel-statistics-ca-20100323.png r1 manage 152.1 K 2010-03-23 - 05:56 BryanCaron  
PNGpng apel-status-aus-20100322.png r1 manage 15.1 K 2010-03-23 - 05:56 BryanCaron  
PNGpng apel-status-ca-20100322.png r1 manage 24.3 K 2010-03-23 - 05:57 BryanCaron  
PNGpng dashboard-production-all-201023.png r1 manage 92.2 K 2010-03-23 - 07:15 BryanCaron  
PNGpng dashboard-production-ca-201023.png r1 manage 39.9 K 2010-03-23 - 06:47 BryanCaron  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2010-03-23 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback