TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100406 (2010-04-06, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-04-06

General News

  • ADC News [20100406]
    • Past Week
      • ADC Weekly Meeting - none due to Easter holiday
      • confirmed LHC stop scheduled for Thursday April 8 - 08:00 to 18:00 CEST.
        • opportunity for sites to perform small interventions during this day
        • longer 3 day technical stop still tentatively scheduled for April 26 to 28

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100406]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2 - upgrade of WNs to SL5 completed [20100326]. New gLite WN install and CE configuration validated through SAM tests; ATLAS release installation nearly completed Now in the process of validating new gLite WN install and CE configuration [20100405]
      • VICTORIA-LCG2 - ATLAS software release installation and validation completed; marked online for production [20100330]

  • CA Cloud Squad Actions [20100406]
    • VICTORIA-LCG2 production queue in Panda marked to 'test' [20100330]. Validated and marked online by ADCoS shifter [20100331]
      • site analysis queue currently offline pending debugging via SiteTests; manually blacklisted in gLite WMS
    • CA-ALBERTA-WESTGRID-T2 manually blacklisted in gLite WMS [20100405]
    • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
    • New Issue Tracking
    • Open Issues and Ongoing Discussions:
      • need to follow-up with the schedconfig team as to why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • Squid Deployment [20100406]
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100323]
    • CA-ALBERTA-WESTGRID-T2 ERROR (local squid server offline since 20091108; WN upgrade to SL5 requires reinstall of PoolCondPFC setup)
    • CA-SCINET-T2: OK (local squid server online and active)
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (new PoolCondPFC installed to pickup the Frontier configuration; local Squid installation still shows errors but backup is OK)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

  • General Information: WLCG office announced 20100308 that it will hold off publishing T1 and T2 accounting info for another 1 to 2 weeks from that time
    • Sites please confirm their historical APEL data that has been published looks correct
    • See EGEE Accounting Portal

APEL Publishing Status

  • apel-status-ca-20100406.png:
    apel-status-ca-20100406.png

  • apel-status-aus-20100406.png:
    apel-status-aus-20100406.png

APEL Accounting Results

  • apel-statistics-ca-20100406.png:
    apel-statistics-ca-20100406.png

  • apel-statistics-aus-20100406.png:
    apel-statistics-aus-20100406.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517

  • CA-ALBERTA-WESTGRID-T2 site status [20100406]
    • ended scheduled downtime that started with the main campus computing centre power upgrades and continued with the WN migration to SL5 [starting 20100319]
      • network cabling maintenance planned for lighpath connection during this time - postponed until later in downtime period
      • the migration of the cluster WNs from SL4 to SL5 started morning of 20100322 and completed 20100326 with handover to ATLAS for software installation and CE reconfiguration [started 20100328]
      • ATLAS software release installation commenced 20100404 and nearly completed
      • dpool01 node encountered errors due to offline pool idisk02_00 (external NFS mounted pool); all pools hosted by dpool01 ended up going flip-flopping between online/online status causing failures for ATLAS DDM and SAM activities. Temporarily remounted idisk01 and restarted pool node, and then marked it offline so that lock file could be accessed. Ok now [20100404]
    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • spacetokens update request [20100330]
    • ACTION ITEM: Could the site please update asap the spacetoken reservations in the dCache to match the requested values to bring online the profile of the 2009 and 2010 WLC commitments?
      • DATADISK 115 TB, MCDISK 115 TB, PRODDISK 15 TB, SCRATCHDISK 15 TB, GROUPDISK 25 TB, LOCALGROUPDISK 50 TB, HOTDISK 1 TB

Outstanding Issues

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

  • CA-SCINET-T2 (and CA-ALBERTA-WESTGRID-T2) not included in ATLAS SRMv2 SAM tests (20100304) (in progress)

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
    atlas-production-ca-20100406.png

  • How we fared to other ATLAS sites during the past week:
    production-statistics-all-20100406.png

  • Alternate Views (See link for last 3 days only):
    panda-production-statistics-summary-20100406.png
  • CA Cloud (last 3 days):
    panda-production-statistics-ca-20100406.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard:

Australia
name,rank,value
TRFERROR,1,6

SFU
name,rank,value
TRFERROR,1,18
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,9
EXEPANDA_JOBKILL_BYPILOT,3,3
EXEPANDA_GET_SIZEMISMATCH,4,2
EXEPANDA_DQ2_STAGEIN,5,1
EXEPANDA_DQ2PUT_FILECOPYERROR,6,1

TRIUMF
name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,1040
EXEPANDA_Release_MISSING,2,338
TRFERROR,3,27
EXEPANDA_DQ2_STAGEIN,4,20
EXEPANDA_DQ2PUT_FILECOPYERROR,5,17
EXEPANDA_JOBKILL_SIGTERM,6,4
EXEPANDA_JOBKILL_BYPILOT,7,4
TRF_UNKNOWN ,8,3
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,9,1
TRF_SEGVIO ,10,1

Victoria
name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,135
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,74
EXEPANDA_JOBKILL_NOLOCALSPACE,3,10
EXEPANDA_DQ2GET_INFILE,4,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,2
TRFERROR,6,2
TRF_SEGVIO ,7,1

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • sites offline for SL4 to SL5 migration: CA-ALBERTA-WESTGRID-T2
    • information missing in dashboard (production) still for CA-SCINET-T2.
    • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
      • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)

  • Yuri's notes:
    • active tasks from PandaDB: 567
      • distribution by cloud: CA: 23 CERN: 1 DE: 174 ES: 29 FR: 154 IT: 36 ND: 17 NL: 54 TW: 3 UK: 14 US: 62
    • Sat Apr.3
      • - CA/CA-ALBERTA-WESTGRID-LCG2 many FT f./GRIDFTP_ER./Broken pipe.GGUS 57012 assigned.
    • Sun Apr.4
      • - CA/TRIUMF job f./15.6.6.5/i686-slc5-gcc43-opt not found.GGUS 57021 assigned:result of schedconfig update procedure for INSTALLEDSW table
      • - CA/TRIUMF AOD FT f./USER_ER./source file doesn't exist.GGUS 57028 assigned,Savannah 65316:seems lost files.
    • Mon Apr.5
      • -> Cloud/site PROD efficiency < 50% : CA,DE,TW
      • - CA/CA-SCINET-T2 500 job f./killed by signal 15.Savannah 65339.
      • - CA/CA-SCINET-T2 122 job f./Put er./copy timed out.GGUS 57035 assigned

-- BryanCaron - 06 Apr 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100406.png r1 manage 859.1 K 2010-04-06 - 15:44 BryanCaron  
PNGpng apel-statistics-ca-20100406.png r1 manage 1306.2 K 2010-04-06 - 15:44 BryanCaron  
PNGpng apel-status-aus-20100406.png r1 manage 15.0 K 2010-04-06 - 15:45 BryanCaron  
PNGpng apel-status-ca-20100406.png r1 manage 24.5 K 2010-04-06 - 15:45 BryanCaron  
PNGpng atlas-production-ca-20100406.png r1 manage 444.4 K 2010-04-06 - 16:06 BryanCaron  
PNGpng panda-production-statistics-ca-20100406.png r1 manage 472.1 K 2010-04-06 - 16:20 BryanCaron  
PNGpng panda-production-statistics-summary-20100406.png r1 manage 1904.3 K 2010-04-06 - 16:19 BryanCaron  
PNGpng production-statistics-all-20100406.png r1 manage 957.6 K 2010-04-06 - 16:32 BryanCaron  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2010-04-06 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback