TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100316 (2010-03-16, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-03-16

General News

  • ADC News [20100316]
    • Past Week: ADC Weekly Meetings - now scheduled for Monday each week
      • [20100310] http://indico.cern.ch/conferenceDisplay.py?confId=86824
      • [20100315] http://indico.cern.ch/conferenceDisplay.py?confId=88030
        • batch system resource specifications update from G. Stewart
          • scratch space per job slot = 20 GB
          • max cputime to be kept at 48 hours (based on old cpu HEPSPEC performance) but should be re-normalized to latest HEPSPEC for modern cpus
          • max walltime to be reduced from 96 to 52 hours but should also be re-normalized to latest HEPSPEC
          • new minimum VMEM limit of 3 GB but preferred not to have no limit so as to let athena kill jobs and not the batch system
    • February 2010 re-processing ADC post-mortem is scheduled for 20100318 agenda

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100316]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status
      • CA-ALBERTA-WESTGRID-T2 - upgrade to SL5 of WNs scheduled for weekend of March 21 (to follow scheduled power maintenance)
      • VICTORIA-LCG2 - scheduled downtime over the past weekend for SL4 to SL5 migration - software release installation and validation in progress

  • CA Cloud Squad Actions [20100316]
    • marked VICTORIA-LCG2 site offline in Panda starting 20100315 for WN migration from SL4 to SL5 [20100315]
      • requested manual blacklisting of the site for glite WMS jobs in Ganga pending successful completion and validation of software releases
    • requested ADCoS shifter to test and mark online SFU-LCG2 (snowpatch) production queue following return from scheduled downtime for SL4 to SL5 migration [20100310]
      • analysis queue marked online directly by cloud contact
    • DDM default policy of no ESD distribution to all T2s in CA cloud is being applied since last week
    • requested update to ToA to include local squid cache installation details for CA-SCINET-T2 [20100309]
      • changes successfully applied during subsequent PoolCondPFC update
    • active discussion and iteration in regards to the condorg configuration for pilot job submission to CA-SCINET-T2
      • adjustments backfill queue settings using boost and depth for condorg
      • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct failing pilots on WNs (see site report below)
    • schedconfig parameter copysetup for the site updated at the request of cloud contact to correct problem of excessive input file copying to WNs observed by nagios alarm for a HammerCloud job received at the site [20100311]
      • related issue: it is unclear why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts; query to be sent to schedconfig team
    • Open Issues and Ongoing Discussions:
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • Squid Deployment [20100316]
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100316]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: OK (local squid server online and active)
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 ERROR (local squid up but upgrade to SL5 WNs requires the new PoolCondPFC to be installed to pickup the Frontier configuration)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

  • gLite 3.1 release update 62 [20100316] (announcement)
    • highlights include new versions of LFC and DPM, APEL, WMS, lcg-ManageVOTag, glite-info-templates and glite-info-provider-service

SAM Results

APEL Accounting

APEL Publishing Status

  • apel-status-aus-20100316.png:
    apel-status-aus-20100316.png

  • apel-status-ca-20100316.png:
    apel-status-ca-20100316.png

APEL Accounting Results

  • apel-statistics-aus-20100316.png:
    apel-statistics-aus-20100316.png

  • apel-statistics-ca-20100316.png:
    apel-statistics-ca-20100316.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100329

  • CA-ALBERTA-WESTGRID-T2 site status [20100316]
    • corrected configuration of apel and accounting log file parsing. Greater than 300k accounting records published to APEL which should appear in next few days on the CESGA portal [20100316]
    • scheduled downtime booked into GOCDB for main campus computing centre power upgrades [starting 20100319]
    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • SFU accounting information in APEL corrected again following move of MON box from snowpatch to snowpatch-hep [20100210]
    • was missing data from bugaboo and is now corrected
    • historical accounting information from 2008 was downloaded by APEL support staff and should appear in portal soon

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • scheduled downtime completed for WN migration from SL4 to SL5 [20100312] https://goc.gridops.org/downtime/list?id=73055437
    • queues marked offline in Panda by cloud contact prior to start of downtime; queues drained over weekend
    • SAM tests successful for site following SL5 installation on nodes; 32-bit version of glite WN software for SL4 being used with compatibility libraries since native 32-bit SL5 builds are not available from glite
    • site manually blacklisted in Ganga at request of cloud contacts to avoid brokering of glite WMS jobs to the site
    • central software release installation and verification requested as the next step

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • ToA entry for CA-SCINET-T2 updated by ADC to include local squid at the request of cloud contact and confirmed active following the update of the PoolCondPFC at the site

  • APEL accounting statistics published to central service [20100312]and visible via the site synchronization test page and the CESGA portal [20100315]

  • active discussion regarding condorg pilot submission configuration and associated numbers of pilots seen queued/running as a function of time at the site
    • adjustments backfill queue settings using boost and depth for condorg
    • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs
      • "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

Outstanding Issues

  • CA-SCINET-T2 (and CA-ALBERTA-WESTGRID-T2) not included in ATLAS SRMv2 SAM tests (20100304) (in progress)

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

  • HammerCloud test to site showed a few problematic jobs where the number of input files was large (>30) and occupied around 3 GB each on the local WN disk [20100310]
    • http://panda.cern.ch:25980/server/pandamon/query?job=1051607495
    • triggered Nagios diskfree-home alarm at the site
    • tracked down by Panda pilot expert to the queue copysetup parameter that enabled direct access to storage which did not have the two booleans as different
      • updated by schedconfig people and verified by subsequent tests that the 14 GB limit per job slot is not exceeded

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

Reference Link for the Last Week

  • Within the CA Cloud:
    dashboard-statistics-ca-20100316.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-statistics-all-20100316.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard:

TRIUMF

name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,98
TRF_SEGFAULT ,2,77
TRFERROR,3,69
EXEPANDA_JOBEXPIRED_SIXDAYS,4,42
TRF_UNKNOWN ,5,18
EXEPANDA_DQ2_STAGEIN,6,12
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,11
TRF_SEGVIO ,8,1
EXEPANDA_DQ2PUT_FILECOPYERROR,9,1

SFU

name,rank,value
TRFERROR,1,15
TRF_SEGFAULT ,2,3
TRF_UNKNOWN ,3,1
EXEPANDA_JOBKILL_BYPILOT,4,1

Australia

name,rank,value
TRFERROR,1,11
EXEPANDA_JOBKILL_BYPILOT,2,1
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,3,1
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,1

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • SCINET is still missing in dashboard. Reminder sent to ATLAS Monitoring expert.
    • very little production activity at VICTORIA-LCG2 and CA-ALBERTA-WESTGRID-T2 during the past week; likely due to the fact that these are the two remaining SL4 clusters within the CA cloud

  • Yuri's notes:
    • active tasks from PandaDB: 59
    • distribution by cloud:
      • CA:4 DE:9 ES:2 FR:11 IT:5 ND:6 NL:5 TW:1 UK:3 US:13

    • Tue Mar 9
      • CA/CA-SCINET-T2 FT f./USER_ER./failed to contact on remote SRM.GGUS 56286 in progress:internal network switch problem recovered on reboot

-- BryanCaron - 16 Mar 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100316.png r1 manage 836.5 K 2010-03-16 - 17:07 BryanCaron  
PNGpng apel-statistics-ca-20100316.png r1 manage 1215.6 K 2010-03-16 - 17:08 BryanCaron  
PNGpng apel-status-aus-20100316.png r1 manage 15.1 K 2010-03-16 - 17:06 BryanCaron  
PNGpng apel-status-ca-20100316.png r1 manage 23.4 K 2010-03-16 - 17:06 BryanCaron  
PNGpng dashboard-statistics-all-20100316.png r1 manage 923.9 K 2010-03-16 - 17:41 BryanCaron  
PNGpng dashboard-statistics-ca-20100316.png r1 manage 486.4 K 2010-03-16 - 17:29 BryanCaron  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2010-03-16 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback