TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100223 (2010-02-23, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-02-23

General News

  • ADC News [20100223]
    • Past Week: ADC Weekly Meeting [20100217] http://indico.cern.ch/conferenceDisplay.py?confId=85474
      • MC09 production
        • redigi and rereco with 15.6.3.3 started that week for 280M events
        • plan to redo all MC09 simulations in coming weeks so T2s should be busy
      • reprocessing of collision data
        • data consolidation at Tier-1s and CERN started [20100219], and data distribution within clouds ongoing

  • site SL4 to SL5 migration plans for releases 15.6.3 and beyond [20100223]
    • each site currently running with SL4 WNs should please provide an updated on their WN transition to SL5 plans and deployment status

  • CA Cloud Squad Actions [20100223]
    • updated CA cloud T2 site pairings and shares for MCDISK to include 15% share to VICTORIA-LCG2 [20100219]
    • submitted updated site WN disk space information to Panda schedconfig for those with above default (14 GB) amounts [20100219]
    • temporary increase in timeout parameter to 7200s for FTS TRIUMF-TORONTO2 channel [20100217]
    • SFU-LCG2-snowpatch-hep-atlas-lcgpbs (SL4) queue set offline for production to avoid first queue info (SL5 bugaboo) being used for incorrect job brokering in Panda. SL4 analysis queues still OK. [20100217]
    • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
    • centrally published WLCG availability and reliability reports to now (February 2010 onward) to include updated T2 sites CA-SCINET-T2 and CA-ALBERTA-WESTGRID-T2 [20100218]
    • Australia removed from manual blacklist for analysis in Ganga and marked online in Panda at request/action by cloud contact [20100215]
    • Ongoing Discussion:
    • Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • gLite News [20100223]
    • updated release of CA rpms (IGTF 1.33 and 1.34)
    • update to lcg-vomscerts-5.8.0-1.noarch.rpm required before 20100225 to accommodate new host certificate of lcg-voms.cern.ch
    • update 61 to gLite 3.1 announced
      • includes above mentioned update to lcg-vomscerts and new version of dCache srmclient

  • Squid deployment [20100223]
    • tested in SAM for ATLAS https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: Error (Frontier setup now installed but error connecting to either ALBERTA or SFU Squid cache)
      • ACTION ITEM: SFU and ALBERTA to confirm configuration of Squid ACLs to accept connections from CA-SCINET-T2
    • SFU-LCG2 OK (for both CEs but test for SL4 last ran 20100201; local is OK; backup squid at Victoria ok)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

  • announcement that the APEL central database is back online with data updated to accounting portal [20100217]

APEL Publishing Status

  • apel-status-aus-20100223.png:
    apel-status-aus-20100223.png

  • apel-status-ca-20100223.png:
    apel-status-ca-20100223.png

APEL Accounting Results

  • apel-statistics-aus-20100223.png:
    apel-statistics-aus-20100223.png

  • apel-statistics-ca-20100223.png:
    apel-statistics-ca-20100223.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100315

  • CA-ALBERTA-WESTGRID-T2 site status [20100223]

    • scheduled downtime booked into GOCDB for main campus computing centre power upgrades [20100319]

    • open items to be followed-up:
      • APEL accounting information publishing (open ticket in progress with new email to central apel support) ggus 54707
      • SL5 WNs and additional CE [20100216]
        • test system installed for ATLAS verification. Needed set of RPMs are installed and software release being tested with kit validation
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • Action Item: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • production job failures for 15.6.3.x release on snowpatch (SL4) based upon brokering info for bugaboo (SL5) [20100217]

  • release installation problems on bugaboo for 15.6.3.x (see below)

Outstanding Issues

  • pending release installations for 15.6.3.x on bugaboo (20100202)
    • installation files not replicated to SE
    • hanging lcg-rep of release tarball from remote to local SE
    • eventually passed final cache release installation [20100218]
    • cause of installation and lcg-rep hanging still not understood

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

  • apel publishing [20090825]
    • apel publishing information looks much improved and realistic (see above plots) following the return of the central APEL services [20100217]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above) but returns intermittently
      • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)
      • also atcan group appears at this site only

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • cloud contact requested updated site MC share to DDM Ops following the addition of 22 TB to the site MCDISK storage [20100219]

  • bad node (c03b02) identified via python error in production activity and marked offline by site admins [20100218]

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • CA-SCINET-T2
    • migrated to NFS server for atlas software area Friday afternoon [20100219]
      • increased maximum number of concurrent jobs from 256 to 512
      • CE still had old software area mounted until 20100222

Outstanding Issues

  • slow execution (> 2 hours) of build jobs compared to other sites (approx. 10 mins)
    • software installation area still under GPFS.
    • GPFS/NFS server node being built but requires testing.
    • also to test NFS server using DDN LUNs directly.

  • SAM Frontier Tests currently failing (see above and next item)

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA
    • Action Item to SFU and ALBERTA: Confirm Squid ACLs to accept CA-SCINET-T2 network

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • large number of file transfer failures to TRIUMF-LCG2_DATATAPE [20100222]
    • Job failed with forced timeout after 43200 seconds
    • see above for the submitted ggus ticket (55817)

  • less than 1 day scheduled downtime for dCache migration from pnfs to chimera [20100215]

  • urgent restart of dCache/SRM postgres database performed [20100219]

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • 6 nodes marked offline in PBS for rack relocation in the data centre [20100216]

  • removed from manual blacklist for analysis in Ganga and marked online in Panda at request/action by cloud contact [20100215]
    • still some issues observed with job failures during compilation step [20100216]
      • is this still an issue?

  • patched Dell r710 network card driver bug which should cure storage timeout and unreachable errors [20100219]

  • increased size of DPM mysql cache from 8 MB to 1 GB to optimise DPM head node and correct storage access errors (lcg_cr) [20100223]

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 37060 3573 474901511 3477362 91.2% 99.3%
CA-ALBERTA-WESTGRID-T2 7083 76 158170924 1442566 98.9% 99.1%
SFU-LCG2 4670 840 93277338 238308 84.8% 99.7%
VICTORIA-LCG2 2697 209 119723058 1391019 92.8% 98.9%
Australia-ATLAS 1800 65 31219872 1247363 96.5% 96.2%
total 53310 4763 877292703 7796618 91.8% 99.1%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:
me,rank,value
EXEPANDA_JOBKILL,1,3091
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,146
EXEPANDA_DQ2PUT_FILECOPYERROR,3,87
TRFERROR,4,58
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,5,38
EXEPANDA_GET_NOSUCHFILE,6,36
EXEPANDA_UNKNOWN_CATCHALL,7,27
EXEPANDA_DQ2_STAGEIN,8,27
TRF_SEGFAULT ,9,26
EXEPANDA_ATHENA_START-FAILED-MISSING-INSTALLATION,10,17
EXEPANDA_DQ2GET_INFILE,11,9
TRF_UNKNOWN ,12,6
EXEPANDA_GRID-PROXY_INVALID,13,2
TRF_SEGVIO ,14,2
EXEPANDA_PILOTEXCEPTION,15,1
EXEPANDA_JOBKILL_SIGTERM,16,1
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,17,1

Alberta:
me,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,73
EXEPANDA_GET_SIZEMISMATCH,2,2
EXEPANDA_DQ2PUT_FILECOPYERROR,3,1
TRFERROR,4,1

Australia:
name,rank,value
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,1,21
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,16
TRFERROR,3,14
EXEPANDA_JOBKILL_BYPILOT,4,6
EXEPANDA_DQ2PUT_FILECOPYERROR,5,6
EXEPANDA_GET_GLOBUSSYSTEMERROR,6,2

SFU:
name,rank,value
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,1,832
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,3
EXEPANDA_GET_SIZEMISMATCH,3,2
EXEPANDA_DQ2PUT_FILECOPYERROR,4,1
TRFERROR,5,1
EXEPANDA_PUT_LFCMODULEIMPORT,6,1

Toronto/SCINET: (missing)

Victoria:
name,rank,value
EXEPANDA_TRF_INSTALL-DIR-NOT-FOUND,1,89
EXEPANDA_JOBEXPIRED_SIXDAYS,2,82
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,27
EXEPANDA_DQ2_STAGEIN,4,5
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,5,2
EXEPANDA_DQ2GET_INFILE,6,2
TRFERROR,7,1
EXEPANDA_DQ2PUT_FILECOPYERROR,8,1

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
BNL 151264 21296 1585000453 291534657 87.7% 84.5%
SARA 91408 23889 1063262318 367198435 79.3% 74.3%
NDGF 94383 4213 1875493860 19393486 95.7% 99%
FZK 84971 7862 1545893595 90693992 91.5% 94.5%
RAL 55491 9137 1004233866 150693504 85.9% 87%
LYON 46289 16061 661835323 220549004 74.2% 75%
TRIUMF 53310 4763 877292703 7796618 91.8% 99.1%
CNAF 27451 11126 560241468 88723709 71.2% 86.3%
None 26823 3848 501091390 23453420 87.5% 95.5%
PIC 23228 1920 261276345 13170956 92.4% 95.2%
ASGC 17622 363 584459120 28560285 98% 95.3%
total 672240 104478 1.0520080441e+10 1301768066 86.5% 89%
  • My Notes:
    • large discrepancy seen between Panda and dashboard for Australia in finished jobs (7.5k vs 1.8k). Other sites too but not this large.
    • SCINET missing in dashboard - no response from support since last week.
    • Lower than 95% efficiencies:
      • TRIUMF: jobs killed by sender
      • SFU: snowpatch used for SL5 releases (put offline on 18 Feb) for production.
      • Victoria: WN c03b02 had a problem and caused the setups to fail.

  • Yuri's notes:
    • - active tasks from PandaDB: 379
      • -- distribution by cloud:
      • CA:19 CERN:1 DE:44 ES:24 FR:57 IT:16 ND:54 NL:13 TW:2 UK:35 US:114
    • Tue Feb.16
      • + CA/CA-SCI.NET AOD FT f./timeout.GGUS 55626 verified:timeout increased 1800->2700s for TRIUMF-TORONTO2 and TORONTO2-TRIUMF channel.
    • Sun Feb.21
      • - CA/CA-SCINET-T2 job f./missing 15.6.3.7 inst.GGUS 55761 assigned: the installation seems to have completed at about 8pm local time.
    • Mon Feb.22
      • - CA/TRIUMF-LCG2_DATATAPE many FT f./SRMV2STAGER timeout.Elog 9734.
      • - CA/TRIUMF-LCG2_DATADISK FTS f./getTransferJobStatus:Not authorised to query request.GGUS 55782 assigned:One switch (FTS<->Oracle DB)was rebooted due to memory leak.

Weekly Meeting Notes

-- BryanCaron - 22 Feb 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt ADC-CA-notes-20100223.txt r1 manage 4.6 K 2010-02-23 - 22:40 BryanCaron  
PDFpdf DashboardSummary.pdf r1 manage 111.2 K 2010-02-23 - 19:59 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 98.2 K 2010-02-23 - 19:59 AsokaDeSilva  
PDFpdf PandaStats.pdf r1 manage 253.6 K 2010-02-23 - 19:59 AsokaDeSilva  
PNGpng apel-statistics-aus-20100223.png r1 manage 831.6 K 2010-02-23 - 19:36 BryanCaron  
PNGpng apel-statistics-ca-20100223.png r1 manage 1093.2 K 2010-02-23 - 19:37 BryanCaron  
PNGpng apel-status-aus-20100223.png r1 manage 14.5 K 2010-02-23 - 19:37 BryanCaron  
PNGpng apel-status-ca-20100223.png r1 manage 22.2 K 2010-02-23 - 19:37 BryanCaron  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2010-02-23 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback