TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091201 (2009-12-01, AsokaDeSilva) EditAttach

Tier1 and Tier-2 Weekly Report 2009-12-01 [In Progress]

General News

  • ADC News [20091201]
    • ATLAS software and computing week at CERN agenda

  • security announcement reminders [20091201]
    • vulnerability announced to sites via EGEE broadcast [20091105]
      • please take indicated steps as soon as possible and announce to list (downtime and completion)
    • critical and important java security errata announced for SL [20091116]

  • information system reminders [20091201]
    • sites should please verify their published logicalCPU, physicalCPU and kSI2K values from their site BDIIs.
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU and CA-ALBERTA-WESTGRID-T2.

  • CA Cloud Squad Actions [20091201]
    • corrupt file on TRIUMF-LCG2_MCDISK [20091201]
    • missing EVNT datasets at sites but believed to be present in Panda - check DQ2 [20091024] [expired; ticket closed in Savannah]
    • checksum mismatch on dataset file(s) at Victoria [20091105]
      • since these were EVNT files the situation is similar to the above even though there is no Savannah bug for it. Therefore consider it expired and closed.
    • missing input files for tasks in CA cloud [20091108]

  • Squid deployment [20091201]
    • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning: all sites good (WARN is ok).
      • ALBERTA-LCG2 WARN (local squid server offline since 20091108)
      • SFU-LCG2 WARN (for both CEs; Victoria squid is the backup but not online)
      • TORONTO-LCG2 WARN (local squid not online; primary backup points to ALBERTA squid but fails due to network routing issue; secondary backup to SFU OK)
      • TRIUMF-LCG2 OK (for both CEs; local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 WARN (local squid not online; backup points to SFU squid)
      • ATLAS_AUSTRALIA - ERROR - to be followed-up
    • ReadReal test passes at all sites; manual fix to PFC at TRIUMF and SFU pending release of update to dq2-tools

  • dCache REMINDERS

    • golden release of dCache announced: version 1.9.5 (see dCache user-forum announcement)
      • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement)
      • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
        • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • WestGrid-Alberta: all nodes at 1.9.3-3
        • TRIUMF: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
        • VICTORIA: 1.8.0.12p5
        • ALBERTA: 1.8.0.12p6

  • REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

OPS VO

  • ops-availability-t2-20091201.png:
    ops-availability-t2-20091201.png

  • ops-reliability-t2-20091201.png:
    ops-reliability-t2-20091201.png

ATLAS VO

  • atlas-availability-t2-20091201.png:
    atlas-availability-t2-20091201.png

  • atlas-reliability-t2-20091201.png:
    atlas-reliability-t2-20091201.png

APEL Accounting

APEL Publishing Status

  • apel-status-ca-20091201.png:
    apel-status-ca-20091201.png

  • apel-status-aus-20091201.png:
    apel-status-aus-20091201.png

APEL Accounting Results

  • apel-statistics-ca-20091201.png:
    apel-statistics-ca-20091201.png

  • apel-statistics-aus-20091201.png:
    apel-statistics-aus-20091201.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • single SAM test failures for CE-sft-lcg-rm and SRMv2-put reported [20091117]
    • subsequent tests are successful

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • installation status: WLCG on WestGrid-Alberta cluster [20091110]
    • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
      • certified as production ready by CERN ROC and under setup and validation for ATLAS activities
      • Data Distribution
        • successfully passing FT after two updates (ToA and FT result collection) by ATLAS DDM Ops
        • request made to DDM ops to have DATADISK and MCDISK shares updated for new site at 15% and 0%, respectively
      • Job Submission
        • ATLAS software installation currently underway (SL4 WNs)
      • investigating two remaining issues being observed in SAM tests
        • grid certificate CRLs not correctly updating; CRL update info missing from .r0 files
        • intermittent Job Submission (JS) errors in SAM tests from OPS VO where globus-gatekeeper encounters LCAS error and fails to map user to local pool account and therefore does not submit job
          • only appears with OPS VO job submissions so far
      • savannah ticket open in regards to migrating current ALBERTA-LCG2 site storage to CA-ALBERTA-WESTGRID-T2 site
      • update to dCache 1.9.5-18 (golden release) on hold
      • update setup to multi-homed pools to separate internal and external traffic (instructions from Sergey) on hold

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • few minor issues pending correction on existing T2 facility:
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • scheduled downtime [20091112 until 20091114] https://goc.gridops.org/downtime?id=48755482
    • power breakers upgrade
      • few worker nodes lost - cannot be powered on
    • DDN and Lustre work
      • Lustre updated to 1.8.1.1 + patches - extended downtime required awaiting outside expert intervention over the weekend
      • required to relocate release data from one shelf to another - problems with hard-links not surviving procedure; KV being run of all base releases to confirm installation
    • dCache upgrades
      • update to 1.9.5-8 attempted but resulted in 'no space available errors'; rolled back to 1.9.2-8
    • Snowpatch was available but no site SE services.

  • SL5 (bugaboo) queue marked in 'test' within Panda pending follow-up post scheduled downtime
    • additional separate analysis queue (ANALY_SFU_bugaboo) requested of Panda team and created

Outstanding Issues

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • SCRATCHDISK became full [20091112]
    • increased size of SCRATCHDISK by approx. 0.75 TB
    • Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Outstanding Issues

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • CA-SCINET-T2 installation:
    • new site certification by CERN ROC: https://gus.fzk.de/ws/ticket_info.php?ticket=49823
    • dCache 1.9.4 now on pools but finding some configuration issues
    • dCache admin node being updated to CentOS 53
    • DDN firmware update debugging from vendor during downtime
    • designated node for squid cache installation

Outstanding Issues

  • cleanup of suspended transfers from 20091031 still pending on SE

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • unscheduled downtime planned for 2 hours [20091117]
    • oracle security update

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

  • attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
    • pilots (without payload) successfully running
    • next target is successful SAM tests
    • SE functionality still to be tested on current dCache instance with two pools

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 22434 595 514408078 3674400 97.4% 99.3%
SFU-LCG2 2229 37 66560540 1321646 98.4% 98.1%
Australia-ATLAS 1881 9 61049473 158244 99.5% 99.7%
VICTORIA-LCG2 1046 16 65796853 750626 98.5% 98.9%
ALBERTA-LCG2 890 2 33755576 163933 99.8% 99.5%
TORONTO-LCG2 105 0 5812444 0 100% 100%
total 28585 659 747382964 6068849 97.7% 99.2%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
TRF_UNKNOWN ,1,310
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,92
TRFERROR,3,63
EXEPANDA_DQ2_STAGEIN,4,37
EXEPANDA_JOBKILL_SIGTERM,5,28
TRF_SEGFAULT ,6,27
EXEPANDA_DQ2PUT_FILECOPYERROR,7,24
EXEPANDA_UNKNOWN_CATCHALL,8,7
EXEPANDA_JOBKILL_BYPILOT,9,5
EXEPANDA_GET_ADLER32MISMATCH,10,2


Alberta:

name,rank,value
TRF_SEGFAULT ,1,1
EXEPANDA_JOBKILL_BYPILOT,2,1


Australia:

name,rank,value
EXEPANDA_JOBKILL_SIGTERM,1,8
TRFERROR,2,1


SFU:

name,rank,value
EXEPANDA_JOBDISPATCHER_HEARTBEAT,1,17
EXEPANDA_JOBKILL_SIGTERM,2,12
EXEPANDA_JOBKILL_BYPILOT,3,7
EXEPANDA_PUT_GLOBUSSYSTEMERROR,4,1


Toronto:

(nothing)


Victoria:

name,rank,value
TRF_SEGFAULT ,1,7
EXEPANDA_JOBKILL_BYPILOT,2,5
EXEPANDA_JOBKILL_BYSERVER,3,2
TRF_OUTFILE ,4,1
EXEPANDA_PUT_LFCMODULEIMPORT,5,1

Notes

  • How we fared to other ATLAS sites ...
cloud success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
FZK 191397 11626 5012695967 169834570 94.3% 96.7%
BNL 175095 14080 4128225630 56182015 92.6% 98.7%
LYON 149676 3569 3714841146 106241381 97.7% 97.2%
RAL 137841 14451 3944945323 289819240 90.5% 93.2%
SARA 90846 5602 2058883058 92294533 94.2% 95.7%
NDGF 60587 2445 1829962980 41835152 96.1% 97.8%
None 7968 45383 215872190 10044548 14.9% 95.6%
PIC 40833 1075 754292284 33652054 97.4% 95.7%
TRIUMF 28585 659 747382964 6068849 97.7% 99.2%
CNAF 14055 15040 622663670 107851632 48.3% 85.2%
ASGC 25067 502 786759157 26099981 98% 96.8%
CERN 398 187 4172719 3661392 68% 53.3%
total 922348 114619 2.3820697088e+10 943585347 88.9% 96.2%

  • My Notes:
    • dpool13 issues at TRIUMF.
    • bugaboo is offline
    • Toronto is offline.

-- BryanCaron - 01 Dec 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 104.1 K 2009-12-01 - 19:49 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 88.6 K 2009-12-01 - 19:49 AsokaDeSilva  
PNGpng apel-statistics-aus-20091201.png r1 manage 79.7 K 2009-12-01 - 18:56 BryanCaron  
PNGpng apel-statistics-ca-20091201.png r1 manage 133.5 K 2009-12-01 - 18:56 BryanCaron  
PNGpng apel-status-aus-20091201.png r1 manage 15.1 K 2009-12-01 - 18:55 BryanCaron  
PNGpng apel-status-ca-20091201.png r1 manage 21.7 K 2009-12-01 - 18:55 BryanCaron  
PNGpng atlas-availability-t2-20091201.png r1 manage 252.4 K 2009-12-01 - 18:54 BryanCaron  
PNGpng atlas-reliability-t2-20091201.png r1 manage 163.5 K 2009-12-01 - 18:54 BryanCaron  
PNGpng ops-availability-t2-20091201.png r1 manage 280.1 K 2009-12-01 - 18:53 BryanCaron  
PNGpng ops-reliability-t2-20091201.png r1 manage 170.7 K 2009-12-01 - 18:53 BryanCaron  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2009-12-01 - AsokaDeSilva
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback