TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091124 (2009-12-01, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2009-11-24 [Incomplete]

General News

  • ADC News [20091117]
    • new project tags for data-taking in 2009 to be added at T1 and T2s [20091117]
      • data09_1beam, data09_900GeV and data09_2TeV to go along with the data09_cos already defined in ATLASDATADISK
    • site downtime information from ADC Operations [20091112]
      • reminder that we are in data-taking mode!
      • refrain from any upgrades apart from absolutely necessary (security) patches and to please keep those as short as possible
      • anticipate a 2 to 3 week period starting around 20091218 (LHC machine standby) where other maintenance and upgrades can be performed
    • UAT Postmortem this week [20091119] http://indico.cern.ch/conferenceDisplay.py?confId=74076
    • new MC AOD dataset distribution policy discussed within ADC and ICB so as to take into account operational constraints (among others) [20091117]
      • T2s are for user analysis but should be considered if:
        • at least 100 TBs
        • pass HammerCloud validation criteria for efficiency and throughput: 90% efficiency, 150M events per day throughput at the site
        • exceed reliability criteria

  • vulnerability announced to sites via EGEE broadcast [20091105]
    • please take indicated steps as soon as possible and announce to list (downtime and completion)

  • critical and important java security errata announced for SL [20091116]

  • information system reminders [20091117]
    • sites should please verify their published logicalCPU, physicalCPU and kSI2K values from their site BDIIs.
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU and CA-ALBERTA-WESTGRID-T2.

  • Squid deployment [20091117]
    • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning: all sites good (WARN is ok).
      • ALBERTA-LCG2 WARN (local squid server offline since 20091108)
      • SFU-LCG2 WARN (for both CEs; Victoria squid is the backup but not online)
      • TORONTO-LCG2 WARN (local squid not online; primary backup points to ALBERTA squid but fails due to network routing issue; secondary backup to SFU OK)
      • TRIUMF-LCG2 OK (for both CEs; local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 WARN (local squid not online; backup points to SFU squid)
    • ReadReal test passes at all sites; manual fix to PFC at TRIUMF and SFU pending release of update to dq2-tools

  • dCache REMINDERS

    • golden release of dCache announced: version 1.9.5 (see dCache user-forum announcement)
      • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement)
      • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
        • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • WestGrid-Alberta: all nodes at 1.9.3-3
        • TRIUMF: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
        • VICTORIA: 1.8.0.12p5
        • ALBERTA: 1.8.0.12p6

  • REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

OPS VO

  • ops-availability-20091117.png:
    ops-availability-20091117.png

  • ops-reliability-20091117.png:
    ops-reliability-20091117.png

ATLAS VO

  • atlas-availability-20091117.png:
    atlas-availability-20091117.png

  • atlas-reliability-20091117.png:
    atlas-reliability-20091117.png

APEL Accounting

APEL Publishing Status

  • apel-status-20091117.png:
    apel-status-20091117.png

APEL Accounting Results

  • apel-statistics-20091117.png:
    apel-statistics-20091117.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • single SAM test failures for CE-sft-lcg-rm and SRMv2-put reported [20091117]
    • subsequent tests are successful

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • installation status: WLCG on WestGrid-Alberta cluster [20091110]
    • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
      • 3 day period of SAM tests started by CERN ROC following update to GOCDB ("candidate" to "uncertified") [20091112]
        • posted results ended prior to 3 day period [20091113] with zero errors and have prompted CERN ROC for confirmation of certification or restart of new 3 day period
        • began copyto maintenance tasks of ALBERTA-LCG site dCache to relocate contents of the borrowed WestGrid-UA dCache pools [20091113]
        • Short-Term To-Do List:
          • complete copyto maintenance tasks for 4 WestGrid-UA pools (3 of 4 completed as of this morning)
          • need to modify dCache configuration to have multihomed pools to separate internal and external traffic; instructions kindly provided by Sergey
          • update dCache to 1.9.5-18 (golden release)
          • request update of ToA to include CA-ALBERTA-WESTGRID-T2 once confirmation is received from CERN ROC of the site certification

Outstanding Issues

  • few minor issues pending correction on existing T2 facility:
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • scheduled downtime [20091112 until 20091114] https://goc.gridops.org/downtime?id=48755482
    • power breakers upgrade
      • few worker nodes lost - cannot be powered on
    • DDN and Lustre work
      • Lustre updated to 1.8.1.1 + patches - extended downtime required awaiting outside expert intervention over the weekend
      • required to relocate release data from one shelf to another - problems with hard-links not surviving procedure; KV being run of all base releases to confirm installation
    • dCache upgrades
      • update to 1.9.5-8 attempted but resulted in 'no space available errors'; rolled back to 1.9.2-8
    • Snowpatch was available but no site SE services.

  • SL5 (bugaboo) queue marked in 'test' within Panda pending follow-up post scheduled downtime
    • additional separate analysis queue (ANALY_SFU_bugaboo) requested of Panda team and created

Outstanding Issues

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel required

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • SCRATCHDISK became full [20091112]
    • increased size of SCRATCHDISK by approx. 0.75 TB
    • Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Outstanding Issues

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • CA-SCINET-T2 installation:
    • new site certification by CERN ROC: https://gus.fzk.de/ws/ticket_info.php?ticket=49823
    • dCache 1.9.4 now on pools but finding some configuration issues
    • dCache admin node being updated to CentOS 53
    • DDN firmware update debugging from vendor during downtime
    • designated node for squid cache installation

Outstanding Issues

  • cleanup of suspended transfers from 20091031 still pending on SE

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • unscheduled downtime planned for 2 hours [20091117]
    • oracle security update

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

  • attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
    • pilots (without payload) successfully running
    • next target is successful SAM tests
    • SE functionality still to be tested on current dCache instance with two pools

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 23795 207 313514506 1629137 99.1% 99.5%
SFU-LCG2 3537 56 74619665 747209 98.4% 99%
VICTORIA-LCG2 1305 21 54123439 353640 98.4% 99.4%
ALBERTA-LCG2 1102 142 29189073 3923505 88.6% 88.2%
TORONTO-LCG2 978 54 56711104 295257 94.8% 99.5%
total 30717 480 528157787 6948748 98.5% 98.7%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor:

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
TRF_UNKNOWN ,1,91
TRFERROR,2,51
EXEPANDA_DQ2_STAGEIN,3,23
TRF_SEGFAULT ,4,16
EXEPANDA_JOBKILL_BYPILOT,5,8
EXEPANDA_DQ2PUT_FILECOPYERROR,6,6
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,7,5
EXEPANDA_JOBKILL_SIGTERM,8,3
EXEPANDA_JOBDISPATCHER_HEARTBEAT,9,3
EXEPANDA_PUT_GLOBUSSYSTEMERROR,10,2
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,11,1


Alberta:

name,rank,value
TRFERROR,1,138
EXEPANDA_JOBKILL_BYPILOT,2,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,3,1
TRF_SEGFAULT ,4,1


SFU:

name,rank,value
TRF_SEGFAULT ,1,29
TRF_UNKNOWN ,2,17
EXEPANDA_JOBKILL_BYPILOT,3,5
EXEPANDA_DQ2PUT_FILECOPYERROR,4,3
TRFERROR,5,2
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,1


Toronto:

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,49
TRF_SEGFAULT ,2,2
TRFERROR,3,2
EXEPANDA_JOBKILL_BYPILOT,4,1


Victoria:

name,rank,value
TRF_SEGFAULT ,1,10
EXEPANDA_DQ2_STAGEIN,2,8
EXEPANDA_JOBKILL_BYPILOT,3,2
TRF_UNKNOWN ,4,1

Notes

  • How we fared to other ATLAS sites ...
site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
FZK 203666 10279 5039086798 135388323 95.2% 97.4%
BNL 179629 12216 3492824758 202689955 93.6% 94.5%
LYON 162772 6053 3745598459 102292352 96.4% 97.3%
RAL 127984 15235 3897816130 179298603 89.4% 95.6%
SARA 66726 3933 1857311277 68688510 94.4% 96.4%
NDGF 66783 1294 1938445440 29605500 98.1% 98.5%
PIC 43699 2544 875230515 100712505 94.5% 89.7%
CNAF 21145 15436 859470443 73392048 57.8% 92.1%
TRIUMF 30717 480 526304163 6948748 98.5% 98.7%
None 23108 3102 563588496 20193131 88.2% 96.5%
ASGC 24547 538 753150139 27041625 97.9% 96.5%
CERN 5864 1405 39450724 11162428 80.7% 77.9%
total 956640 72515 2.3588277342e+10 957413728 93% 96.1%
  • Yuri's report:
    • active tasks from PandaDB: 204
    • by cloud: CA:8 CERN:1 DE:20 ES:12 FR:7 IT:5 ND:27 NL:4 TW:1 UK:14 US:105

  • My Notes:
    • ce2@triumf switched to SL5 cluster on 23 Nov.
    • Toronto set offline (declared downtime) 24 Nov. See announcement and GGUS 53605

-- BryanCaron - 24 Nov 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 93.0 K 2009-11-24 - 19:23 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 1282.1 K 2009-11-24 - 19:23 AsokaDeSilva  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2009-12-01 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback