TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091117 (2009-11-17, AsokaDeSilva) EditAttach

Tier1 and Tier-2 Weekly Report 2009-11-17

General News

  • ADC News [20091117]
    • new project tags for data-taking in 2009 to be added at T1 and T2s [20091117]
      • data09_1beam, data09_900GeV and data09_2TeV to go along with the data09_cos already defined in ATLASDATADISK
    • site downtime information from ADC Operations [20091112]
      • reminder that we are in data-taking mode!
      • refrain from any upgrades apart from absolutely necessary (security) patches and to please keep those as short as possible
      • anticipate a 2 to 3 week period starting around 20091218 (LHC machine standby) where other maintenance and upgrades can be performed
    • UAT Postmortem this week [20091119] http://indico.cern.ch/conferenceDisplay.py?confId=74076
    • new MC AOD dataset distribution policy discussed within ADC and ICB so as to take into account operational constraints (among others) [20091117]
      • T2s are for user analysis but should be considered if:
        • at least 100 TBs
        • pass HammerCloud validation criteria for efficiency and throughput: 90% efficiency, 150M events per day throughput at the site
        • exceed reliability criteria

  • vulnerability announced to sites via EGEE broadcast [20091105]
    • please take indicated steps as soon as possible and announce to list (downtime and completion)

  • critical and important java security errata announced for SL [20091116]

  • information system reminders [20091117]
    • sites should please verify their published logicalCPU, physicalCPU and kSI2K values from their site BDIIs.
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU and CA-ALBERTA-WESTGRID-T2.

  • Squid deployment [20091117]
    • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
    • currently status as of this morning: all sites good (WARN is ok).
      • ALBERTA-LCG2 WARN (local squid server offline since 20091108)
      • SFU-LCG2 WARN (for both CEs; Victoria squid is the backup but not online)
      • TORONTO-LCG2 WARN (local squid not online; primary backup points to ALBERTA squid but fails due to network routing issue; secondary backup to SFU OK)
      • TRIUMF-LCG2 OK (for both CEs; local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 WARN (local squid not online; backup points to SFU squid)
    • ReadReal test passes at all sites; manual fix to PFC at TRIUMF and SFU pending release of update to dq2-tools

  • dCache REMINDERS

    • golden release of dCache announced: version 1.9.5 (see dCache user-forum announcement)
      • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement)
      • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)

    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • SFU: all nodes at 1.9.2-8 and client 1.9.2-4
        • Toronto: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • WestGrid-Alberta: all nodes at 1.9.3-3
        • TRIUMF: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
        • VICTORIA: 1.8.0.12p5
        • ALBERTA: 1.8.0.12p6

  • REMINDER #4: site and LFC clean-up needed of files and (empty) directories associated with ccrc08 [2009-06-09]

SAM Results

OPS VO

  • ops-availability-20091117.png:
    ops-availability-20091117.png

  • ops-reliability-20091117.png:
    ops-reliability-20091117.png

ATLAS VO

  • atlas-availability-20091117.png:
    atlas-availability-20091117.png

  • atlas-reliability-20091117.png:
    atlas-reliability-20091117.png

APEL Accounting

APEL Publishing Status

  • apel-status-20091117.png:
    apel-status-20091117.png

APEL Accounting Results

  • apel-statistics-20091117.png:
    apel-statistics-20091117.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • single SAM test failures for CE-sft-lcg-rm and SRMv2-put reported [20091117]
    • subsequent tests are successful

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • installation status: WLCG on WestGrid-Alberta cluster [20091110]
    • site certification by CERN ROC started [20090921] https://gus.fzk.de/ws/ticket_info.php?ticket=51713
      • 3 day period of SAM tests started by CERN ROC following update to GOCDB ("candidate" to "uncertified") [20091112]
        • posted results ended prior to 3 day period [20091113] with zero errors and have prompted CERN ROC for confirmation of certification or restart of new 3 day period
        • began copyto maintenance tasks of ALBERTA-LCG site dCache to relocate contents of the borrowed WestGrid-UA dCache pools [20091113]
        • Short-Term To-Do List:
          • complete copyto maintenance tasks for 4 WestGrid-UA pools (3 of 4 completed as of this morning)
          • need to modify dCache configuration to have multihomed pools to separate internal and external traffic; instructions kindly provided by Sergey
          • update dCache to 1.9.5-18 (golden release)
          • request update of ToA to include CA-ALBERTA-WESTGRID-T2 once confirmation is received from CERN ROC of the site certification

Outstanding Issues

  • few minor issues pending correction on existing T2 facility:
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • scheduled downtime [20091112 until 20091114] https://goc.gridops.org/downtime?id=48755482
    • power breakers upgrade
      • few worker nodes lost - cannot be powered on
    • DDN and Lustre work
      • Lustre updated to 1.8.1.1 + patches - extended downtime required awaiting outside expert intervention over the weekend
      • required to relocate release data from one shelf to another - problems with hard-links not surviving procedure; KV being run of all base releases to confirm installation
    • dCache upgrades
      • update to 1.9.5-8 attempted but resulted in 'no space available errors'; rolled back to 1.9.2-8
    • Snowpatch was available but no site SE services.

  • SL5 (bugaboo) queue marked in 'test' within Panda pending follow-up post scheduled downtime
    • additional separate analysis queue (ANALY_SFU_bugaboo) requested of Panda team and created

Outstanding Issues

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel required

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • SCRATCHDISK became full [20091112]
    • increased size of SCRATCHDISK by approx. 0.75 TB
    • Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Outstanding Issues

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues

  • CA-SCINET-T2 installation:
    • new site certification by CERN ROC: https://gus.fzk.de/ws/ticket_info.php?ticket=49823
    • dCache 1.9.4 now on pools but finding some configuration issues
    • dCache admin node being updated to CentOS 53
    • DDN firmware update debugging from vendor during downtime
    • designated node for squid cache installation

Outstanding Issues

  • cleanup of suspended transfers from 20091031 still pending on SE

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • unscheduled downtime planned for 2 hours [20091117]
    • oracle security update

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

  • attempts to restore the McGill site to up-and-running state have resumed [2009-03-25]
    • pilots (without payload) successfully running
    • next target is successful SAM tests
    • SE functionality still to be tested on current dCache instance with two pools

Production

Summary

site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
TRIUMF-LCG2 13310 186 313926888 1260875 98.6% 99.6%
VICTORIA-LCG2 1002 67 56533899 327010 93.7% 99.4%
ALBERTA-LCG2 977 48 32769391 151597 95.3% 99.5%
TORONTO-LCG2 740 4 58986281 114277 99.5% 99.8%
SFU-LCG2 515 98 19686792 4614391 84% 81%
total 16544 403 481903251 6468150 97.6% 98.7%

DashboardSummary.pdf

Failed Jobs

From Panda Monitor: (This is for all of ATLAS - having difficulty parsing for only CA today.)

FailedJobs.pdf

From Dashboard:

You can find the meanings of the transform error codes on this link.

TRIUMF:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,170
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,2,10
EXEPANDA_JOBKILL_BYPILOT,3,9
EXEPANDA_DQ2_STAGEIN,4,3
EXEPANDA_JOBKILL_SIGTERM,5,2
TRF_UNKNOWN ,6,1
TRF_SEGFAULT ,7,1
EXEPANDA_DQ2PUT_FILECOPYERROR,8,1
TRFERROR,9,1


Albetra:

name,rank,value
EXEPANDA_GET_REPLICANOTFOUND,1,38
EXEPANDA_DQ2_SERVERERROR,2,20
TRFERROR,3,12
EXEPANDA_JOBEXPIRED_SIXDAYS,4,4
TRF_SEGFAULT ,5,1


SFU:

name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,97
EXEPANDA_JOBKILL_BYPILOT,2,1


Toronto:

name,rank,value
TRF_SEGFAULT ,1,2
TRFERROR,2,1
EXEPANDA_DQ2_STAGEIN,3,1


Victoria:

name,rank,value
EXEPANDA_JOBEXPIRED_SIXDAYS,1,42
EXEPANDA_GET_REPLICANOTFOUND,2,21
EXEPANDA_JOBKILL_BYPILOT,3,2
TRFERROR,4,1
EXEPANDA_JOBDISPATCHER_NOREPLYTOSENTJOB,5,1
EXEPANDA_JOBDISPATCHER_HEARTBEAT,6,1

Notes

  • How we fared to other ATLAS sites ...
site success failure success (walltime) failure (walltime) efficiency efficiency (walltime)
FZK 176898 4875 4184539035 63712831 97.3% 98.5%
BNL 173253 6124 3178577287 88251630 96.6% 97.3%
LYON 128872 6076 3185353966 227616118 95.5% 93.3%
RAL 96300 11106 3250661026 109643419 89.7% 96.7%
SARA 71478 4120 1772986154 75204960 94.6% 95.9%
NDGF 62741 1112 1930832520 22784478 98.3% 98.8%
PIC 41900 5825 870760743 123108266 87.8% 87.6%
ASGC 26270 794 823167130 22590587 97.1% 97.3%
None 20666 3231 508374546 29840369 86.5% 94.5%
CNAF 16870 3655 579630258 48169043 82.2% 92.3%
TRIUMF 16544 403 481903251 6468150 97.6% 98.7%
CERN 9 612 12018 3607790 1.4% 0.3%
total 831801 47933 2.0766797934e+10 820997641 94.6% 96.2%

  • Yuri's notes:
    • active tasks from PandaDB: 103
    • by cloud: CA:13 CERN:7 DE:11 ES:10 FR:7 IT:6 ND:9 NL:11 TW:1 UK:8 US:20
    • Fri Nov.13.
      • + CA/SFU 100 FT f./CONN._ER./SRM.GGUS 53256 solved: DT completed.
    • Mon Nov.16.
      • - CA/TRIUMF DATATAPE FT f./SRMV2STAGER err.GGUS 53308 in progress: errors are due to CERN SRM instabilities.GGUS rerouted.Elog 6963.
  • My notes:
    • SFU downtime (drained queues 10 Nov., snowpatch online on 17 Nov. bugaboo still offline.)

-- BryanCaron - 17 Nov 2009

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DashboardSummary.pdf r1 manage 94.7 K 2009-11-17 - 19:36 AsokaDeSilva  
PDFpdf FailedJobs.pdf r1 manage 1214.0 K 2009-11-17 - 19:37 AsokaDeSilva  
PNGpng apel-statistics-20091117.png r1 manage 132.0 K 2009-11-17 - 15:18 BryanCaron  
PNGpng apel-status-20091117.png r1 manage 22.0 K 2009-11-17 - 15:19 BryanCaron  
PNGpng atlas-availability-20091117.png r1 manage 163.1 K 2009-11-17 - 15:17 BryanCaron  
PNGpng atlas-reliability-20091117.png r1 manage 110.5 K 2009-11-17 - 15:18 BryanCaron  
PNGpng ops-availability-20091117.png r1 manage 157.2 K 2009-11-17 - 15:16 BryanCaron  
PNGpng ops-reliability-20091117.png r1 manage 104.8 K 2009-11-17 - 15:17 BryanCaron  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2009-11-17 - AsokaDeSilva
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback