TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20091222 (2009-12-22, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2009-12-22 [In Progress]

General News

* ADC News [20091222]

* procedural reminder for sites in regards to scheduled/unscheduled downtimes and panda queues [20091215]

  • all requests for site downtime and restart with regards to panda queues (analysis and production) should go through cloud contacts (Asoka, Bryan, Di)
  • cloud contacts with then contact or notify ADC operations in regards to actions taken for panda queues
  • more detailed procedural info to be circulated

* availability of ADC Cloud support during Christmas period (Dec 18 - Jan 3)

  • in general email contact for response to ADC issues over holiday period
  • Bryan, Di, Reda, Joe (Asoka away December 18 to January 3)
  • Tier-2 sites should clarify today their availability and coverage plans
    • Toronto/SciNet
    • Alberta/WestGrid
    • SFU
    • Victoria
    • TRIUMF
    • Australia

* security announcement reminders [20091215]

  • SL5 acpid vulnerability announced to sites via (limited) EGEE broadcast [20091208]
  • critical and important java security errata announced for SL [20091116]

* information system reminders [20091215]

  • sites will be contacted in the next week if their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities
    • in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
      • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

* CA Cloud Squad Actions [20091208]

  • corrupt file on TRIUMF-LCG2_MCDISK [20091201]
  • mc08*merge.AOD* dataset subscription for input to GangaRobot submissions to CA-ALBERTA-LCG2 [20091214]

* Squid deployment [20091208]

  • tested in SAM for ATLAS: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=CERN&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
  • currently status as of this morning (WARN is ok)
    • ALBERTA-LCG2 WARN (local squid server offline since 20091108)
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • SFU-LCG2 OK (for both CEs; Victoria squid is the backup but not online)
    • TORONTO-LCG2 WARN (no local squid; primary backup points to ALBERTA squid but fails due to network routing issue + squid offline; secondary backup to SFU OK)
    • TRIUMF-LCG2 WARN (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (local squid access error; backup points to SFU squid)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

* update to golden release of dCache announced last week (1.9.5-10) (see dCache user-forum announcement)

  • recall previously announced end-of-life for dCache versions < 1.9.4 (also see dCache user-forum announcement from 20091011)
  • also remember the previously announced security updates to postgresql for releases packaged with SL3/4/5 (i386/x86_64) (link)
    • (OLD) REMINDER #1: long reported SRM subsystem vulnerability in dCache [2009-07-27]
      • see dcache user-forum archives or forwarded email from Reda
      • all production versions affected and require advertised patches [2009-08-04]
        • 1.9.1-11, 1.9.2-11, 1.9.3-3, 1.9.4-2
      • For reference, here is the current installation status:
        • ALBERTA-LCG2: 1.8.0.12p6
        • CA-ALBERTA-WESTGRID-T2: all nodes at 1.9.3-3
        • CA-SCINET-T2: mixture of nodes at 1.9.5-4 and 1.9.5-9
        • SFU-LCG2: all nodes at 1.9.2-8 and client 1.9.2-4
        • TORONTO-LCG2: admin, chimera nodes at 1.9.0-8, srm/gip at 1.9.1-7, pools at 1.9.1-7
        • TRIUMF-LCG2: all nodes at 1.9.3-3, SRM headnode at 1.9.3-4 (bug fix).
      • VICTORIA-LCG2: 1.8.0.12p5

SAM Results

OPS VO

ATLAS VO

APEL Accounting

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS
    • downsizing of site dCache completed [20091214]
      • single occurrence of SAM errors for SE actions prior to removal of ATLAS space tokens [20091214]

  • CA-ALBERTA-WESTGRID-T2 site installation status [20091215]
    • infrastructure setting and debugging
      • grid certificate CRLs correctly updating; CRL update info missing from .r0 files is normal for latest release of fetch-crl
      • intermittent job submission (JS) errors from SAM tests found to be due to misconfigured lcg-voms.cern.ch.lsc coming from yaim configuration step (20091203)
      • savannah ticket open in regards to migrating current ALBERTA-LCG2 site storage to CA-ALBERTA-WESTGRID-T2 site
        • migration complete with all storage now part of CA-ALBERTA-WESTGRID-T2 and LFC cleaned of ALBERTA-LCG2 entries
      • update to host resolution to dCache pools from WNs to correct failing gridftp copies observed during ATLAS software installations [20091211]
      • added 3 of the 4 x 5.5 TB pools from ALBERTA-LCG2 site [20091213]
      • actions on hold until LHC winter technical stop
        • update to dCache 1.9.5-10 (golden release)
        • update setup to multi-homed pools to separate internal and external traffic (instructions from Sergey)
      • savannah bug filed requesting site be added to Tier-2 availability and reliability display pages
    • ATLAS grid commissioning
      • DDM integration complete
        • missing entry in data distribution dashboard (action on DDM ops)
      • ATLAS software installation proceeding and almost complete following gridftp and host name resolution fixes
      • site production and analysis queues created in panda to be tested when release installs confirmed

Outstanding Issues

  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected

  • few minor issues pending correction on existing T2 facility (ALBERTA-LCG2):
    • full decoupling of WLCG accounts from cluster NIS to be done (see problem reported 2009-04-14)
    • pnfsd.log to be added to log-rotate [2009-03-31]
    • few WNs to be debugged with respect to missing stderr/stdout
    • still need to add 2nd Gigabit Ethernet nic to dCache pools for internal network access from WNs to avoid NAT

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • dCache downtime of approx. 10 mins for restart (20091209)
    • correct dCache headnode configuration which resulted in all transfers going through 2 NAT servers

  • status of bugaboo (SL5) commissioning status within atlas
    • ANALY_bugaboo and production queues set online (20091211)

  • issues seen with atlas software installation (14.2.25) area reference
    • checked on all WNs; under further investigation

  • SFU-LCG2_MCDISK being cleaned by DDM Ops to help resolve discrepancy between DQ2 and SRM usage statistics (20091215)

Outstanding Issues

  • apel publishing results for September and October 2009 [20091110]
    • low values published but not expected when comparing with ATLAS production dashboard for the same period
    • GlueHostBenchmarkSF00 set to zero for both clusters in site BDII - fixed by site - reparsing and republishing to apel performed but still low values reported

  • SFU storage access protocol sanity check [20091022]
    • missing dcap - Status of this?

  • snowpatch - fairshare status wrt different pilots not running [20090918]
    • all are members of the overall atlas group - sub-group fairshare (?)
    • NOTE: updated fairshare values to be implemented pending info to arrive from Bryan based upon NRAC allocations for ATLAS [20091103]

  • apel publishing [20090825]
    • apel publishing information missing for October (see above plots) [20091013]
    • publishing status for dteam appears empty since mid-July and coincides with appearance of the atcan group in the status graphs (see above)
    • confirmed to coincide with change in VOMS group and role for direct tests by Sergey
    • low-priority follow-up needed given that dteam jobs are observed regularly and consistently at other sites (Alberta for example)

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • scheduled downtime for CE maintenance and upgrade [20091209]
    • downtime entered into GOCDB but not published via site downtime subscriptions (either atlas operations or atlas canada tier2 list)
    • site panda queues (analy_ and production) set to online 20091210

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently on unscheduled downtime in GOCDB until 20091216

Outstanding Issues

  • Squid/Frontier access tests via ReadReal [20091028]
    • strange network path issue seen when attempting to reach Alberta squid server, attempting to reach via LightPath but fails
    • added secondary backup of SFU squid into ToA

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • dccp get timeout errors reported for analysis jobs (20091209)

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • unscheduled downtime due to disk server issues (20091209)
    • resulted in transfer errors arising at TRIUMF dCache as FTS may have cancelled transfers when errors occurred
    • resolved and transfers confirmed successful (20091214)

  • ANALY_ queue set to test status (20091210)
    • test jobs fails to build; base release 15.5.1 reinstalled but error still present ; under investigation
    • GangaRobot already running successfully at site but still blacklisted (why?)

Outstanding Issues

McGill

Recent Issues

  • status?

Outstanding Issues

Production

Summary

Failed Jobs

From Panda Monitor:

From Dashboard:

You can find the meanings of the transform error codes on this link.

Notes

  • How we fared to other ATLAS sites ...

  • My Notes:

  • ADCoS (Yuri's notes):

  • Panda Stats:

-- BryanCaron - 22 Dec 2009

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2009-12-22 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback