Tier1 and Tier-2 Weekly Report 2010-07-27

General News

• ADC News [20100727]
  • This Week
    • ADC Weekly [20100726]
      • http://indico.cern.ch/conferenceDisplay.py?confId=102340
      • highlights: reprocessing tests from tape (T1s)
    • WLCG Management Board [20100727]
      • http://indico.cern.ch/conferenceDisplay.py?confId=82000

• CA Cloud Squad Actions [20100727]
  • new schedconfig (and general site configuration) mechanism announced by schedconfig team for use by cloud squads [20100723]
    • will test for updating site queue values for maxinputsize given the recent fix for pilotController
  • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
    • context is the SFU MCDISK blacklisting as discussed in http://savannah.cern.ch/support/?115809
  • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
  • ADC Savannah Site Exclusion entries created by CA cloud squad:
    • TRIUMF downtimes (FTS and dCache)
      • http://savannah.cern.ch/support/?115749
      • http://savannah.cern.ch/support/?115750
  • panda queue operations:
    • Alberta-WG1 analysis queue marked online (Cloud Squad) and validation requested of site production queue (ADCoS) following return from scheduled downtime [20100720]
    • Victoria-WG production queue validated and marked online by ADCoS following request of CA cloud squad [20100722]
      • site analysis queue already online
    • ANALY_ALBERTA-WG1 marked online by ADCoS shifter following increase of SCRATCHDISK area size [20100722]
      • ADCoS shifter marked queue online without testing; direct testing by CA cloud squad was successful
    • TRIUMF production queues tested and marked online by ADCoS shifter following request of CA cloud squad [20100722]
      • initial ADCoS shifter tests misconfigured and failed
      • analysis queue tested and online by CA cloud squad
    • Australia-ATLAS queues offlined and restarted by ADCoS shifter during site scheduled downtime [20100722]
      • reminder sent to ADCoS to include CA cloud squad in Site Exclusion Savannah notifications for all CA cloud sites (including Australia-ATLAS)
    • CA-SCINET-T2 offlined and restarted by ADCoS shifter during site scheduled downtime [20100723]

• • Open Issues and Ongoing Discussions:
    • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
      • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
        • resolved by changing WN client NFS mounts to v3 from original NFSv4
    • ROOTSYS truncated at 127 characters issue [20100608]
      • http://savannah.cern.ch/bugs/?68546 (open)
      • currently affecting Victoria (Tier3) and CA-SCINET-T2
      • varying character lengths below and above 127 characters show variable success and failure
      • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
        • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
    • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • reminder message posted to the Dashboard Savannah entry for this problem https://savannah.cern.ch/bugs/index.php?64507 [20100518] (updated 20100713)
    • reviewing draft Squad How-To and providing feedback [20100407]
      • also contains useful information for sites
    • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
    • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
      • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
    • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • Savannah http://savannah.cern.ch/bugs/?62993 (ticket closed 20100301 but discussion ongoing within Panda team regarding which BDII variables to query)
  • Outstanding Actions to Do:
    • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
      • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

• ROC_Canada News [20100727]
  • current shift schedule (until July 30): https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ROCShifts
  • Operations Tips: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ROCOperationTips

• Squid Deployment [20100727]
  • CA cloud Frontier and Squid deployment information (versions and network information)
  • SAM tests: (link1), (link2): currently status as of this morning (WARN is ok) [20100720]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required; backup to SFU squid passes)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2; backup to SFU squid)
    • CA-VICTORIA-WESTGRID-T2 OK (both for local at VICTORIA-LCG2 and backup to SFU squid)
    • SFU-LCG2 OK (both CEs; local squid is OK; backup squid at VICTORIA-LCG2 OK)
    • VICTORIA-LCG2 OK (OK for local squid and connection to SFU squid)
    • ATLAS_AUSTRALIA - OK (local squid OK; backup to VICTORIA-LCG2 OK)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)

• Site Action Items:
  • CA-ALBERTA-WESTGRID-T2
    • reinstall local squid server

SAM Results

• ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: investigate cause of lower measure of availability and reliability for May 2010

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • http://savannah.cern.ch/bugs/?68656 DDM Operations: SCRATCHDISK shows DQ2 larger than srm_total on DQ2 accounting (20100610) (in progress)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59747 gstat 'check-se' problem detected at CA-ALBERTA-WESTGRID-T2 (20100705) (on hold)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• CA-ALBERTA-WESTGRID-T2 site status [20100720]
  • scheduled downtime completed 2010-07-16 to 2010-07-19 registered in GOCDB for scheduled power interruption to main campus server room which hosts the Tier-2 resources

Outstanding Issues

• CA-ALBERTA-WESTGRID-T2 site status [20100511]
  • DDM: discrepancy between SRM and DQ2 usage (see above savannah)
    • tests on postgres dump of dCache databases pending
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
  • network cabling maintenance planned for lighpath connection [postponed from 20100319]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• westgrid site router ACLs updated to enable outbound connectivity to SFU and local squids [20100714]
• site squid ACLs update to accept connections from Australia-ATLAS, and corrected for SFU-LCG2 [20100715]

Outstanding Issues

• VICTORIA-LCG2 SCRATCHDISK became full [20091112]
  • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60283 NAGIOS org.sam.SRM-Put-/ops/Role=lcgadmin failed on lcg-se1.scinet.utoronto.ca@CA-SCINET-T2 (20100721) (solved 20100721)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60368 DESTINATION errors during transfers from TRIUMF-LCG2 to CA-SCINET-T2 (20100722) (verified 20100725)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60134 Jobs at CA-SCINET-T2 failing with "Get error: dccp get was timed out" (20100715) (solved 20100722)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59744 gstat 'check-ce' and 'check-se' problem detected at CA-SCINET-T2 (20100706) (assigned)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=58997 Many jobs Get error: Failed to get LFC replicas at CA/CA-SCINET-T2 (2010-06-13) (solved 2010-06-13)
    • re-opened and marked unsolved (20100709)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=58116 LFC error at CA/CA-SCINET-T2 (20100510) (unsolved 20100709)
  • http://savannah.cern.ch/bugs/?65337 CA-SCINET-T2 pilot: Job killed by signal 15 [20100405](open)
• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• TORONTO-LCG2 site
  • site offline to ATLAS; currently suspended by CA_ROC in GOCDB
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53830 TORONTO-LCG2 update to lcg-CE (20091203) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53605 TORONTO-LCG2 lcg_cp timeout (20091124) [on hold]

• scheduled downtime for CA-SCINET-T2 [20100723] for 24 hours
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19367&grid_id=0

Outstanding Issues

• pilot issues [20100511]
  • significant number of immediately failing pilots for both TRIUMF and LMU factories
  • need to logon to WN directly and run the wrapper script by hand

• dCache error investigations
  • Occassional "There is no file with pnfsid=.." errors (20100430)
    • under discussion with dCache experts at TRIUMF and elsewhere
  • dCache PutUseSpaceCallbacks errors (20100504)

• large number of LFC related errors (20100404)
  • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

• SciNet-TRIUMF lightpath network use and optimization [20100413]
  • BGP established on Force 10 router for link to TRIUMF [20100407]
    • Information Request: is a failover test to the ORAN planned? A: Not yet.

• observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
  • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

• discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59748 gstat 'check-se' problem detected at TRIUMF-LCG2 (20100705) (on hold 20100706)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• scheduled downtimes last week
  • FTS oracle RAC migration [20100719]
    • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19158&grid_id=0
  • dCache outage for DDN and drive firmware upgrade [20100721]
    • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19237&grid_id=0

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60344 AUSTRALIA-ATLAS_SCRATCHDISK failed transfers (20100721) (verified 20100722)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60488 ROC_Asia/Pacific/Australia-ATLAS - availability/reliability statistics for June 2010 (20100726) (solved 20100727)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60172 No Nagios test results for agh2.atlas.unimelb.edu.au (assigned 20100716)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• scheduled downtimes this last week for e1000 driver fix and other firmware upgrades [12 hours]
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19370&grid_id=0
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19371&grid_id=0

Outstanding Issues

• identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault
  • causing issues with regards to release 15.5.1 testing. A modified setup for pilot users added to profile.d to get around this in the interim
  • bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=563759

McGill

Recent Issues

• the technical information and requirements in the issued RFP are available upon request; please contact Bryan who has a copy forwarded from Simon [20100720]

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

• Within the CA Cloud:
  

• Within the None Cloud (see https://savannah.cern.ch/bugs/index.php?64507)
  

• Within the CA Cloud (Panglia View):
  

• How we fared to other ATLAS sites during the past week:
  

• Alternate Views (See link for last 3 days only):
  

• CA Cloud (last 3 days):
  

Failed Jobs

• From Panda Monitor (last week):
  

• From Dashboard (last week):

• • TRIUMF-LCG2

name,rank,value
EXEPANDA_Release_MISSING,1,7
EXEPANDA_JOBKILL_BYPILOT,2,3
EXEPANDA_DQ2_STAGEIN,3,1

• • VICTORIA-LCG2

name,rank,value
EXEPANDA_JOBKILL_BYPILOT,1,2

You can find the meanings of the transform error codes on this link.

• My Notes:
  • information missing in dashboard (production) still for CA-SCINET-T2.
    • 3rd reminder sent to Dashboard Savannah (20100615) and dashboard-support@cern.ch
    • previous reminder: sent to Dashboard Savannah (20100518)
      • https://savannah.cern.ch/bugs/index.php?64507

• Yuri's notes: Not Available this week (E. Korolkova)
  • week (June 20 - July 26)

2) Wed Jul.21
+ AUSTRALIA-ATLAS GGUS:60344
  Transfer timeouts. Disk server rebooted. Solved    

3) Thu Jul.22
+ TRIUMF - scheduled downtime (dCache) ~6h.

• Stats:
  • See Key for what the headers mean

-- BryanCaron - 21 Jul 2010