TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100810 (2010-08-10, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-08-10

General News

  • ADC News [20100810]
    • This Week
      • ADC Weekly [20100809]
        • http://indico.cern.ch/conferenceDisplay.py?confId=103515
        • Highlights:
          • surpassed 1 inverse-picobarn of collected data this past weekend along with various LHC machine records
          • central catalog unavailability last week due to deletion agent upgrade
          • production activity: anticipate low level of activity in remainder of August until preparations complete in about 2 weeks for the G4 reprocessing campaign
          • distributed analysis: brief discussion regarding the need for robust tests and metrics as input to an accurate procedure for auto-blacklisting
      • ADC Operations Daily [20100812]

  • CA Cloud Squad Actions [20100810]
    • DaTRI requests for approval
    • Pending action this week: update values for maxinputsize for SFU-LCG2 (snowpatch and bugaboo) within panda schedconfig [20100810]
    • queue operations
      • CA-ALBERTA-WESTGRID-T2: analysis and production queues set offline (test) [20100809] during unscheduled downtime (power outage)
      • SFU-LCG2: analysis and production in panda set offline (test) [20100807] during unscheduled downtime from expired CE host certificate
        • analysis set online [20100809] by cloud squad with request to ADCoS to test production
        • production set online [20100810] by ADCoS shifters
      • CA-VICTORIA-WESTGRID-T2: analysis offline (test) in panda [20100806], production offline in panda [20100807] due to network issues affecting access to site
        • analysis set online [20100809] by cloud squad with request to ADCoS to test production
        • production set online [20100810] by ADCoS shifters
      • VICTORIA-LCG2: production and analysis offline in panda, blacklisted in WMS [20100804]
    • preparations for HammerCloud test of TRIUMF via Ganga this week to compare direct i/o via dcap and FileStager
    • Open Issues and Ongoing Discussions:
      • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
      • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
      • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
        • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
          • resolved by changing WN client NFS mounts to v3 from original NFSv4 (root cause within NFSv4 still unknown) [OPEN]
      • ROOTSYS truncated at 127 characters issue [20100608]
        • http://savannah.cern.ch/bugs/?68546 (open)
        • currently affecting Victoria (Tier3) and CA-SCINET-T2
        • varying character lengths below and above 127 characters show variable success and failure
        • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
          • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • Squid Deployment [20100803]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [20100720]
      • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required; backup to SFU squid passes)
      • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2; backup to SFU squid)
      • CA-VICTORIA-WESTGRID-T2 OK (both for local at VICTORIA-LCG2 and backup to SFU squid)
      • SFU-LCG2 OK (both CEs; local squid is OK; backup squid at VICTORIA-LCG2 OK)
      • VICTORIA-LCG2 OK (OK for local squid and connection to SFU squid)
      • ATLAS_AUSTRALIA - OK (local squid OK; backup to VICTORIA-LCG2 OK)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)

  • Site Action Items:
    • CA-ALBERTA-WESTGRID-T2
      • reinstall local squid server [DONE 20100809]

SAM Results

  • ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: investigate cause of lower measure of availability and reliability for May 2010

APEL Accounting

  • draft Tier-2 accounting report for July 2010 [20100810]
    • (Link to full report)
    • July 2010 is the last month for which VICTORIA-LCG2 accounting statistics will be included in the MoU committed resource summary; CA-VICTORIA-WESTGRID-T2 numbers from August 2010 onward
accounting-aus-201007.png accounting-ca-201007.png

APEL Publishing Status

apel-status-aus-20100810.png apel-status-ca-20100810.png

APEL Accounting Results

apel-statistics-aus-20100810.png apel-statistics-ca-20100810.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • CA-ALBERTA-WESTGRID-T2 site status [20100810]
    • unscheduled outage due to power outage during severe thunderstorms [20100809]
    • reinstalled squid server via kvm virtual machine and in production [20100809]
      • need to confirm ACLs to allow access from all CA cloud Tier-2 sites
    • dCache SRM space usage discprepancy work-around to be tested this week on dump of postgresql database

Outstanding Issues

  • CA-ALBERTA-WESTGRID-T2 site status [20100511]
    • DDM: discrepancy between SRM and DQ2 usage (see above savannah)
      • tests on postgres dump of dCache databases pending
    • additional open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]
    • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
      • completed [20100809]

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • serious network disruptions to victoria campus [20100806]
    • site PFC installation failures with result of cloud squad marking analysis site offline [20100806]
    • subsequent information system missing; cloud squad marked site offline for production [20100807]
    • returned to analysis online status [20100809] by cloud squad with request to ADCoS shifters for production testing [20100809]
    • see above ggus
    • unscheduled downtime

Outstanding Issues

  • VICTORIA-LCG2 SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full
    • UPDATE: removed from issues list to track. ADC now issues automatic notification to cloud squads of sites/tokens with reduced space to be handled by auto-cleanup

Toronto

Recent Issues

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • the technical information and requirements in the issued RFP are available upon request; please contact Bryan who has a copy forwarded from Simon [20100720]

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
production-dashboard-summary-ca-20100810.png

production-dashboard-non-scinet1-20100810.png production-dashboard-non-scinet2-20100810.png production-dashboard-non-sfu-20100810.png production-dashboard-non-vic-20100810.png

  • Within the CA Cloud (Panglia View):
atlas-production-week-ca-20100810.png atlas-analysis-week-ca-20100810.png

  • How we fared to other ATLAS sites during the past week:
production-dashboard-summary-20100810.png

  • Alternate Views (See link for last 3 days only):
panda-dashboard-summary-all-20100810.png

  • CA Cloud (last 3 days):
panda-dashboard-ca-20100810.png

Failed Jobs

  • From Panda Monitor (last week):
panda-dashboard-errors-header-20100810.png panda-dashboard-errors-aus-20100810.png panda-dashboard-errors-ca-20100810.png panda-dashboard-errors-sfu-20100810.png panda-dashboard-errors-triumf-20100810.png panda-dashboard-errors-victoria-20100810.png

  • From Dashboard (last week):

    • VICTORIA-LCG2 
name,rank,value
TRF_UNKNOWN ,1,1


name,rank,value
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,1,31
TRF_UNKNOWN ,2,13
TRF_SEGFAULT ,3,13
TRFERROR,4,12
EXEPANDA_JOBKILL_BYPILOT,5,4
EXEPANDA_DQ2_STAGEIN,6,3
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,2
EXEPANDA_DQ2PUT_FILECOPYERROR,8,1

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRF_UNKNOWN ,1,2
EXEPANDA_DQ2PUT_FILECOPYERROR,2,2
TRFERROR,3,1

    • Australia-ATLAS 
EXEPANDA_GET_GLOBUSSYSTEMERROR,1,13
EXEPANDA_DQ2PUT_FILECOPYERROR,2,4


name,rank,value
TRF_SEGFAULT ,1,1

You can find the meanings of the transform error codes on this link.

  • Yuri's notes: week (Aug 3 - Aug 9) 

   - active tasks from PandaDB: 107 
      -- distribution by cloud: 
         CA:4 CERN:1 DE:12 ES:3 FR:8 IT:13 ND:6 NL:7 TW:1 UK:33 US:19

  - completed 402,380  managed, group MC production, validation and reprocessing jobs.                                                                                                                                                                 
   - average ~57,483 jobs per day
   - failed 36,658 jobs
   - average efficiency: 
      -- jobs     - 91.7%
      -- walltime - 92.5%

4) Thu Aug.5
- CA/TRIUMF-LCG2 SCRATCHDISK still FT f./file name too long.Savannah
   71030 updated.Rerouting is done for this dataset.

6) Sun Aug.8
- CA/CA-VICTORIA-WESTGRID-T2_SCRATCHDISK->PIC FT f./SOURCE/SEC._ER:SRM
   Authentication failed.GGUS 60975 assigned:network issue resolved,site
   claimed UD,completed Aug.9.

-- BryanCaron - 04 Aug 2010
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng accounting-aus-201007.png r1 manage 31.6 K 2010-08-10 - 16:24 BryanCaron  
PNGpng accounting-ca-201007.png r1 manage 25.7 K 2010-08-10 - 16:24 BryanCaron  
PNGpng apel-statistics-aus-20100810.png r1 manage 88.3 K 2010-08-10 - 16:17 BryanCaron  
PNGpng apel-statistics-ca-20100810.png r1 manage 167.4 K 2010-08-10 - 16:17 BryanCaron  
PNGpng apel-status-aus-20100810.png r1 manage 14.9 K 2010-08-10 - 16:18 BryanCaron  
PNGpng apel-status-ca-20100810.png r1 manage 25.0 K 2010-08-10 - 16:18 BryanCaron  
PNGpng atlas-analysis-week-ca-20100810.png r1 manage 25.4 K 2010-08-10 - 16:32 BryanCaron  
PNGpng atlas-production-week-ca-20100810.png r1 manage 21.8 K 2010-08-10 - 16:32 BryanCaron  
PNGpng panda-dashboard-ca-20100810.png r1 manage 24.9 K 2010-08-10 - 16:49 BryanCaron  
PNGpng panda-dashboard-errors-aus-20100810.png r1 manage 43.3 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-errors-ca-20100810.png r1 manage 191.5 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-errors-header-20100810.png r1 manage 7.2 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-errors-sfu-20100810.png r1 manage 30.1 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-errors-triumf-20100810.png r1 manage 75.5 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-errors-victoria-20100810.png r1 manage 18.6 K 2010-08-10 - 18:31 BryanCaron  
PNGpng panda-dashboard-summary-all-20100810.png r1 manage 128.9 K 2010-08-10 - 16:37 BryanCaron  
PNGpng production-dashboard-non-scinet1-20100810.png r1 manage 9.5 K 2010-08-10 - 18:12 BryanCaron  
PNGpng production-dashboard-non-scinet2-20100810.png r1 manage 18.0 K 2010-08-10 - 18:12 BryanCaron  
PNGpng production-dashboard-non-sfu-20100810.png r1 manage 5.0 K 2010-08-10 - 18:12 BryanCaron  
PNGpng production-dashboard-non-vic-20100810.png r1 manage 11.8 K 2010-08-10 - 18:12 BryanCaron  
PNGpng production-dashboard-summary-20100810.png r1 manage 94.4 K 2010-08-10 - 17:06 BryanCaron  
PNGpng production-dashboard-summary-ca-20100810.png r1 manage 48.0 K 2010-08-10 - 17:06 BryanCaron  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2010-08-10 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback