TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20101026 (2010-10-26, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-10-26

Daily Notes

  • Tue Oct 19
    • start of LHC technical stop (until Fri PM at CERN)
    • CA-ALBERTA-WESTGRID-T2 blacklisted in DDM due to SE problems
      • panda queues tested and set online after fix confirmed
    • patch to version 1.0.2 pyfactory announced and installed at TRIUMF
    • new ATLAS TWiki access rules in place at CERN - active ATLAS membership required to view
      • update 20101025: all ADC info considered non-sensitive so will be exempt from the new policy

  • Wed Oct 20
    • CA-ALBERTA-WESTGRID-T2 unscheduled downtime due to ldap failure (7 hrs)
    • no panda tests to Alberta observed within HammerCloud - corrected by HammerCloud experts
    • problematic release (15.6.10) set to be cleaned for reinstallation at CA-ALBERTA-WESTGRID-T2

  • Thur Oct 21
    • Autumn 2010 Phase 1 reprocessing estimated data volumes announced
    • update to siteTests released
    • unscheduled downtime of a few hours for CA-ALBERTA-WESTGRID-T2 due to ldap server problems

  • Sat Oct 23
    • missing libtermcap 32-bit rpm identified for WNs at various CA Tier-2 sites
      • CA-SCINET-T2, SFU-LCG2, CA-VICTORIA-WESTGRID-T2

  • Mon Oct 25
    • DDM storage space alert for SFU-LCG2_DATADISK (1.54 TB free)
      • auto-clean process started
    • transfer errors to CA-SCINET-T2_DATADISK
    • stuck dbrelease file within HOTDISK space token identified and cleared at CA-ALBERTA-WESTGRID-T2

  • Tue Oct 26
    • SFU-LCG2_DATADISK downtime for writes within DDM - autoclean active
    • scheduled downtime at Australia-ATLAS for DPM server upgrade (4 hours) - queues set to brokeroff by ADCoS
    • GGUS ticket: FTS failures for STAR channels to CA Tier-2s

General News

panda-dashboard-pd2p-ca-20101026.png
    • identified problem with regular HammerCloud test panda submissions being absent from ANALY_ALBERTA-WG1 [20101020]
      • incorrect spelling of sitename in HammerCloud database corrected by admins
    • SFU-LCG2_DATADISK approaching being full [20101025]
    • site WN package dependency update: libtermcap 32-bit required [20101023]
    • Open Issues and Ongoing Discussions:
      • small logfiles going to tape endpoints at T1s [20101018]
        • http://savannah.cern.ch/bugs/?74125
        • logfile destination needs to be explicitly set but functionality to do this is currently not present - to be followed-up within ADC [20101025]
      • temporary increase in size of HOTDISK at TRIUMF from 1 to 1.5 TB prior to central cleanup of unneeded DBRelease versions for T1s [20101013]
      • published site installed resource capacity information to be investigated and confirmed against WLCG information [20100928]
      • CREAM CE deployment preparations
      • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • issue of sites within Cloud None within the production dashboard [20100829]
        • issue raised to ADC operations email list given that there was no ADC Weekly meeting yesterday
        • reply from ADC monitoring coordinator (A. Read) is that this long-standing problem is known and being worked on, but is difficult to fix
        • this problem will be on the agenda for the next ADC monitoring meeting scheduled for next week
        • Previously Reported Information on this topic:
          • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
            • reminder message posted to the Dashboard Savannah entry for this problem https://savannah.cern.ch/bugs/index.php?64507 [20100518] (updated 20101012)
            • news from ADC monitoring coordinator A. Read (20101013):
              • "The cloud issue has proven difficult to deal with in the present production dashboard framework. This month and next we are discussing changes in direction which we hope will make it much more straightforward to deal with the clouds."
      • CA-VICTORIA-WESTGRID-T2 site missing from the ATLAS dashboard SAM Visualization page [20100824]
      • Pending action: update values for maxinputsize for SFU-LCG2 (snowpatch and bugaboo) within panda schedconfig [20100817]
      • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
      • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
      • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
        • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
          • resolved by changing WN client NFS mounts to v3 from original NFSv4 (root cause within NFSv4 still unknown) [OPEN]
      • ROOTSYS truncated at 127 characters issue [20100608]
        • http://savannah.cern.ch/bugs/?68546 (open)
        • currently affecting Victoria (Tier3) and CA-SCINET-T2
        • varying character lengths below and above 127 characters show variable success and failure
        • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
          • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • gLite News [20101026]
    • no news this week

  • Squid Deployment [20101026]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [2011019]
      • CA-ALBERTA-WESTGRID-T2 WARN (both for local squid and backup to SFU)
        • squid server (scache.atlas.ualberta.ca) offline since 20101025
      • CA-SCINET-T2: OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 WARN (both for local squid and backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)

SAM Results

APEL Accounting

APEL Publishing Status

apel-status-aus-20101026.png apel-status-ca-20101026.png

APEL Accounting Results

apel-statistics-aus-20101026.png apel-statistics-ca-20101026.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • cream CE testing
  • dCache testbed installation
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3 - targetting mid-September for upgrade - delayed
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • installed libtermcap and libtermcap-devel on bugaboo and snowpatch cluster WNs [20101024]

  • all site CE host certificates set to expire 20101030
    • request for renewals to be submitted 20101026

Outstanding Issues

  • cream CE installation and testing started [20100929]
    • test installation (blackhole.westgrid.ca) at first which will subsequently be applied to update the lcg-CEs (snowpatch then bugaboo)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • installed libtermcap and some additional rpms listed in HEP_OSlibs_SL5 (but not the metarpm itself) on cluster WNs [20101025]

Outstanding Issues

  • CREAM CE installation to be performed in next few weeks [20101005]
  • Occasionally some jobs arrive without stage-in .gpg files and have to be terminated. Cause unknown (20100928)

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • installed libtermcap 32-bit rpm on cluster WN image [20101024]

Outstanding Issues 

14.2.25 lcg-ce2 (removed; but on lcg-ce1; tag to be fixed)
15.3.1   lcg-ce1 (removed; but on lcg-ce2; tag to be fixed)
15.5.4   lcg-ce2 (failed)
15.6.3.1   lcg-ce2 (failed)
15.6.3.8 lcg-ce2 (pending)
15.6.3.9 to 15.6.3.18 lcg-ce1 (pending)
15.6.10 lcg-ce2 (failed)

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand
    • Q: Is this issue still a problem and under investigation? (BC) [20101026]

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)
    • Q: Are these still an issue and under investigation? (BC) [20101026]

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'
    • UPDATE: Will remove from next report given the above related GGUS tickets have been marked solved (BC) [20101026]

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation
    • UPDATE: Will remove from next report given that we do not expect pilots to be submitted to the site other than from the TRIUMF pyfactory (BC) [20101026]

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"
  • UPDATE: Will remove from next report as this no longer appears to be an issue (BC) [20101026]

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • temporary increase in size of HOTDISK from 1 to 1.5 TB to avoid DDM blacklist while the central list of required DBReleases is coordinated by ADC [20101013]
    • cleanup by ADC across sites did not liberate much space

  • configured CREAM CE [20100929]
    • under local test and debugging of ENV settings

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • scheduled downtime to PPS for 3 days starting 20101018

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • no news

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
production-dashboard-ca-20101026.png

production-dashboard-none-scinet-20101026.png production-dashboard-none-uvic-20101026.png

  • Within the CA Cloud (Panglia View):
panglia-production-ca-week-20101026.png panglia-analysis-ca-week-20101026.png

  • How we fared to other ATLAS sites during the past week:
production-dashboard-all-20101026.png

panglia-production-all-week-20101026.png panglia-analysis-all-week-20101026.png

  • Alternate Views (See link for last 3 days only):
    • not available this week - Panda Monitor errors

  • CA Cloud (last 3 days):
    • not available this week - Panda Monitor errors 

site (finished/failed) [last 7 days]:
cloud:     CA (101130/6433)

TRIUMF (36083/187)   
CA-SCINET-T2 (14644/528)   
SFU-LCG2 (7325/17)   
Australia-ATLAS (5705/74)   
CA-ALBERTA-WESTGRID-T2 (4656/261)   
CA-VICTORIA-WESTGRID-T2 (4766/105)

ANALY_TRIUMF (12966/2628)   
ANALY_SCINET (5140/788)   
ANALY_ALBERTA-WG1 (3164/1575)   
ANALY_VICTORIA-WG1 (2233/64)
ANALY_SFU (1960/137)   
ANALY_AUSTRALIA (1984/49)   
ANALY_SFU_bugaboo (504/20)

Failed Jobs

  • From Panda Monitor (last week)
    • not available this week - Panda Monitor errors

  • From Dashboard (last week): 

name,rank,value
EXEPANDA_DQ2_STAGEIN,1,85
TRFERROR,2,37
TRF_UNKNOWN ,3,14
EXEPANDA_JOBDISPATCHER_HEARTBEAT,4,6
EXEPANDA_JOBKILL_BYPILOT,5,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,6,2
EXEPANDA_UNKNOWNERROR_JOBWRAPPERCRASH,7,1
TRF_PYT_SYNTAX ,8,1

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,11
EXEPANDA_GET_SIZEMISMATCH,2,5
TRF_UNKNOWN ,3,2
TRFERROR,4,1

    • Australia-ATLAS 
name,rank,value
EXEPANDA_DQ2_STAGEIN,1,29
EXEPANDA_DQ2PUT_FILECOPYERROR,2,23
TRFERROR,3,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,4,4
EXEPANDA_JOBKILL_BYPILOT,5,3
TRF_UNKNOWN ,6,1



name,rank,value
TRFERROR,1,106
TRF_SVRINIT ,2,11
TRF_UNKNOWN ,3,4
TRF_EXC ,4,2
EXEPANDA_DQ2PUT_FILECOPYERROR,5,2
EXEPANDA_DQ2_STAGEIN,6,1

    • CA-SCINET-T2 (ce1) 
name,rank,value
TRF_SVRINIT ,1,157
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,2,41
EXEPANDA_JOBKILL_BYPILOT,3,37
EXEPANDA_DQ2PUT_FILECOPYERROR,4,17
EXEPANDA_JOBKILL_SIGTERM,5,14
EXEPANDA_PUT_LFCADDFILEINFOFAILED,6,11
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,7,8
TRF_UNKNOWN ,8,5
EXEPANDA_PUT_LFCMODULEIMPORT,9,5
EXEPANDA_JOBDISPATCHER_HEARTBEAT,10,5
TRFERROR,11,3
EXEPANDA_DQ2GET_INFILE,12,2
EXEPANDA_CMTCONFIGMISMATCH,13,1

    • CA-SCINET-T2 (ce2) 
name,rank,value
RF_SVRINIT ,1,151
EXEPANDA_JOBKILL_BYPILOT,2,38
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,3,27
EXEPANDA_DQ2PUT_FILECOPYERROR,4,10
EXEPANDA_PUT_LFCADDFILEINFOFAILED,5,10
TRF_UNKNOWN ,6,6
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,6
EXEPANDA_PUT_LFCMODULEIMPORT,8,4
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,9,2
EXEPANDA_JOBKILL_SIGTERM,10,1

    • CA-VICTORIA-WESTGRID-T2 
name,rank,value
TRFERROR,1,27
EXEPANDA_JOBKILL_BYPILOT,2,23
TRF_SEGVIO ,3,17
EXEPANDA_DQ2_STAGEIN,4,17
EXEPANDA_JOBDISPATCHER_HEARTBEAT,5,2
EXEPANDA_DQ2PUT_FILECOPYERROR,6,2

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes: (October 19 - 25, 2010) 

- active tasks from PandaDB: 781
      -- distribution by cloud:
       CA:4 CERN:1 DE:164 ES:10 FR:133 IT:71 ND:17 NL:8 TW:3 UK:205 US:165

1) Tue Oct.19
+ CA/CA-ALBERTA-WESTGRID-T2_SCRATCHDISK FT f./AsyncWait er./locality is
  unavailable.GGUS 62310 solved:restarted dcache pool node which had
  previously shown 3 problematic pools (2 readonly; 1 offline).All online.

-- BryanCaron - 2010-10-26
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20101026.png r1 manage 882.3 K 2010-10-26 - 15:03 BryanCaron  
PNGpng apel-statistics-ca-20101026.png r1 manage 1412.1 K 2010-10-26 - 15:03 BryanCaron  
PNGpng apel-status-aus-20101026.png r1 manage 15.0 K 2010-10-26 - 15:03 BryanCaron  
PNGpng apel-status-ca-20101026.png r1 manage 25.2 K 2010-10-26 - 15:03 BryanCaron  
PNGpng panda-dashboard-pd2p-ca-20101026.png r1 manage 499.6 K 2010-10-26 - 15:37 BryanCaron  
PNGpng panda-dashboard-statistics-summary-ca-20101026.png r1 manage 1210.6 K 2010-10-26 - 18:49 BryanCaron  
PNGpng panglia-analysis-all-week-20101026.png r1 manage 35.6 K 2010-10-26 - 15:08 BryanCaron  
PNGpng panglia-analysis-ca-week-20101026.png r1 manage 31.3 K 2010-10-26 - 15:08 BryanCaron  
PNGpng panglia-production-all-week-20101026.png r1 manage 32.7 K 2010-10-26 - 15:08 BryanCaron  
PNGpng panglia-production-ca-week-20101026.png r1 manage 29.8 K 2010-10-26 - 15:08 BryanCaron  
PNGpng production-dashboard-all-20101026.png r1 manage 953.5 K 2010-10-26 - 15:13 BryanCaron  
PNGpng production-dashboard-ca-20101026.png r1 manage 412.4 K 2010-10-26 - 15:13 BryanCaron  
PNGpng production-dashboard-none-scinet-20101026.png r1 manage 128.8 K 2010-10-26 - 15:22 BryanCaron  
PNGpng production-dashboard-none-uvic-20101026.png r1 manage 135.4 K 2010-10-26 - 15:22 BryanCaron  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2010-10-26 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback