TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100914 (2010-09-21, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-09-14

  • NOTE: weekly report is complete with the exception of ggus ticket and production job error information

General News

  • ADC News [20100914]
    • This Week
      • ADC Weekly [20100913] http://indico.cern.ch/conferenceDisplay.py?confId=107126
      • Highlights
        • MC production
          • geometry for mc10 production is incorrect with fixes validated over the weekend; will need to restart the the G4 simulation campaign (~ wednesday)
        • CREAM CE update
          • summer release of condor 7.5.3 addresses many issues
          • ATLAS ready to scale up ttests through updated pyfactory so as to use CREAM CE exclusively for sites that have one
          • ATLAS sites should install CREAM CE over the next 1 to 2 months
    • Upcoming Events
  • CA Cloud Squad Actions [20100914]
    • DaTRI requests approved
      • 4 mc09 containers to each T2 LOCALGROUPDISK for the purpose of future HammerCloud tests: Total = 4.3 TB per site [20100908]
        • replication to CA-ALBERTA-WESTGRID-T2 on hold pending site adjustments to space tokens
    • other DaTRI requests completed (last week)
      • TRIUMF-LCG2_LOCALGROUPDISK : 436 GB
    • ATLAS Site Exclusion Savannah entries created
    • queue operations
      • Australia-ATLAS: analysis and production queues set to offline [20100907] by ADCoS at request of site due to scheduled downtime
        • validated and returned to online status by ADCoS shifters [20100909][production and analysis]
          • in the future the site will make use of SiteTests to validate site analysis queues in parallel with ADCoS validation for production
      • CA-SCINET-T2: analysis and production queues set to test [20100908] by cloud squad due to scheduled downtime
    • cloud squad related savannah and ggus tickets [20100914]
    • Open Issues and Ongoing Discussions:
      • submitted ticket to DDM Savannah requesting migration of GROUPDISK hosted space from CA-ALBERTA-WESTGRID-T2 to CA-VICTORIA-WESTGRID-T2 [20100903]
      • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • issue of sites within Cloud None within the production dashboard [20100829]
        • issue raised to ADC operations email list given that there was no ADC Weekly meeting yesterday
        • reply from ADC monitoring coordinator (A. Read) is that this long-standing problem is known and being worked on, but is difficult to fix
        • this problem will be on the agenda for the next ADC monitoring meeting scheduled for next week
        • Previously Reported Information on this topic:
          • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • CA-VICTORIA-WESTGRID-T2 site missing from the ATLAS dashboard SAM Visualization page [20100824]
      • Pending action: update values for maxinputsize for SFU-LCG2 (snowpatch and bugaboo) within panda schedconfig [20100817]
      • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
      • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
      • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
        • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
          • resolved by changing WN client NFS mounts to v3 from original NFSv4 (root cause within NFSv4 still unknown) [OPEN]
      • ROOTSYS truncated at 127 characters issue [20100608]
        • http://savannah.cern.ch/bugs/?68546 (open)
        • currently affecting Victoria (Tier3) and CA-SCINET-T2
        • varying character lengths below and above 127 characters show variable success and failure
        • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
          • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • Squid Deployment [20100914]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [2010907]
      • CA-ALBERTA-WESTGRID-T2 OK (both for local squid and backup to SFU)
      • CA-SCINET-T2: OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 OK (both for local squid and backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)

SAM Results

egi-avail-reliab-aug2010-header.png egi-avail-reliab-aug2010-CA.png egi-avail-reliab-aug2010-AUS.png

APEL Accounting

APEL Publishing Status

apel-status-aus-20100914.png

apel-status-ca-20100914.png

APEL Accounting Results

apel-statistics-aus-20100914.png

apel-statistics-ca-20100914.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • CA-ALBERTA-WESTGRID-T2 site status [20100907]
    • completed scheduled downtime planned for 20100901 (3 hours) coincident with TRIUMF downtime adding additional storage LUNs to dCache pools
      • resized all space tokens (with the exception so far of MCDISK and GROUPDISK) to the target values
    • production activity problems with release 15.6.10 (see above)
      • query into A. de Salvo regarding possible re-install of release + re-validation
      • re-validation failed (20100831) and so reinstallation required and requested
    • dCache SRM space usage discprepancy work-around scripts under investigation (postponed from previous weeks)
      • working to install version of postgresql-python interface compatible with version 8 of postgresql that is installed on the SL48 dCache admin node

Outstanding Issues

  • DDM: discrepancy between SRM and DQ2 usage (see above savannah) [20100511]
    • tests on postgres dump of dCache databases pending
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3 - targetting mid-September for upgrade
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • (un)scheduled downtimes starting 20100819 due to DDN array issues [ended 20100825]

Outstanding Issues

Toronto

Recent Issues

  • analysis job issues
    • See "Notes" section below

SciNet Report from the last week (L. Groer)

  • dCache updates prior to upgrade to 1.9.5-22:
    • Updated 30k records in the srmspacefile table that were missing. Used script from Dmitry Litvintsev. Note that "standard" method for creating the list of missing pnfsid's and size information did not work by scripting to the dCache admin interface due to an ssh bug (corrected in dcache 1.9.5-10) so the required information needed to be constructed by hand from sql queries.
    • Removed all references to a temporary pool area that was used originally for testing
    • Enable Berkeley database for all pool nodes and migrate data from flat-file format. Pools now start in 10-20 seconds rather than the typical 2-5 minutes.
    • Clean up of configuration files and log areas before archiving the 1.9.5-9 setups
    • Cleanup of node backup scripts and modification of local repositories

  • dCache upgrade to 1.9.5-22:
    • Upgrade all 4 head nodes, 4 door nodes and 4 pool nodes to 1.9.5-22
    • Work being done to correct authorization and permission handling for all users and all areas

  • NOT DONE YET (dCache)
    • Entries that are in srmspacefile that do not correspond to an existing chimera database entry were identified but not removed yet (fairly trivial operation).
    • Move dcache pool meta data off DDN LUNs onto local drives for enhanced performance and easier backups.
    • Finalize the gPlazma/dcache.kpwd setups to be commensurate with required authorization levels.
    • Add last remaining DDN LUNs (16 + 8 TB) into the configuration (these were the old NFS software areas)
    • Understand if the VOMS issue pointed out by Di is affecting our site.

  • Swap space:
    • With a few exceptions, SciNet nodes have 2GB of physical memory per core and no internal disks. There are a few nodes with internal disks and the HEP image has been modified to provide swap space if there is an internal disk available. There are only a few nodes and the primary purpose is to provide a performance baseline as other options for providing additional vmem are considered.

  • Migration of software area:
    • The software area has been served via NFS from one of our DDN LUN's. This has been migrated to RAIDn array of SAS disks on one of the pool head nodes. The migration had originally been intended solely for performance testing. However, it has been placed into production after FS corruption on the DDN LUN that occurred during unexpected shutdown on Monday night. The software area on DDN is being restored as a fall back if the performance of SAS server does not meet expectations. Note, the new area has the paths modified to fix the ROOT bug with path lengths.

  • Upgraded torque client on lcg-ce2.
    • In order to address double counting in BDII report of running and queued jobs, the CE's need to be able to identify a job's submitting CE. A new version of the torque client that was compiled with the flag --withtcl and a brief .qstatrc makes this possible. This is deployed on lcg-ce2. Once we confirm that it works as expected and does not break anything, we will deploy the same setup on lcg-ce1.

  • Other HEP Image Changes:
    • gpfs-3.3 client
    • minor clean-up
    • additional monitoring hooks being added (e.g. ganglia)

Outstanding Issues

  • pilot issues [20100511]
    • significant number of immediately failing pilots for both TRIUMF and LMU factories
    • need to logon to WN directly and run the wrapper script by hand

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)

  • large number of LFC related errors (20100404)
    • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • no news

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
production-dashboard-ca-20100914.png

  • Within the CA Cloud (Panglia View):
panglia-ca-production-20100914.png panglia-ca-analysis-20100914.png

  • How we fared to other ATLAS sites during the past week:
production-dashboard-all-20100914.png

  • Alternate Views (See link for last 3 days only):
panda-dashboard-summary-20100914.png

  • CA Cloud (last 3 days):
panda-dashboard-ca-20100914.png

Failed Jobs

  • From Panda Monitor (last week) Not Available

  • From Dashboard (last week): 

name,rank,value

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value

    • Australia-ATLAS 
name,rank,value





name,rank,value

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes: (September 7 - September 14, 2010) 

- active tasks from PandaDB: 132
      -- distribution by cloud:
       CA:6 CERN:4 DE:19 ES:8 FR:12 IT:6 ND:12 NL:13 TW:8 UK:15 US:29

2) Wed Sep.8
+ CA/CA-SCINET-T2_LOCALGROPDISK FT f./DEST./GENERAL_F./Protocol(s)
  specified not supported:[gsiftp].GGUS 61939 verified:site in SD.

5) Sat Sep.11
- CA/TRIUMF-LCG2->CA-SCINET-T2 FT f./GRIDFTP_ER.:FTP door not ready.GGUS
  62006 on hold,62009 in progress.

6) Sun Sep.12
- CA/CA-SCINET-T2-DATADISK<-AUSTRALIA,TRIUMF >40000 FT f./SECURITY_ER.:
  permission denied.GGUS 62009 in progress:auth issues on gridftp doors in
  dCache 1.9.5-22 installed over the weekend.

-- BryanCaron - 2010-09-13
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-published-statistics-aus-august2010.png r1 manage 369.9 K 2010-09-13 - 17:44 BryanCaron  
PNGpng apel-published-statistics-ca-august2010.png r1 manage 378.7 K 2010-09-13 - 17:44 BryanCaron  
PNGpng apel-statistics-aus-20100914.png r1 manage 879.0 K 2010-09-14 - 19:27 BryanCaron  
PNGpng apel-statistics-ca-20100914.png r1 manage 1306.7 K 2010-09-14 - 19:27 BryanCaron  
PNGpng apel-status-aus-20100914.png r1 manage 15.5 K 2010-09-14 - 19:27 BryanCaron  
PNGpng apel-status-ca-20100914.png r1 manage 25.8 K 2010-09-14 - 19:27 BryanCaron  
PNGpng panda-dashboard-ca-20100914.png r1 manage 741.1 K 2010-09-14 - 19:55 BryanCaron  
PNGpng panda-dashboard-summary-20100914.png r1 manage 1604.2 K 2010-09-14 - 19:41 BryanCaron  
PNGpng panglia-ca-analysis-20100914.png r1 manage 29.2 K 2010-09-14 - 19:32 BryanCaron  
PNGpng panglia-ca-production-20100914.png r1 manage 28.2 K 2010-09-14 - 19:32 BryanCaron  
PNGpng production-dashboard-all-20100914.png r1 manage 927.3 K 2010-09-14 - 19:43 BryanCaron  
PNGpng production-dashboard-ca-20100914.png r1 manage 441.6 K 2010-09-14 - 19:43 BryanCaron  
Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2010-09-21 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback