TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20101221 (2010-12-21, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-12-21

General News

  • ADC News [20101221]
    • This Week
    • Last Week
      • CERN Computing Centre power cut [20101218] affected many services and resulted in many job failures
      • ATLAS ATLR (panda database) hardware fc switch upgrade (few hours) [20101215]
      • TAG reprocessing submissions started [20101216] with subscriptions to CERN, T1s and T2s [20101217]
  • CA Cloud Squad Actions [20101221]
    • DaTRI requests approved / processed (last week)
      • TRIUMF-LCG2_LOCALGROUPDISK: 0 / 0 GB
      • Australia-ATLAS_LOCALGROUPDISK: 0 / 0 GB
      • CA-ALBERTA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • CA-SCINET-T2_LOCALGROUPDISK: 0 / 0 GB
      • CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • SFU-LCG2_LOCALGROUPDISK: 0 / 0 GB
    • ATLAS Site Exclusion Savannah entries [20101221]
    • queue operations
      • HammerCloud autoexclusions - none
      • CA-ALBERTA-WESTRID-T2 - production and analysis queues set to test by cloud squad for scheduled dowtime [20101219]
      • Australia-ATLAS - production and analysis queues set brokeroff by ADCoS for scheduled downtime [20101217]
    • other cloud squad related savannah and ggus tickets [20101221]
      • http://savannah.cern.ch/bugs/?74125 small logfiles being written to TAPE [20101018]
        • looks to be resolved with recent HITS merging tasks ; to be followed [20101207]
        • closed [20101214]
    • PD2P (dynamic data placement) in CA cloud (20101221)
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
pd2p-ca-20101221.png
    • Open Issues and Ongoing Discussions:
      • temporary increase in size of HOTDISK at TRIUMF from 1 to 1.5 TB prior to central cleanup of unneeded DBRelease versions for T1s [20101013]
        • All 8 digit DBrelease cleaning was done. The gain is space is marginal according to DQ2 monitoring [20101022]
      • published site installed resource capacity information to be investigated and confirmed against WLCG information [20100928]
      • CREAM CE deployment preparations
      • HammerCloud tests of TRIUMF completed via panda to compare direct i/o via dcap, FileStager and full file copy
      • issue of sites within Cloud None within the production dashboard [20100829]
        • issue raised to ADC operations email list given that there was no ADC Weekly meeting yesterday
        • reply from ADC monitoring coordinator (A. Read) is that this long-standing problem is known and being worked on, but is difficult to fix
        • this problem will be on the agenda for the next ADC monitoring meeting scheduled for next week
        • UPDATE (20101221): will follow-up in new year with this as the production system dashboard is no longer used and instead there is a historical job history available within the ADC Monitoring pages
        • Previously Reported Information on this topic:
          • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
            • reminder message posted to the Dashboard Savannah entry for this problem https://savannah.cern.ch/bugs/index.php?64507 [20100518] (updated 20101012)
            • news from ADC monitoring coordinator A. Read (20101013):
              • "The cloud issue has proven difficult to deal with in the present production dashboard framework. This month and next we are discussing changes in direction which we hope will make it much more straightforward to deal with the clouds."
      • CA-VICTORIA-WESTGRID-T2 site missing from the ATLAS dashboard SAM Visualization page [20100824]
      • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks [2010803]
      • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
      • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
      • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
        • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
          • resolved by changing WN client NFS mounts to v3 from original NFSv4 (root cause within NFSv4 still unknown) [OPEN]
      • ROOTSYS truncated at 127 characters issue [20100608]
        • http://savannah.cern.ch/bugs/?68546 (open)
        • currently affecting Victoria (Tier3) and CA-SCINET-T2
        • varying character lengths below and above 127 characters show variable success and failure
        • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
          • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
      • reviewing draft Squad How-To and providing feedback [20100407]
        • also contains useful information for sites
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

  • Squid Deployment [20101221]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [2011019]
      • CA-ALBERTA-WESTGRID-T2 OK (OK for local squid; OK backup to SFU)
      • CA-SCINET-T2: OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 OK (OK for local squid; OK for backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)

SAM Results

  • site service status via Gridview - Please check your site

  • December 2010 Results (only valid until 20101221)
gridview-ops-triumf-20101221.png gridview-ops-aus-20101221.png gridview-ops-alberta-20101221.png gridview-ops-scinet-20101221.png gridview-ops-sfu-20101221.png gridview-ops-victoria-20101221.png

APEL Accounting

accounting-nov2010-aus-20101220.png accounting-nov2010-ca-20101220.png

APEL Publishing Status

apel-status-aus-20101221.png apel-status-ca-20101221.png

APEL Accounting Results

apel-statistics-aus-20101221.png apel-statistics-ca-20101221.png

Christmas Vacation Coverage Plans

  • TRIUMF
    • A. de Silva - away 20101211 to 20110104
  • Australia
  • Alberta
    • E. Pei - away 20101227 to 20110117
  • SciNet
    • N. Knecht - away 20101222 to 20110105
  • SFU
    • S. Chelsky away (but reachable) 20101231 to 20110131
  • Victoria
    • R. Taylor away - 20101221 to 20110102
    • R. Enge away - 20101223 to 20101228
  • McGill

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • corrections completed to dCache for new dteam VOMS on production service

Outstanding Issues

  • dCache door network tuning
  • cream CE testing
  • dCache testbed - futher studies regarding dCache upgrade path
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3 - targetting mid-September for upgrade - delayed
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • corrections completed to dCache for new dteam VOMS on production service

Outstanding Issues

  • cream CE installation and testing started [20100929]
    • test installation (blackhole.westgrid.ca) at first which will subsequently be applied to update the lcg-CEs (snowpatch then bugaboo)

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • CREAM CE debugging ongoing

  • corrections completed to dCache for new dteam VOMS on production service

Outstanding Issues

  • CREAM CE installation to be performed in next few weeks [20101005]
  • Occasionally some jobs fail to stage-in .gpg files via scp and have to be terminated. (20100928)
    • 20101214 - This is believed to be fixed now. The probable cause was LDAP connections being sporadically broken, due to the ldap user account on the LDAP server occasionally hitting the ulimit of 1024 open files. We have not observed the problem for awhile now.

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • LFC communication errors (see ggus above)

  • corrections completed to dCache for new dteam VOMS on production service

Outstanding Issues

  • tcp tuning exercise [20101116]
    • corrected routing issue between doors and pools which previously were transferring data to pools over external nics rather than internal network [20101112]
    • improvements in communication between LFC server and site WNs observed
    • retry ENV variables to be added still

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)
    • Q: Are these still an issue and under investigation? (BC) [20101026]

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • temporary increase in size of HOTDISK from 1 to 1.5 TB to avoid DDM blacklist while the central list of required DBReleases is coordinated by ADC [20101013]
    • cleanup by ADC across sites did not liberate much space

  • configured CREAM CE [20100929]
    • under local test and debugging of ENV settings

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • no news

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

    • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud:

  • Within the CA Cloud (Panglia View):
panglia-production-ca-20101221.png panglia-analysis-ca-20101221.png

  • How we fared to other ATLAS sites during the past week:
panglia-production-all-20101221.png panglia-analysis-all-20101221.png

  • Alternate Views (Source link for last 7 days): 

Time interval:   2010-12-14 00:00:00   to   2010-12-21 00:00:00

cloud:  [FINISHED/FAILED]

US   (751957/84294)   
DE   (438858/38508)   
UK   (293315/56269)   
FR   (244690/75433)   
CERN (191326/33484)   
NL   (169582/17463)   
CA   (141296/11278)   
ES   (121389/15618)   
IT   (116330/10799)   
ND   (98736/2868)   
TW    (87568/6049)

Cloud:             [FINISHED/FAILED]
                  CA (141296/11278)

Production Site:
Australia-ATLAS          (3924/118)
CA-ALBERTA-WESTGRID-T2   (9798/259)
CA-SCINET-T2            (11150/4051)
CA-VICTORIA-WESTGRID-T2 (9225/956)
SFU-LCG2                (8265/303)
TRIUMF                  (38006/1558)

Analysis Site:
ANALY_AUSTRALIA         (8267/378)
ANALY_ALBERTA-WG1       (5973/683)
ANALY_SCINET            (1491/206)
ANALY_VICTORIA-WG1      (3024/751)
ANALY_SFU               (3104/61)
ANALY_SFU_bugaboo       (1587/81)
ANALY_TRIUMF           (37482/1873)

Failed Jobs

  • From Panda Monitor (last week)

panda-dashboard-errors-header-20101221.png panda-dashboard-errors-aus-20101221.png panda-dashboard-errors-ca1-20101221.png panda-dashboard-errors-vic-20101221.png panda-dashboard-errors-sfu-20101221.png panda-dashboard-errors-triumf-20101221.png

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes: (Dec 14 - December 20, 2010) 

- active tasks from PandaDB: 973
      -- distribution by cloud:
    CA:44 CERN:22 DE:72 ES:45 FR:95 IT:25 ND:84 NL:41 TW:41 UK:404 US:100

3) Thu Dec.16
+ CA/SFU-LCG2_DATADISK<-UKI-NORTHGRID-LIV-HEP FT f./GRIDFTP_ER.:Conn.reset
  by peer.GGUS 65412 verified:the new gridftp door setting was not tuned
  for high transfer rate,updated.
+ CA/TRIUMF-LCG2_MCTAPE FT f./SRMV2STAGER:SRM_FAILURE.GGUS 65413 verified:
  after 5 failed bringonline attempts,the 6-th were successful.
+ CA/CA-VICTORIA-WESTGRID-T2 many job f./Get er.:dccp failed with output
  'int' object is not callable.GGUS 65416 solved:dCache problem was fixed.

6) Sun Dec.19
- CA/CA-SCINET-T2 still job f./Failed to get LFC replicas,bad magic number
  GGUS 65003 in progress updated,also stage-out err.:"Failed to add file
  size and checksum to LFC"&"Error in copying the file from job workdir to
  localSE."

-- BryanCaron - 2010-12-21
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng accounting-nov2010-aus-20101220.png r1 manage 363.6 K 2010-12-20 - 22:20 BryanCaron  
PNGpng accounting-nov2010-ca-20101220.png r1 manage 312.7 K 2010-12-20 - 22:20 BryanCaron  
PNGpng apel-statistics-aus-20101221.png r1 manage 826.2 K 2010-12-21 - 15:44 BryanCaron  
PNGpng apel-statistics-ca-20101221.png r1 manage 1245.5 K 2010-12-21 - 15:44 BryanCaron  
PNGpng apel-status-aus-20101221.png r1 manage 15.2 K 2010-12-21 - 15:44 BryanCaron  
PNGpng apel-status-ca-20101221.png r1 manage 25.6 K 2010-12-21 - 15:44 BryanCaron  
PNGpng gridview-ops-alberta-20101221.png r1 manage 16.4 K 2010-12-21 - 15:55 BryanCaron  
PNGpng gridview-ops-aus-20101221.png r1 manage 16.4 K 2010-12-21 - 15:55 BryanCaron  
PNGpng gridview-ops-scinet-20101221.png r1 manage 16.5 K 2010-12-21 - 15:55 BryanCaron  
PNGpng gridview-ops-sfu-20101221.png r1 manage 16.2 K 2010-12-21 - 15:55 BryanCaron  
PNGpng gridview-ops-triumf-20101221.png r1 manage 16.3 K 2010-12-21 - 15:55 BryanCaron  
PNGpng gridview-ops-victoria-20101221.png r1 manage 16.4 K 2010-12-21 - 15:55 BryanCaron  
PNGpng panda-dashboard-errors-aus-20101221.png r1 manage 680.6 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panda-dashboard-errors-ca1-20101221.png r1 manage 2053.7 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panda-dashboard-errors-header-20101221.png r1 manage 80.1 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panda-dashboard-errors-sfu-20101221.png r1 manage 745.7 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panda-dashboard-errors-triumf-20101221.png r1 manage 926.0 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panda-dashboard-errors-vic-20101221.png r1 manage 870.8 K 2010-12-21 - 16:54 BryanCaron  
PNGpng panglia-analysis-all-20101221.png r1 manage 33.7 K 2010-12-21 - 15:44 BryanCaron  
PNGpng panglia-analysis-ca-20101221.png r1 manage 32.0 K 2010-12-21 - 15:44 BryanCaron  
PNGpng panglia-production-all-20101221.png r1 manage 33.4 K 2010-12-21 - 15:44 BryanCaron  
PNGpng panglia-production-ca-20101221.png r1 manage 30.9 K 2010-12-21 - 15:44 BryanCaron  
PNGpng pd2p-ca-20101221.png r1 manage 501.5 K 2010-12-21 - 15:44 BryanCaron  
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2010-12-21 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback