TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20110201 (2011-02-01, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2011-02-01

General News

  • ADC News [20110201]
  • CA Cloud Squad Actions [20110201]
    • DaTRI requests approved / processed (last week)
      • TRIUMF-LCG2_LOCALGROUPDISK: 0 / 0 GB
      • Australia-ATLAS_LOCALGROUPDISK: 0 / 135.9 GB
      • CA-ALBERTA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • CA-SCINET-T2_LOCALGROUPDISK: 0 / 0.8 GB
      • CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • SFU-LCG2_LOCALGROUPDISK: 0 / 0GB
    • ATLAS Site Exclusion Savannah entries [20110201]
    • queue operations
      • HammerCloud autoexclusions
        • CA-SCINET-T2 [20110131] and [20110201] analysis queue set to brokeroff due to dcap door problems at site
      • CA-SCINET-T2 production queues offline by ADCoS shifter due to dcap issues [20110131]
        • set back online (production) by ADCoS and CA cloud squad (analysis) following initial dcap fix [20110131]
        • back offline due to return of errors [20110201]
      • Australia-ATLAS production and analysis queues set to test by ADCoS shifter in advance of scheduled downtime [20110201]
    • other cloud squad related savannah and / or ROC ggus tickets [20110201]
    • PD2P (dynamic data placement) in CA cloud (20110201)
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets) [Last 5 days shown]
      • note that only DATADISK is now a destination for PD2P [since 20110112]
pd2p-ca-5days-20110201.png
    • central cleaning of MCDISK completed at CA cloud Tier-2s and Tier-3s [20110131]
      • sites can remove (and de-register) remaining files (list distributed via email) that are at the site and listed within the TRIUMF LFC
      • remaining files in /atlasmcdisk/ can be removed, including the directory and ATLASMCDISK space token

  • Open Issues and Ongoing Discussions
    • published site installed resource capacity information to be investigated and confirmed against WLCG information [20100928]
      • http://gstat-wlcg.cern.ch/apps/capacities/sites/
      • UPDATE [20110121] - total published resources at each CA Tier2 is correct; further adjustments needed to publish available (pledged) WLCG resources - info to come
      • UPDATE [20110201] - thank you to sites for updating published values of 'Shares'
        • outstanding items to be corrected:
          • SFU-LCG2 - "total online storage" for both all VOs and ATLAS VO incorrectly set at zero
          • CA-ALBERTA-WESTGRID-T2 - "total online storage" for ATLAS VO incorrectly set at zero
          • All sites to update published cpu resource capacities (PhysicalCPUs and LogicalCPUs) to 2011 NRAC levels in April 2011 to coincide with the next WLCG Pledge period
    • CREAM CE deployment preparations
      • new TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20101019]
      • [20110112] the plan is to first do the switch at TRIUMF, sometime after the scheduled downtime that takes place early next week coincident with the ADC outage, followed by a scale-up of the number of WNs at the site that receive jobs from the CREAM CE
    • HammerCloud
    • issue of sites within Cloud None within the production dashboard [20100829]
      • issue raised to ADC operations email list given that there was no ADC Weekly meeting yesterday
      • reply from ADC monitoring coordinator (A. Read) is that this long-standing problem is known and being worked on, but is difficult to fix
      • this problem will be on the agenda for the next ADC monitoring meeting scheduled for next week
      • UPDATE (20101221): will follow-up in new year with this as the production system dashboard is no longer used and instead there is a historical job history available within the ADC Monitoring pages
      • Previously Reported Information on this topic:
        • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
          • reminder message posted to the Dashboard Savannah entry for this problem https://savannah.cern.ch/bugs/index.php?64507 [20100518] (updated 20101012)
          • news from ADC monitoring coordinator A. Read (20101013):
            • "The cloud issue has proven difficult to deal with in the present production dashboard framework. This month and next we are discussing changes in direction which we hope will make it much more straightforward to deal with the clouds."
    • CA-VICTORIA-WESTGRID-T2 site missing from the ATLAS dashboard SAM Visualization page [20100824]
    • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]

  • Security News [20110125]
    • REMINDER: High Risk Vulnerability CVE-2010-3859 kernel: heap overflow in tipc_msg_build() [EGI-ADV-20110701]

  • Squid Deployment [20110201]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [20110125]
      • CA-ALBERTA-WESTGRID-T2 OK (OK for local squid; OK backup to SFU)
      • CA-SCINET-T2: OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 OK (OK for local squid; OK for backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)

SAM Results

  • site service status via Gridview - Please check your site

  • January 2011 (valid until 20110131)
gridview-ops-triumf-20110201.png gridview-ops-aus-20110201.png gridview-ops-alberta-20110201.png gridview-ops-scinet-20110201.png gridview-ops-sfu-20110201.png gridview-ops-victoria-20110201.png

APEL Accounting

APEL Publishing Status

apel-status-triumf-20110201.png apel-status-aus-20110201.png apel-status-ca-20110201.png

APEL Accounting Results

apel-statistics-triumf-20110201.png apel-statistics-aus-20110201.png apel-statistics-ca-20110201.png

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • dCache door network tuning
  • cream CE testing
  • dCache testbed - futher studies regarding dCache upgrade path
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3 - targetting mid-September for upgrade - delayed
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • see section above for SAM Results regarding regular ATLAS VO test failures which occur at regular intervals over the month [20110125]
    • ended 20110125

  • published online storage in tables (see above) showing as 0 TBs both for all VOs and ATLAS VO [20110201]

Outstanding Issues

  • cream CE installation and testing started [20100929]
    • test installation (blackhole.westgrid.ca) at first which will subsequently be applied to update the lcg-CEs (snowpatch then bugaboo)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • CREAM CE debugging ongoing

Outstanding Issues

  • CREAM CE installation to be performed in next few weeks [20101005]
  • Occasionally some jobs fail to stage-in .gpg files via scp and have to be terminated. (20100928)
    • 20101214 - This is believed to be fixed now. The probable cause was LDAP connections being sporadically broken, due to the ldap user account on the LDAP server occasionally hitting the ulimit of 1024 open files. We have not observed the problem for awhile now.

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • tuning applied to dcap service node to overcome access issues (see above tickets) [20110201]

Outstanding Issues

  • SIR to be completed since service errors occurred in December 2010 for a continuous period that lasted greater than 24 hours [See 20101223 and 20101224]

  • gpfs problems required scheduler shutdown and gpfs restart causing all running jobs to be lost [20101225]
    • SIR requested [20110104]

  • tcp tuning exercise [20101116]
    • corrected routing issue between doors and pools which previously were transferring data to pools over external nics rather than internal network [20101112]
    • improvements in communication between LFC server and site WNs observed
    • retry ENV variables to be added still

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)
    • Q: Are these still an issue and under investigation? (BC) [20101026]

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • configured CREAM CE [20100929]
    • under local test and debugging of ENV settings

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud (Panglia View):
panglia-production-week-ca-20110201.png panglia-analysis-week-ca-20110201.png

  • How we fared to other ATLAS sites during the past week:
panglia-production-week-all-20110201.png panglia-analysis-week-all-20110201.png

  • Alternate Views (Source link for last 7 days): 

Time interval:   2011-01-25 00:00:00   to   2011-02-01 00:00:00

cloud:  [FINISHED/FAILED]

US (599174/115745)   
DE (256676/49998)   
FR (230537/37906)   
UK (155809/34533)   
CERN (141130/40554)   
ES (23245/127905)   
IT (141588/5937)   
NL (95976/14221)   
ND (83735/17565)   
CA (58481/10253)   
TW (47150/3658)

Cloud:             [FINISHED/FAILED]
                  CA (58481/10253)

Production Site:
Australia-ATLAS          (876/66)
CA-ALBERTA-WESTGRID-T2   (2119/269)
CA-SCINET-T2            (5370/2163)
CA-VICTORIA-WESTGRID-T2 (2217/172)
SFU-LCG2                (1713/86)
TRIUMF                  (10881/485)

Analysis Site:
ANALY_AUSTRALIA         (2558/69)
ANALY_ALBERTA-WG1       (4327/350)
ANALY_SCINET            (6983/2729)
ANALY_VICTORIA-WG1      (2872/277)
ANALY_SFU               (1402/408)
ANALY_SFU_bugaboo       (440/604)
ANALY_TRIUMF           (16723/2575)

Failed Jobs

  • From Panda Monitor (last week)
You can find the meanings of the transform error codes on this link.

panda-dashboard-errors-header-20110201.png panda-dashboard-errors-aus-20110201.png panda-dashboard-errors-ca1-20110201.png panda-dashboard-errors-vic-20110201.png panda-dashboard-errors-sfu-20110201.png panda-dashboard-errors-triumf-20110201.png

  • My Notes:

  • Yuri's notes: (Jan 25 - 31, 2011) 

- active tasks from Panda-monitor:
      -- distribution by cloud:
 CA:57 CERN:43 DE:188 ES:12 FR:149 IT:16 ND:171 NL:106 TW:68 UK:342
 US:191

5) Sat Jan.29
+ CA/CA-SCINET-T2 many job f./lost heartbeat.GGUS 66770 solved:system
  misconfiguration crashed all of the nodes on these four frames.Fixed.

7) Mon Jan.31
+ CA/CA-SCINET-T2 many job f./Get err.:dccp failed.GGUS 66815 solved:
  issue on dcap door node,restarted.

-- BryanCaron - 2011-02-01
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20110201.png r1 manage 845.7 K 2011-02-01 - 17:04 BryanCaron  
PNGpng apel-statistics-ca-20110201.png r1 manage 1117.3 K 2011-02-01 - 17:04 BryanCaron  
PNGpng apel-statistics-triumf-20110201.png r1 manage 821.8 K 2011-02-01 - 17:04 BryanCaron  
PNGpng apel-status-aus-20110201.png r1 manage 15.9 K 2011-02-01 - 17:04 BryanCaron  
PNGpng apel-status-ca-20110201.png r1 manage 20.4 K 2011-02-01 - 17:04 BryanCaron  
PNGpng apel-status-triumf-20110201.png r1 manage 14.9 K 2011-02-01 - 17:04 BryanCaron  
PNGpng gridview-ops-alberta-20110201.png r1 manage 16.1 K 2011-02-01 - 17:16 BryanCaron  
PNGpng gridview-ops-aus-20110201.png r1 manage 16.0 K 2011-02-01 - 17:16 BryanCaron  
PNGpng gridview-ops-scinet-20110201.png r1 manage 16.2 K 2011-02-01 - 17:16 BryanCaron  
PNGpng gridview-ops-sfu-20110201.png r1 manage 16.4 K 2011-02-01 - 17:16 BryanCaron  
PNGpng gridview-ops-triumf-20110201.png r1 manage 16.2 K 2011-02-01 - 17:16 BryanCaron  
PNGpng gridview-ops-victoria-20110201.png r1 manage 16.2 K 2011-02-01 - 17:16 BryanCaron  
PNGpng panda-dashboard-errors-aus-20110201.png r1 manage 569.3 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panda-dashboard-errors-ca1-20110201.png r1 manage 1856.0 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panda-dashboard-errors-header-20110201.png r1 manage 207.4 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panda-dashboard-errors-sfu-20110201.png r1 manage 424.9 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panda-dashboard-errors-triumf-20110201.png r1 manage 967.5 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panda-dashboard-errors-vic-20110201.png r1 manage 617.3 K 2011-02-01 - 18:04 BryanCaron  
PNGpng panglia-analysis-week-all-20110201.png r1 manage 35.2 K 2011-02-01 - 17:21 BryanCaron  
PNGpng panglia-analysis-week-ca-20110201.png r1 manage 31.3 K 2011-02-01 - 17:21 BryanCaron  
PNGpng panglia-production-week-all-20110201.png r1 manage 32.9 K 2011-02-01 - 17:21 BryanCaron  
PNGpng panglia-production-week-ca-20110201.png r1 manage 30.3 K 2011-02-01 - 17:21 BryanCaron  
PNGpng pd2p-ca-5days-20110201.png r1 manage 505.1 K 2011-02-01 - 17:07 BryanCaron  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2011-02-01 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback