Stress Tests Summary (03 Dec - 11 Dec 2008)

This covers 2 tests: 57 (completed) and 68 (completed). In general, these tests have proved very useful to help improve the sites.

TRIUMF Tier-1

test 57 (99.5%): Ran well with 200 jobs. Requested 500 jobs for next test.

test 68 (91.4%): ~300 jobs accessed the same pool node for files when the jobs started; there were jobs failures after repeated attempts to access the input files.

Alberta Tier-2

test 57 (95.3%): Low throughput of events per second seen; corrected network bottleneck of 100 Mbps via the NAT server through which all access to the SE from WNs must pass; recabled to 1 Gpbs. Ganglia network: day, week.

test 68 (98.7%): No problems so far. Ganglia network week.

For future tests: I would like to avoid sending traffic from the WNs to the dCache pools via the NAT, by adding a 2nd internal GigE network interface to the dCache pools.

SFU Tier-2

test 57 (21.0%): Known issue (failed jobs had no stdout/tderr files to help debug.) Pool user home dir permission on s1 fixed; also found 2 failed nodes and fixed them (there were ssh key errors at s17).

test 68 (38.5%): Same failures seen. Not all ssh keys were properly setup on s17 - fixed Dec 9. Checked ssh keys on all nodes and pool user permissions.

Toronto Tier-2

test 57 (98.4%): Ran well with 126 jobs.

test 68 (97.6%): No problems so far.

Victoria Tier-2

test 57 (23.2 %): SE broke just prior to DAtest started. dcache and pools had to be restarted. Long I/O wait for input files as pools were not available.

test 68 (98.8%): No problems so far.

-- AsokaDeSilva - 10 Dec 2008