Victoria Hermes Tests

Introduction

Proposed program:

• Old benchmark on IB nodes (to evaluate the difference due to having CVMFS now, more data on DCS3700 storage, and hot file replication on Lustre)
• New benchmark on IB nodes (to compare old/new HC)
• New benchmark on ethernet nodes

<ethernet switch replacement >

• New benchmark on ethernet nodes again (before/after switch replacement)

• dCache pools in use at this time:
  • basilisk02 (8 DDN pools, and doors)
  • basilisk05,06,07,08 (DCS3700 storage)

Tests

Old benchmark on IB Nodes

* Dataset and Test Template: Description

* Hardware Description:

Test 20022277

Test 200222777

This failed.

Test 20022281

Test 200222781

This failed

New Benchmark

Dataset and Test Template: Description

Hardware: 7 Hermes Classic nodes (hermes0071-hermes0077) using C300 switch

Test 20022395

Test 20022395

• This was copy to scratch job
• Ran just before our ethernet switch replacement

Test 20022454

Test 20022454

• AGIS changed for the test queue so that direct access is allowed
• Will not work because HC job is configured for copy access instead of direct access.

Benchmark After 21 June Downtime

• Hardware:

Test 20022735

• Dataset and Test Template: Description
• Test 20022735
• Should be a direct access job

Conducted on 82 Hermes Classic nodes (0003-0084), just after switching the cluster to the new BNT ethernet network. Most (62) nodes had an empty CVMFS cache so it will have to fill up for the first jobs. Moreover, the squid cache is empty too. The Lustre replication pool on basilisk04 was not active for this test. During the downtime that just ended I did some data re-balancing, including atlaslocalgroupdisk. The HC input data are in these location:

/pnfs/westgrid.uvic.ca/data/atlas/atlaslocalgroupdisk/mc12_8TeV/AOD/e1493_s1499_s1504_r3658_r3549/mc12_8TeV.147666.Pythia8B_AU2_CTEQ6L1_ccTomu15.merge.AOD.e1493_s1499_s1504_r3658_r3549_tid00970581_00/
/pnfs/westgrid.uvic.ca/data/atlas/atlaslocalgroupdisk/mc12_8TeV/AOD/e1176_a159_a165_r3549/mc12_8TeV.147812.Pythia8_AU2CTEQ6L1_Wtaunu.merge.AOD.e1176_a159_a165_r3549_tid00842350_00/
/pnfs/westgrid.uvic.ca/data/atlas/atlaslocalgroupdisk/mc12_8TeV/AOD/e1233_a159_a165_r3549/

and are perfectly balanced across all pools. The cluster was largely idle at this time. There were < 24 jobs in the atlas queue, and a few other small tests, but some substantial gridftp traffic (writing ~100-200 MB/s on most pool nodes). The jobs complete fairly quickly (~ 30 - 45 minutes) so we never got more than ~150 jobs running concurrently. For that reason, most nodes were unused, and partway through the test, we offlined hermes0003-hermes0064, so then only 20 nodes were in the test.

On the WNs, CPU use only goes up to ~40%, and network traffic is ~ 20 MB/s.

Results

Comparing this test to an earlier one on Hermes Classic (20003582) is difficult because the HC templates are not the same. The job data access patterns are most likely different. However, the software setup time may be a valid comparison. In the recent test it is (7.9 +/- 3.4) s compared to (37.2 +/- 38.4) s in the old test, most likely showing the improvement due to using CVMFS.

Comparing this test to the same template run at other sites: SFU Australia:

• Wallclock Time and Athena Running Time at UVic are about half of the time at SFU
• SFU has 10% higher CPU percentage
• Event rate is about the same at SFU and UVic: 24 Hz
• Download Input Files is approximately the same at SFU and UVic: 36 - 39 s
• Setup software time is about 8 s at UVic compared to 26 s at SFU

-- AsokaDeSilva - 2013-06-11