Victoria HC tests, September 2011

Note that these are stress tests. The purpose of these tests were verify direct access (dcap) prior to changing it to be the default mode of access for the ANALY queue. Refer to CACloudSiteHardware for hardware info.

Preparation

The following datasets were subscribed to the site for the ROOT / D3PD tests

mc10_7TeV.105015.J6_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277268_00
mc10_7TeV.109280.J4_pythia_jetjet_1muon.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277274_00
mc10_7TeV.108405.PythiaB_bbmu15X.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277282_00
mc10_7TeV.105014.J5_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277266_00
mc10_7TeV.109276.J0_pythia_jetjet_1muon.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277270_00 
mc10_7TeV.105010.J1_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277257_00 
mc10_7TeV.105012.J3_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277262_00 
mc10_7TeV.106059.PythiaB_ccmu15X.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277280_00 
mc10_7TeV.105009.J0_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277256_00 
mc10_7TeV.107691.AlpgenJimmyWmunuNp1_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277436_00 
mc10_7TeV.107700.AlpgenJimmyWtaunuNp0_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277446_00 
mc10_7TeV.105012.J3_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277261_00 
mc10_7TeV.105013.J4_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277264_00 
mc10_7TeV.105015.J6_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277267_00 
mc10_7TeV.109277.J1_pythia_jetjet_1muon.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277271_00 
mc10_7TeV.108326.PythiaB_bbe15X.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277283_00 
mc10_7TeV.116250.AlpgenJimmyZeeNp0_Mll10to40_pt20.merge.NTUP_SUSY.e660_s933_s946_r1831_r2040_p428_tid277526_00 
mc10_7TeV.116251.AlpgenJimmyZeeNp1_Mll10to40_pt20.merge.NTUP_SUSY.e660_s933_s946_r1831_r2040_p428_tid277527_00 
mc10_7TeV.116252.AlpgenJimmyZeeNp2_Mll10to40_pt20.merge.NTUP_SUSY.e660_s933_s946_r1831_r2040_p428_tid277528_00 
mc10_7TeV.108319.PythiaDrellYan_mumu.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277521_00 
mc10_7TeV.108322.PythiaDrellYanLowM_ee3.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277524_00 
mc10_7TeV.107681.AlpgenJimmyWenuNp1_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277424_00 
mc10_7TeV.107690.AlpgenJimmyWmunuNp0_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277435_00 
mc10_7TeV.107701.AlpgenJimmyWtaunuNp1_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277448_00
mc10_7TeV.116260.AlpgenJimmyZmumuNp0_Mll10to40_pt20.merge.NTUP_SUSY.e660_s933_s946_r1831_r2040_p428_tid277532_00
mc10_7TeV.108321.PythiaDrellYanLowM_mu3.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277523_00
mc10_7TeV.116271.AlpgenJimmyZtautauNp1_Mll10to40_pt20.merge.NTUP_SUSY.e660_s933_s946_r1831_r2040_p428_tid277539_00
mc10_7TeV.107680.AlpgenJimmyWenuNp0_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277423_00
mc10_7TeV.107660.AlpgenJimmyZmumuNp0_pt20.merge.NTUP_SUSY.e600_s933_s946_r1831_r2040_p428_tid277472_00
mc10_7TeV.106050.PythiaZee_1Lepton.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277678_00
mc10_7TeV.114680.SherpaQCD6jets20GeV.merge.NTUP_SUSY.e719_s933_s946_r1831_r2040_p428_tid279916_00
mc10_7TeV.105009.J0_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277255_00
mc10_7TeV.105013.J4_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277263_00
mc10_7TeV.109281.J5_pythia_jetjet_1muon.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277275_00
mc10_7TeV.105757.PythiaB_mu10mu10X.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277277_00
mc10_7TeV.106087.McAtNloZee_no_filter.merge.NTUP_SUSY.e613_s933_s946_r1831_r2040_p428_tid277679_00
mc10_7TeV.106047.PythiaZmumu_no_filter.merge.NTUP_SUSY.e574_s933_s946_r1831_r2040_p428_tid277677_00
mc10_7TeV.105014.J5_pythia_jetjet.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277265_00
mc10_7TeV.106052.PythiaZtautau.merge.NTUP_SUSY.e574_s934_s946_r1833_r2040_p428_tid277676_00

Note: these tests ran using poweruser allocation through pilots using the poweruser0 DN on the atlas-test queue at Victoria-WG1.

Tests

Athena / AOD tests

• Test 6569

There were some failures related to dcap I/O. They are mostly localized on hermes0051: http://panda.cern.ch:25980/server/pandamon/query?jobsummary=site&site=ANALY_VICTORIA-WG1_TEST

On hermes0051, I experimented with increasing DCACHE_RA_BUFFER from 32768 to 1048576. In fact, this decreased the bandwidth on the node by a factor of ~5 and caused a lot of System CPU use. These error messages in the dcap log on charon01 are probably related:

14 Sep 2011 02:05:17 (DCap-charon01-Unknown-2425439) [basilisk01_pool04 DoorTransferFinished 00009EE5AF916157405AA6D4F5E7 026ED03E] doorTransferArrived : 5 0 client failed 666 "Transfer failed: EOF on input socket (fillBuffer)"

I'm not sure why increasing the readahead buffer had such a bad effect. On all the other worker nodes the results and performance seemed pretty good, with few failures. We're still not getting anywhere close to stressing the DDN storage.

With 300 HC jobs running (and 65 normal analysis jobs), we reach about 500 - 600 MiB/s from each of the 4 pool nodes, with 280 movers total (up to 20 per pool). PoolCosts are roughly 0.02 - 0.10 . Most WNs have 70-80% CPU use and 50 - 60 MiB/s. The LUNs are read at about 70 - 80 MiB/s and are only ~20% busy. We see very brief spikes up to 300-400 MiB/s reading, so this suggests the LUNs can easily go faster but the network cannot, or dcap is not pulling the data efficiently.

• Test 6656

ROOT / D3PD tests

These are comparable with tests at DE or US cloud.

• Test 6581
  • Sep 15, 1225 PST: noticed too few jobs; HC incorrectly configured and was corrected (see HC test page above for configs)
  • Sep 15, ~ 1440 PST: noticed HC test was for default access; changed Panda queue to use dcap by default instead.
  • This test was aborted.

• Test 6589

With ~150 jobs running, there are around 50 active dcap movers at a time. Three of the pool nodes put out ~ 70 - 80 MiB/s, while the 4th does ~160 MiB/s (maybe it has more of the files ?) The LUNs are busier, but only have ~20 MiB/s read, and the CPU is higher with ~20% IO wait. This suggests the IO pattern of these jobs is more random with smaller chunk sizes. Both the dcache admin node and chimera node are significantly busier, in CPU and bandwidth, than for the Athena/AOD test, but the system load is still low overall. From the billing logs, it looks like the dcap connections are shorter and much more frequent. Since the jobs are very short (5-10 minutes), the WN stats are very spiky. Bandwidth is maybe 25 MiB/s per node, with ~ 30% CPU use. The local disks are often busy writing output, and staging in large DBRelease files via dccp.

Benchmark Tests

Using 6 dedicated nodes.

• Test 6658 DCACHE_RA_BUFFER=4096
• Test 6671 DCACHE_RA_BUFFER=16384
• Test 6672 DCACHE_RA_BUFFER=65536
• Test 6682 DCACHE_RA_BUFFER=262144
• Test 6715 DCACHE_RA_BUFFER=32768 (default)
• Test 6725 DCACHE_RAHEAD=FALSE
• Test 6726 DCACHE_RA_BUFFER=1048576
• Test 6744 DCACHE_RA_BUFFER=524288

Remarks

The results are approximately the same as DCACHE_RA_BUFFER varies from disabled to 262144. For 524288 and 1048576, all jobs failed. Tentatively, this may suggest (as one possibility) that the readahead is not fetching useful portions of data. It is just extra network traffic that is being discarded by the application. How exactly does the dcap readahead buffer mechanism work?

Conclusion

We did not change the DCACHE_RA_BUFFER from the default of 32768 .

-- AsokaDeSilva - 2011-09-14