Victoria Cloud HC Tests

Introduction

Purpose was to run and also benchmark jobs on FutureGrid. Input datasets used were triumf19Aug2010.txt which consisted of the following:

mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
total files: 1971
total size: 2.01 TB
total events: 9851996
> dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
total files: 1000
total size: 755.12 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210_tid125845_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
total files: 500
total size: 757.96 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210_tid124453_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
total files: 1000
total size: 777.51 GB
total events: 9996249
> dq2-list-datasets-container mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123186_00

Note there are 5 datasets in those 4 containers. (Note: the above datasets reside in LOCALGROUPDISK and so new HC sites reflecting this need to be created.) Additional HC configurations: uncheck "Resubmit enabled" and set "Num datasets per bulk = 5". For Panda, filesize set by Dan/Johannes to -o[defaults_DQ2JobSplitter]filesize=13336 (in MB). For Ganga, configure HC to submit with filesize=13336 so that the number of jobs will match that of the Panda test.

FutureGrid cluster

The VMs at FutureGrid each show one core of an Intel X5570 @ 2.93GHz. Our VMs request 2 GB of RAM.

We are using the Hotel cluster at FutureGrid, descibed here. The cluster seems identical to Hermes aside from the CPU clock speed. The OS images run on this hard drive.

Tests

Round 1 201202

• Test 3800 ANALY_VICTORIA-WG1-CLOUD_TEST (using lcgcp)
  • This is similar to Test 3582 except that it is run on FutureGrid and lcgcp is used instead of direct-io.
  • We booted up to 88 VMs to run these jobs
  • At one point, dCache monitoring shows ~ 45 transfers occurring, each at ~ 1.2 MB/s each. Earlier with test jobs, we had ~ 15 transfers occuring at ~4-6 MB/s each. Later, there were ~85 transfers, at roughly ~ 0.7-1.2 MB/s each. This seems to suggest an aggregate bandwidth limitation due to an issue with the route to FutureGrid (should be fixed soon).
  • The FutureGrid VMs do not seem to be behind any type of web cache, so downloading the image from the vmrepo is slow. We should try a squid VM, but, we might not be able to get the nimbus workspace-control agent to use the http proxy. This test failed because the 32-bit version of libxml2 was missing in the VM image.

• Test 3878 ANALY_VICTORIA-WG1-CLOUD_TEST (using lcgcp)
  • The route issue to FutureGrid that was limiting the bandwidth should be fixed now.
  • The VM image is now hosted by a better server
  • The max number of VMs will be only 60 (to make sure there is spare network bandwidth available on our pool nodes, which is currently limited)
  • A few jobs failed because they ran on the older VM version that was missing 32-bit libxml2
  • When logging in to some VMs, I observed that some were very unresponsive (took >~ 5 minutes to log in and do some simple commands). Maybe the physical hosts were starved for IOPS, or possibly swapping?

• Test 3895 ANALY_VICTORIA-WG1-CLOUD_TEST (using curl)
  • Max number of VMs still 60
  • Observations from 'top': the CPU use is ~ 30 - 70%, often going into the D state for uninterruptible sleep (possibly waiting on IO). There is only ~ 8 MB of unused RAM.

Round 2 - 201205

For this round of testing, we were able to use 8-core VMs with 8 job slots per physical node, instead of previously using 8 single-core VMs per physical node. This should have many benefits: less bandwidth used for CVMFS because there are fewer CVMFS caches to fill up, and similarly less local disk contention because there is only 1 OS image and 1 CVMFS partition on the local disk instead of 8 of each. However, the FutureGrid nodes apparently reserve one core for I/O, so the 8th core in our VMs is an oversubscription.

Also, our 4 pool nodes have been reconfigured with their public IPs on 10Gb NICs; previously the public interface was 1Gb and the total capacity of 4 * 1 Gb was a limitation.

• http://hammercloud.cern.ch/hc/app/atlas/test/20005608/ using curl
  • Up to ~ 190 jobs are running concurrently.
  • The 4 pool nodes have about ~ 120 - 240 MiB/s of traffic each (but this includes a small amount of gridftp traffic, and some local dcap)
  • This test got accidentally aborted

• http://hammercloud.cern.ch/hc/app/atlas/test/20005624/ using curl
  • We were able to use up to 128 cores on FutureGrid concurrently for this test
  • A few of these jobs ran on Hermes on "leftover" VMs. Some jobs also ran on the FutureGrid Foxtrot cloud due to a technical error. The jobs on Foxtrot took about 4 hours, whereas the others took about 30 minutes. Unfortunately this skews the results. Each data download process goes at < 1 MB/s on Foxtrot.
  • On a FGHotel VM where 8 athena processes were busy running, I observed each at only ~ 20% CPU. There seems to be contention with reading data from the scratch disk. dstat reported ~ 25 MB/s reading. Based on our HC experience on our T2 worker nodes, around 60 - 100 MB/s read speed can be required to feed all 8 cores on a node.
  • On a FGHotel VM where 3 jobs were busy staging in data, I observed a write speed to scratch disk of about 4 - 8 MB/s . On the same node, I did a curl download to /dev/null, and the average speed was ~ 1.5 MB/s. Since 3 * ~ 1.5 MB/s is roughly consistent with 4 to 8 MB/s, this implies the disk write speed was not the bottleneck. The network download speed seems lower than it should be when writing to /dev/null. When downloading an 872 MB image file from images.heprc.uvic.ca and writing to /home/, the rate was 7.6 MB/s. Doing the same on /scratch was about the same, 8.0 MB/s. Why is the curl stage-in slow? The dCache disks and servers do not seem particularly busy.
  • A few minutes later I tried another curl download to /dev/null and the speed was 10 MB/s. By this time fewer jobs were running. Perhaps there is network congestion somewhere? The dCache traffic decreased slightly, but it was still very light overall (only 30-60 MB/s per node). Compare ~ 120 - 240 MB/s each on test 5608.

• http://hammercloud.cern.ch/hc/app/atlas/test/20005655/ using curl
  • Up to ~ 220 jobs ran at once. Each pool node has only ~ 50 - 60 MB/s of traffic. So that's around 1.6 Gbit/s in total, or roughly 1 MB/s per job.

Results

We need to look into the performance of the scratch storage, for reading data while processing events. If we can get ext4 support installed in CernVM, that may give a significant improvement. Also, now that we have a single 8-core VM instead of 8 single-core VMs on a node, are we accessing different disks for the scratch storage? If the scratch storage performance is worse, that doesn't really make sense to me ... I thought it would all be on a high-performance clustered file system, so the number or size of partitions provided to our VMs wouldn't matter.

-- AsokaDeSilva - 2012-02-23