CA Cloud Jan 2011 HC Tests

Introduction

Purpose was to study dcap and dccp access and to run stress tests since there are new sites and hardware commissioned on the cloud.

Input datasets used were triumf19Aug2010.txt which consisted of the following:

mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
total files: 1971
total size: 2.01 TB
total events: 9851996

> dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00


mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
total files: 1000
total size: 755.12 GB
total events: 4994464

> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210_tid125845_00

mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
total files: 500
total size: 757.96 GB
total events: 4994464

> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210_tid124453_00

mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
total files: 1000
total size: 777.51 GB
total events: 9996249

> dq2-list-datasets-container mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123186_00

Note there are 5 datasets in those 4 containers.

(Note: the above datasets reside in LOCALGROUPDISK and so new HC sites reflecting this need to be created.)

Additional HC configurations: uncheck "Resubmit enabled" and set "Num datasets per bulk = 5".

For Panda, filesize set by Dan/Johannes to -o[defaults_DQ2JobSplitter]filesize=13336 (in MB).

For Ganga, configure HC to submit with filesize=13336 so that the number of jobs will match that of the Panda test.

Tests

(Table below shows the test link for the sites; if a test is redone, the new links are listed under the appropriate columns.)

Test	Test Link	Alberta-WG1	Australia	SCINET	SFU snowpatch	SFU bugaboo	TRIUMF	Victoria-WG1
Panda (default - dccp)	2677			2691	cannot be done	2807
Panda (direct access - dcap)	2500			2646	2783	2783	2646	2525
Panda (filestager)	2647	(redo)		2679	2795	2795	2666

Notes:

• Run times are from logs until there was nothing to monitor. It is not indicative of how long the jobs are really running on the worker nodes.
• Some tests cannot be done:
  • SFU snowpatch: not enough input file space on WNs for some tests.
  • SFU WMS jobs: cannot differentiate between snowpatch and bugaboo since the SE are the same.

Additional Testing

March 2011

Victoria (96 core): 12 nodes (96 cores) reserved and a new queue (atlas-test) created. ptest pilots sent to atlas-test and HC tests rerun:

Default Access: test 2331, test 2343

• the dccp errors show up in both tests

test 2316

FileStager: test 2317

Observations and comments:

1. dcap This method puts the most load on dCache; the network bandwidth is 200 - 400 MB/s per pool node, and the limitation is reading the LUNs at 40 - 60 MB/s per LUN for random I/O. Each worker node receives data at 50 - 70 MB/s. In total there are up to 200 simultaneous dcap movers in dCache, spread across all pools, with 4 - 18 per pool.
2. filestager The WN disks are 100% busy writing data and can receive data over the network potentially twice as fast as it can be written. It seems that the data usually remains cached in memory during the file staging, so there is actually no benefit in writing the data to local disk.
3. default There is severe IO wait and disk contention, both for the reading and writing phases of the job. The CPU use is very low.

Results

See these PDF files:

• Australia: FileStager looks better
• SCINET: Direct access looks better
• SFU-bugaboo:!FileStager looks better
• SFU-snowpatch:!FileStager looks better. (no default test because of limited disk space on WN for input files).
• TRIUMF: FileStager looks better
• Victoria: Marginal difference between Direct access and FileStager but Athena is a bit faster with FileStager.
  • Dedicated 96 core test shows marginal differences between Direct access and FileStager. Default access is performing poorly probably because of local disk i/o (8 cores / WN and they were all accessing files on the local disk)