CA Cloud Oct 2010 HC Tests

Introduction

Purpose was to study dcap and dccp access and to run stress tests since there are new sites and hardware commissioned on the cloud.

Input datasets used were triumf19Aug2010.txt which consisted of the following:

mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
total files: 1971
total size: 2.01 TB
total events: 9851996

> dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00


mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
total files: 1000
total size: 755.12 GB
total events: 4994464

> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210_tid125845_00


mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
total files: 500
total size: 757.96 GB
total events: 4994464

> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210_tid124453_00


mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
total files: 1000
total size: 777.51 GB
total events: 9996249

> dq2-list-datasets-container mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123186_00

Note there are 5 datasets in those 4 containers.

(Note: the above datasets reside in LOCALGROUPDISK and so new HC sites reflecting this need to be created.)

Additional HC configurations: uncheck "Resubmit enabled" and set "Num datasets per bulk = 5".

For Ganga, configure HC to submit with filesize=13336 so that the number of jobs will match that of the Panda test.

Tests

(Table below shows the test link for the sites; if a test is redone, the new links are listed under the appropriate columns.)

Test	Test Link	Alberta-WG1	Australia	SCINET	SFU snowpatch	SFU bugaboo	TRIUMF	Victoria-WG1
Panda (default - dccp)	1648				cannot be done (not enough WN disk)	1771
Panda (direct access - dcap)	1913		1979	1979	1709		skipped	1709
Panda (filestager)	2073	redo		2112			skipped
WMS (default - dcap)	1675	redo		redo	cannot be done (cannot separate bugaboo/snowpatch)	cannot be done (cannot separate bugaboo/snowpatch)		redo
WMS (default - filestager)	2028				cannot be done (cannot separate bugaboo/snowpatch)	cannot be done (cannot separate bugaboo/snowpatch)	skipped	redo

Notes:

• Run times are from logs until there was nothing to monitor. It is not indicative of how long the jobs are really running on the worker nodes.
• Some tests cannot be done:
  • SFU snowpatch: not enough input file space on WNs for some tests.
  • SFU WMS jobs: cannot differentiate between snowpatch and bugaboo since the SE are the same.
• TRIUMF opted out of these tests because of other activities (They were done in Aug.)
• The release used for this test (15.6.6) and the MC test itself was removed around Nov 30; so the Alberta FileStager test could not be done on time when the pilot fix for the test was made available.

Panda Tests

• Test 1648:
  • This is a default access job (all input files copied - dccp - to WN first).
  • Run time ~ 2010-10-30 20:13:39 CET - 2010-10-31 20:11:19 CET
  • ANALY_SFU had significant failures because of limited local disk space for input files. ANALY_SFU_bugaboo was missing in this test (redone as test 1771).

• Test 1709:
  • This is a direct access job (dcap)
  • Ran from ~ 2010-11-06 01:00:48 CET - 2010-11-07 01:01:01 CET.
  • Significant failures seen for some sites:
    • Australia: eg. panda job 1140755198 (fixed Nov 10 in schedconfig):

Py:AthFile           INFO opening [rfio:/ags3.atlas.unimelb.edu.au/gridstorage/pool1/data/atlas/2010-09-08/AOD.123082._000161.pool.root.1.3403208.0]...
Py:inputFilePeeker WARNING Unable to open file [rfio://ags3.atlas.unimelb.edu.au//gridstorage/pool1/data/atlas/2010-09-08/AOD.123082._000161.pool.root.1.3403208.0]

• • • SCINET: eg; panda job 1140755013, more than 200 jobs failed:

Py:AthFile           INFO opening [dcap://142.150.188.83:22125//pnfs/scinet.utoronto.ca/data/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e517_s764_s767_r1204_r1210/mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00/AOD.123082._000019.pool.root.1]...
Py:inputFilePeeker WARNING Unable to fill inputFileSummary from file dcap://142.150.188.83:22125//pnfs/scinet.utoronto.ca/data/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e517_s764_s767_r1204_r1210/mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00/AOD.123082._000019.pool.root.1. File is probably empty. Will try again with next (if any).
Py:AthFile           INFO opening [dcap://142.150.188.83:22125//pnfs/scinet.utoronto.ca/data/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e517_s764_s767_r1204_r1210/mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00/AOD.123082._000022.pool.root.1]...
Py:inputFilePeeker WARNING Unable to fill inputFileSummary from file dcap://142.150.188.83:22125//pnfs/scinet.utoronto.ca/data/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e517_s764_s767_r1204_r1210/mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00/AOD.123082._000022.pool.root.1. File is probably empty. Will try again with next (if any).
...

• • • SFU bugaboo: never ran (fairshare ?)
    • ALBERTA-WG1: about 70 jobs failed because of dccp timeouts. Site experienced SE issues over the weekend.

• Test 1913:
  • This is a direct access job (dcap) - (retry 1)
  • Ran from ~ 2010-11-19 23:00:37 CET - 2010-11-21 23:00:17 CET.
  • Failures seen:
    • Australia: (not understood since schedconfig was unchanged) some jobs ran successfully

Py:AthFile           INFO opening [rfio:/dpm/atlas.unimelb.edu.au/home/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e507_s765_s767_r1215_r1210/mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00/AOD.125847._000847.pool.root.1]...
Py:inputFilePeeker    INFO Successfully filled inputFileSummary from file rfio:///dpm/atlas.unimelb.edu.au/home/atlas/atlaslocalgroupdisk/mc09_7TeV/AOD/e507_s\
765_s767_r1215_r1210/mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00/AOD.125847._000847.pool.root.1

while others failed to access rfio.

Py:AthFile           INFO opening [rfio:/ags9.atlas.unimelb.edu.au/gridstorage/pool4/data/atlas/2010-09-08/AOD.123186._000581.pool.root.1.3402964.0]...
Py:inputFilePeeker WARNING Unable to open file [rfio://ags9.atlas.unimelb.edu.au//gridstorage/pool4/data/atlas/2010-09-08/AOD.123186._000581.pool.root.1.3402964.0]
Py:inputFilePeeker WARNING caught:
'f' must be a file-like object

• • SCINET: nothing seems to have been submitted.
  • These tests were redone as 1979

• Test 2073 :
  • This is a file stager test (previous ones - not detailed on this Twiki - were incorrect because of a pilot bug.).
  • ran from 2010-12-03 21:00:40 CET - 2010-12-05 21:00:34 CET.
  • Prior tests had issues fixed (SCINET site debug string, Australia incorrect schedcofig values)
  • Alberta and SCINET failed. Test 2112 redone for SCINET. Alberta release was removed before retest could be scheduled.

WMS Tests

(Priority was to get the Panda HC tests results so this was not fully pursued in light of the WMS limitations for SFU and manual intervention required for setting the filesize).

• Test 1675:
  • This is a DQ2_LOCAL (dcap) job.
  • ran from 2010-11-04 21:30:59 - 2010-11-05 21:23:27 CET.
  • Only Australia,TRIUMF had enough successful runs.

• Test 2028:
  • This is a FileStager job.
  • ran from 2010-11-30 09:19:59 - 2010-12-02 09:00:09 CET.
  • Victoria did not go through (was blacklisted / outage).

Results

• Tests ran for approximately 1-2 days each between Oct 30 - Dec 10, a large time interval with other site activities.
• Some problems found and fixed (site, Panda schedConfig, Panda pilot)
• See the pdf files below (click on the test number to go to the HC page for raw data / more information).
• Reminder of what you are seeing:
  • Panda default = all input files copied to WN first before job starts
  • Panda/WMS Direct Access (DA) = network read of input files
  • Panda/WMS FileStager (FS) = one file is copied to WN and jobs starts; in the meantime next file is copied in the background.

Alberta

• Alberta.pdf
• Still has SE issues so should retest after resolution.
• DA/FS shows better results since for default test, wallclock includes initial lcg-cp of all inputs to WNs.
• Significantly longer Athena running times for DA/FS. Why - jobs waiting for input files and SE issues ? (the longer athena running time reduces Athena event rate).

Australia

• Australia.pdf
• This site has results for all tests but note that site SE is dpm and not dCache.
• CPU/Walltime clearly better for FS.
• Significantly less Athena running time for FS.

SCINET

• SCINET.pdf
• See differences between Panda and WMS (for FS). Although other sites show shape differences; their means are usually comparable. Do not understand this yet. These SCINET FS jobs ran ~ Dec 3 and Dec 7 so they are not so far apart in time.
• Significant difference in Athena running time for FS (WMS vs Panda) - since these are diskless WNs, could there have been a network bottleneck on one of the days ?
• Wallclock shows FS (Panda) outperforming Panda default.

SFU - snowpatch

• SFU-snowpatch.pdf
• Limited WN disk makes testing challenging. Also no WMS since we cannot distinguish between snowpatch & bugaboo.
• FS better than dcap access.

SFU - bugaboo

• SFU-bugaboo.pdf
• No WMS since we cannot distinguish between snowpatch & bugaboo.
• FS or dcap are similar.

Victoria

• Victoria.pdf
• FS or dcap are similar.