SCINET Tests

Introduction

Purpose was to run and also benchmark jobs at SCINET for local disks versus GPFS for scratch. Input datasets used were triumf19Aug2010.txt which consisted of the following:

mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
total files: 1971
total size: 2.01 TB
total events: 9851996
> dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
total files: 1000
total size: 755.12 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210_tid125845_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
total files: 500
total size: 757.96 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210_tid124453_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
total files: 1000
total size: 777.51 GB
total events: 9996249
> dq2-list-datasets-container mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123186_00

Note there are 5 datasets in those 4 containers. (Note: the above datasets reside in LOCALGROUPDISK and so new HC sites reflecting this need to be created.) Additional HC configurations: uncheck "Resubmit enabled" and set "Num datasets per bulk = 5". For Panda, filesize set by Dan/Johannes to -o[defaults_DQ2JobSplitter]filesize=13336 (in MB). For Ganga, configure HC to submit with filesize=13336 so that the number of jobs will match that of the Panda test.

Tests

• Test 3815 ANALY_SCINET_TEST
  • 250+ cores including nodes with and without local disks
  • These tests had an incorrect mix of nodes.

• Test 3911 ANALY_SCINET_TEST (copy-to-scratch / current default)
  • 80 cores with local scratch disks
  • 8 cores/node accessing the local disk (7200 rpm SATA)

• Test 3929 ANALY_SCINET_TEST (direct access)
  • 80 cores with local scratch disks
  • 8 cores/node accessing the local disk (7200 rpm SATA)

• Test 3993 ANALY_SCINET_TEST (direct access)
  • 80 cores with NO local disk
  • 8 cores/node accessing GPFS scratch area.
  • Many failed because of a site issue and will redo this test as test 4012.

• Test 4012 ANALY_SCINET_TEST (direct access)
  • Identical to test 3993 above.
  • 80 cores with NO local disk
  • 8 cores/node accessing GPFS scratch area.

• Test 4015 ANALY_SCINET_TEST (copy-to-scratch / current default)
  • 80 cores with NO local disk
  • 8 cores/node accessing GPFS scratch area.

• Test 4175 ANALY_SCINET_TEST (direct access)
  • Identical to test 3993 above.
  • 160 cores with NO local disk
  • 8 cores/node accessing GPFS scratch area.

• Test 4208 ANALY_SCINET_TEST (direct access)
  • Identical to test 3929 above.
  • 160 cores with local scratch disks
  • 8 cores/node accessing the local disk (7200 rpm SATA)

• Test 4210 ANALY_SCINET_TEST (copy-to-scratch)
  • Identical to test 3929 above.
  • 160 cores with local scratch disks
  • 8 cores/node accessing the local disk (7200 rpm SATA)

Hardware

Worker nodes

• 2 Xeon x5540 CPU's @ 2.53 GHz (8 cores/node)
• 1 GigE with 4.2:1 blocking (42 nodes share one 10 GigE link to core network)
• Disks:
  • 80 nodes have 250 GB, 7200 RPM SATA II drives (WD2502ABYS-23B7A)
  • Remainder use GPFS file system for scratch space
• 16 GB RAM
  • 0.8 GB system image resides in ramdisk
  • nodes with disk have 12 GB of swap space to provide about 27 GB available virtual memory
  • nodes that use GPFS use 1 GB for GPFS page-pool and provide about 14 GB of memory

Software

• 4x146GB 15 kRPM SAS drives in RAID-0 (IBM-ESXS Model: ST3146356SS)
• Provided via NFS from a pool node

Storage

• 2 DDN DCS 9900 couplets shared with GPFS file systems
• 43x16TB pools
• 4 pool nodes
  • Xeon E5430 @ 2.66 GHz (4 cores)
  • 8 GB RAM on node providing software NFS server, 4 GB on other three
  • 10 GigE connection to core network
• 4 door nodes
  • 1 dcap, 3 gridftp doors
  • Xeon E5430 @ 2.66 GHz (4 cores)
  • 4 GB RAM
  • 1 GigE each to internal and external networks

Configuration during testing

• Nodes on five switches were used during the testing
• There are 42 nodes on each switch
  • 26 nodes were in the GPC general use pool
  • 16 nodes were configured to run ATLAS jobs on each switch (8 with disk, 8 without)
    • 2/4 nodes from each frame were used for these tests
    • The remaining 14/12 nodes were in use by non-test ATLAS jobs

Results

Please see comparisons file.

• Significant improvement with local disk + direct-access especially compared to current gpfs + copy-to-scratch.
• Copy-to-scratch: long time to download input files even to local.
• Possible local + direct-access bottleneck seen (even at Victoria ~70-80% cpu use and 100MB /s /node). At SCINET, saw ~40% cpu and ~ 40MB /s /node (theoretical 125 MB/s) and cache at ~ 500MB.s. All indications are that it was waiting for dcap data (idle cpu).
• Athena running times a little long (2500 +/- 450s), even for direct-access + local. Compare with Victoria ~1500 +/- 250 s. Hardware (cpu, network?) difference or dcache ?
• See a surprisingly big difference between local vs gpfs for direct access in athena run times. This should only be dbRelease access + compiled sw libs ? Are there local swap disks in one test and on gpfs on another ?