Ior2016

IOR Benchmark The IOR software is used for benchmarking parallel file systems using POSIX, MPIIO, or HDF5 interfaces.

These IOR tests were performed on the UAB 4.7 PB gpfs storage fabric for performance estimation.

Downloaded from: https://github.com/LLNL/ior

IOR User guide is available at: http://www.csm.ornl.gov/essc/io/IOR-2.10.1.ornl.13/USER_GUIDE

Before build

Download IOR from github

export LDFLAGS=-L/usr/lpp/mmfs/lib
export CPPFLAGS=-I/usr/lpp/mmfs/include
LIBS=-lgpfs ./configure 

to avoid the gpfs_fcntl issue https://github.com/LLNL/ior/issues/15

Running IOR

Direct Running with out mpi

 
./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /data/user/tanthony/ior-master/

 ./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -numTasks 3 -o /data/user/tanthony/ior-master/

 srun -N2 --time=10:00:00 --mem=4096 --partition=medium --job-name=iortest  ./ior -f IOR.input

IOR -a POSIX -N 3 -b 500m -d 5 -t 128k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C

Running with mpi

Direct run

module load mpich/ge/gcc/64/3.2 
mpiexec -n 3 ./ior -a POSIX -N 3 -b 10m -d 5 -t 256k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C 

Test 0 started: Tue Sep  6 12:01:17 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB   Used FS: 1.7%   Inodes: 128.0 Mi   Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
    api                = POSIX
    test filename      = /data/user/tanthony/ior-master/
    access             = file-per-process
    pattern            = segmented (1 segment)
    ordering in a file = sequential offsets
    ordering inter file= constant task offsets = 1
    clients            = 3 (3 per node)
    repetitions        = 4
    xfersize           = 131072 bytes
    blocksize          = 500 MiB
    aggregate filesize = 1.46 GiB
Using Time Stamp 1473181277 (0x57cef65d) for Data Signature
Max Write: 4921.20 MiB/sec (5160.25 MB/sec)
Max Read:  11274.53 MiB/sec (11822.20 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        4921.20    4086.66    4646.19     332.18    0.32463 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0
read        11274.53   11176.68   11229.53      38.65    0.13358 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0

Finished: Tue Sep  6 12:01:58 2016

Increasing number of threads

mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o  /data/user/tanthony/ior-master/ -vv -k -wWr -C -F -i4
Using synchronized MPI timer
Start time skew across all tasks: 0.00 sec

Test 0 started: Tue Sep  6 12:17:53 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB   Used FS: 1.7%   Inodes: 128.0 Mi   Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
    api                = POSIX
    test filename      = /data/user/tanthony/ior-master/
    access             = file-per-process
    pattern            = segmented (1 segment)
    ordering in a file = sequential offsets
    ordering inter file= constant task offsets = 1
    clients            = 3 (3 per node)
    repetitions        = 4
    xfersize           = 262144 bytes
    blocksize          = 1 GiB
    aggregate filesize = 3 GiB
Using Time Stamp 1473182273 (0x57cefa41) for Data Signature

Max Write: 6295.34 MiB/sec (6601.14 MB/sec)
Max Read:  10185.52 MiB/sec (10680.30 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        6295.34    6139.65    6225.21      55.59    0.49352 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0
read        10185.52    9304.62    9676.96     325.32    0.31781 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0

Running via srun

srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C 

Increasing the number of nodes

• Running with three nodes

srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 2g -d 5 -t 2m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C
Max Write: 11845.79 MiB/sec (12421.21 MB/sec)
Max Read:  15105.21 MiB/sec (15838.96 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write       11845.79    5360.16    9535.91    2958.26    0.73459 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0
read        15105.21   12168.91   13998.89    1303.39    0.44299 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0 

• Running with 6 nodes

srun -N6 -n6 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 6 ./ior -a POSIX -N 6 -b 3g -d 5 -t 4m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C

Max Write: 20743.88 MiB/sec (21751.53 MB/sec)
Max Read:  19156.05 MiB/sec (20086.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write       20743.88    4791.11   15196.25    7362.94    1.88494 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0
read        19156.05   11148.20   15872.29    3424.16    1.22674 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0

• Running with 6 nodes and 16 tasks

srun -N16 -n16 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 16 ./ior -a POSIX -N 16 -b 4g -d 5 -t 4m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C


Max Write: 21229.09 MiB/sec (22260.31 MB/sec)
Max Read:  19458.28 MiB/sec (20403.48 MB/sec)