Ior2016
Attention: Research Computing Documentation has Moved
https://docs.rc.uab.edu/
https://docs.rc.uab.edu/
Please use the new documentation url https://docs.rc.uab.edu/ for all Research Computing documentation needs.
As a result of this move, we have deprecated use of this wiki for documentation. We are providing read-only access to the content to facilitate migration of bookmarks and to serve as an historical record. All content updates should be made at the new documentation site. The original wiki will not receive further updates.
Thank you,
The Research Computing Team
IOR Benchmark The IOR software is used for benchmarking parallel file systems using POSIX, MPIIO, or HDF5 interfaces.
These IOR tests were performed on the UAB 4.7 PB gpfs storage fabric for performance estimation.
Downloaded from: https://github.com/LLNL/ior
Before build
Download IOR from github
export LDFLAGS=-L/usr/lpp/mmfs/lib export CPPFLAGS=-I/usr/lpp/mmfs/include LIBS=-lgpfs ./configure
to avoid the gpfs_fcntl issue https://github.com/LLNL/ior/issues/15
Running IOR
Direct Running with out mpi
./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /data/user/tanthony/ior-master/ ./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -numTasks 3 -o /data/user/tanthony/ior-master/ srun -N2 --time=10:00:00 --mem=4096 --partition=medium --job-name=iortest ./ior -f IOR.input IOR -a POSIX -N 3 -b 500m -d 5 -t 128k -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C
Running with mpi
Direct run
module load mpich/ge/gcc/64/3.2 mpiexec -n 3 ./ior -a POSIX -N 3 -b 10m -d 5 -t 256k -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C
Test 0 started: Tue Sep 6 12:01:17 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB Used FS: 1.7% Inodes: 128.0 Mi Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
api = POSIX
test filename = /data/user/tanthony/ior-master/
access = file-per-process
pattern = segmented (1 segment)
ordering in a file = sequential offsets
ordering inter file= constant task offsets = 1
clients = 3 (3 per node)
repetitions = 4
xfersize = 131072 bytes
blocksize = 500 MiB
aggregate filesize = 1.46 GiB
Using Time Stamp 1473181277 (0x57cef65d) for Data Signature
Max Write: 4921.20 MiB/sec (5160.25 MB/sec)
Max Read: 11274.53 MiB/sec (11822.20 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 4921.20 4086.66 4646.19 332.18 0.32463 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0
read 11274.53 11176.68 11229.53 38.65 0.13358 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0
Finished: Tue Sep 6 12:01:58 2016
Increasing number of threads
mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o /data/user/tanthony/ior-master/ -vv -k -wWr -C -F -i4
Using synchronized MPI timer
Start time skew across all tasks: 0.00 sec
Test 0 started: Tue Sep 6 12:17:53 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB Used FS: 1.7% Inodes: 128.0 Mi Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
api = POSIX
test filename = /data/user/tanthony/ior-master/
access = file-per-process
pattern = segmented (1 segment)
ordering in a file = sequential offsets
ordering inter file= constant task offsets = 1
clients = 3 (3 per node)
repetitions = 4
xfersize = 262144 bytes
blocksize = 1 GiB
aggregate filesize = 3 GiB
Using Time Stamp 1473182273 (0x57cefa41) for Data Signature
Max Write: 6295.34 MiB/sec (6601.14 MB/sec)
Max Read: 10185.52 MiB/sec (10680.30 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 6295.34 6139.65 6225.21 55.59 0.49352 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0
read 10185.52 9304.62 9676.96 325.32 0.31781 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0
Running via srun
srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C
Increasing the number of nodes
- Running with three nodes
srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 2g -d 5 -t 2m -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C Max Write: 11845.79 MiB/sec (12421.21 MB/sec) Max Read: 15105.21 MiB/sec (15838.96 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 11845.79 5360.16 9535.91 2958.26 0.73459 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0 read 15105.21 12168.91 13998.89 1303.39 0.44299 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0
- Running with 6 nodes
srun -N6 -n6 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 6 ./ior -a POSIX -N 6 -b 3g -d 5 -t 4m -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C Max Write: 20743.88 MiB/sec (21751.53 MB/sec) Max Read: 19156.05 MiB/sec (20086.57 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 20743.88 4791.11 15196.25 7362.94 1.88494 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0 read 19156.05 11148.20 15872.29 3424.16 1.22674 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0
- Running with 6 nodes and 16 tasks
srun -N16 -n16 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 16 ./ior -a POSIX -N 16 -b 4g -d 5 -t 4m -o /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C Max Write: 21229.09 MiB/sec (22260.31 MB/sec) Max Read: 19458.28 MiB/sec (20403.48 MB/sec)