Ior2016

From Cheaha
Revision as of 15:04, 13 October 2016 by Tanthony@uab.edu (talk | contribs) (Formatted the page)
Jump to navigation Jump to search


Attention: Research Computing Documentation has Moved
https://docs.rc.uab.edu/


Please use the new documentation url https://docs.rc.uab.edu/ for all Research Computing documentation needs.


As a result of this move, we have deprecated use of this wiki for documentation. We are providing read-only access to the content to facilitate migration of bookmarks and to serve as an historical record. All content updates should be made at the new documentation site. The original wiki will not receive further updates.

Thank you,

The Research Computing Team

IOR Benchmark The IOR software is used for benchmarking parallel file systems using POSIX, MPIIO, or HDF5 interfaces.

These IOR tests were performed on the UAB 4.7 PB gpfs storage fabric for performance estimation.

Downloaded from: https://github.com/LLNL/ior

Before build

Download IOR from github

export LDFLAGS=-L/usr/lpp/mmfs/lib
export CPPFLAGS=-I/usr/lpp/mmfs/include
LIBS=-lgpfs ./configure 

to avoid the gpfs_fcntl issue https://github.com/LLNL/ior/issues/15

Running IOR

Direct Running with out mpi

 
./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /data/user/tanthony/ior-master/

 ./ior -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -numTasks 3 -o /data/user/tanthony/ior-master/

 srun -N2 --time=10:00:00 --mem=4096 --partition=medium --job-name=iortest  ./ior -f IOR.input

IOR -a POSIX -N 3 -b 500m -d 5 -t 128k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C

Running with mpi

Direct run

module load mpich/ge/gcc/64/3.2 
mpiexec -n 3 ./ior -a POSIX -N 3 -b 10m -d 5 -t 256k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C 
Test 0 started: Tue Sep  6 12:01:17 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB   Used FS: 1.7%   Inodes: 128.0 Mi   Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
    api                = POSIX
    test filename      = /data/user/tanthony/ior-master/
    access             = file-per-process
    pattern            = segmented (1 segment)
    ordering in a file = sequential offsets
    ordering inter file= constant task offsets = 1
    clients            = 3 (3 per node)
    repetitions        = 4
    xfersize           = 131072 bytes
    blocksize          = 500 MiB
    aggregate filesize = 1.46 GiB
Using Time Stamp 1473181277 (0x57cef65d) for Data Signature
Max Write: 4921.20 MiB/sec (5160.25 MB/sec)
Max Read:  11274.53 MiB/sec (11822.20 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        4921.20    4086.66    4646.19     332.18    0.32463 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0
read        11274.53   11176.68   11229.53      38.65    0.13358 0 3 3 4 1 1 1 0 0 1 524288000 131072 1572864000 POSIX 0

Finished: Tue Sep  6 12:01:58 2016

Increasing number of threads

mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o  /data/user/tanthony/ior-master/ -vv -k -wWr -C -F -i4
Using synchronized MPI timer
Start time skew across all tasks: 0.00 sec

Test 0 started: Tue Sep  6 12:17:53 2016
Path: /data/user/tanthony/ior-master
FS: 4766.2 TiB   Used FS: 1.7%   Inodes: 128.0 Mi   Used Inodes: 13.6%
Participating tasks: 3
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on c0089
task 1 on c0089
task 2 on c0089
Summary:
    api                = POSIX
    test filename      = /data/user/tanthony/ior-master/
    access             = file-per-process
    pattern            = segmented (1 segment)
    ordering in a file = sequential offsets
    ordering inter file= constant task offsets = 1
    clients            = 3 (3 per node)
    repetitions        = 4
    xfersize           = 262144 bytes
    blocksize          = 1 GiB
    aggregate filesize = 3 GiB
Using Time Stamp 1473182273 (0x57cefa41) for Data Signature

Max Write: 6295.34 MiB/sec (6601.14 MB/sec)
Max Read:  10185.52 MiB/sec (10680.30 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        6295.34    6139.65    6225.21      55.59    0.49352 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0
read        10185.52    9304.62    9676.96     325.32    0.31781 0 3 3 4 1 1 1 0 0 1 1073741824 262144 3221225472 POSIX 0


Running via srun

srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 1g -d 5 -t 256k -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 4 -vv -F -C 


Increasing the number of nodes

  • Running with three nodes
srun -N3 -n3 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 3 ./ior -a POSIX -N 3 -b 2g -d 5 -t 2m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C
Max Write: 11845.79 MiB/sec (12421.21 MB/sec)
Max Read:  15105.21 MiB/sec (15838.96 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write       11845.79    5360.16    9535.91    2958.26    0.73459 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0
read        15105.21   12168.91   13998.89    1303.39    0.44299 0 3 1 3 1 1 1 0 0 1 2147483648 2097152 6442450944 POSIX 0 
  • Running with 6 nodes
srun -N6 -n6 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 6 ./ior -a POSIX -N 6 -b 3g -d 5 -t 4m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C

Max Write: 20743.88 MiB/sec (21751.53 MB/sec)
Max Read:  19156.05 MiB/sec (20086.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write       20743.88    4791.11   15196.25    7362.94    1.88494 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0
read        19156.05   11148.20   15872.29    3424.16    1.22674 0 6 1 3 1 1 1 0 0 1 3221225472 4194304 19327352832 POSIX 0
  • Running with 6 nodes and 16 tasks

srun -N16 -n16 --partition=beta --time=2:00:00 --mem=4096 mpiexec -n 16 ./ior -a POSIX -N 16 -b 4g -d 5 -t 4m -o  /data/user/tanthony/ior-master/ 1 -e -g -w -r -s 1 -i 3 -vv -F -C


Max Write: 21229.09 MiB/sec (22260.31 MB/sec)
Max Read:  19458.28 MiB/sec (20403.48 MB/sec)