Overlapping Computation & Communication with MPI Non-blocking Calls
The following recommendations are verified with cray-mpich/7.2.0 on CLE 5.2 UP02
The following recommendations may help with MPI non-blocking calls. They have been verified to show improvement with MPI_Iallreduce.
Enable core specialization with aprun
aprun -r 1 # Enable core specialization.
Enable asynchronous progress engine & set MPI thread safety level
export MPICH_NEMESIS_ASYNC_PROGRESS=SC # Enable async progress engine
export MPICH_MAX_THREAD_SAFETY=multiple # MPICH thread safety
Enable MPI shared memory optimizations
export MPICH_SHARED_MEM_COLL_OPT=1 # optimized shared-memory based design for collective operations. Currently supported collective operations are: MPI_Allreduce, MPI_Iallreduce, and MPI_Bcast. 1 enables all.
export MPICH_SMP_SINGLE_COPY_SIZE=1024 # Specifies the minimum message size in bytes to consider for single-copy transfers for on-node messages. This applies only to the SMP (on-node shared memory) device.
?
Link with DMAPP library with following link flags
Static linking:
-Wl,--whole-archive,-ldmapp,--no-whole-archive
Dynamic linking:
-ldmapp
Enable optimized DMAPP collectives
export MPICH_USE_DMAPP_COLL=1 # attempt to use the highly optimized GHAL-based DMAPP collective algorithms, if available.
?Debugging:
export MPICH_GNI_ASYNC_PROGRESS_STATS=enabled # Generates a detailed log. May result in a large stderr file.
export MPICH_ENV_DISPLAY=1 # rank 0 to display all MPICH environment variables and their current settings at MPI initialization time.
export MPICH_VERSION_DISPLAY=1 # display the CRAY MPICH version number as well as build date information.
Results:
The following results show overlap % for various message sizes and rank count
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V4.0.0, MPI-NBC part
#---------------------------------------------------
# Date : Wed Jun 10 13:22:27 2015
# Machine : x86_64
# System : Linux
# Release : 3.0.101-0.31.1_1.0502.8394-cray_gem_c
# Version : #1 SMP Mon Dec 22 19:59:41 UTC 2014
# MPI Version : 3.0
# MPI Thread Environment:
# New default behavior from Version 3.2 on:
# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time
# Calling sequence was:
# ./IMB-NBC Iallreduce Iallreduce_pure
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Iallreduce
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 2
# ( 510 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 1.90 1.00 0.76 0.00
4 1000 14.90 8.23 8.29 19.64
8 1000 14.34 8.13 8.34 25.63
16 1000 14.28 8.11 8.31 25.77
32 1000 5.18 2.42 2.19 0.00
64 1000 5.30 2.54 2.30 0.00
128 1000 5.35 2.62 2.31 0.00
256 1000 6.23 2.71 3.01 0.00
512 1000 6.58 3.01 3.02 0.00
1024 1000 25.13 9.17 9.25 0.00
2048 1000 26.93 9.73 9.75 0.00
4096 1000 39.37 14.86 15.90 0.00
8192 1000 45.68 23.98 25.35 14.42
16384 1000 56.51 29.26 30.36 10.27
32768 1000 71.82 39.03 41.71 21.41
65536 640 130.56 64.68 68.07 3.22
131072 320 192.12 128.23 137.54 53.55
262144 160 456.01 333.18 355.34 65.43
524288 80 877.91 648.10 694.36 66.90
1048576 40 1792.55 1319.45 1403.14 66.28
2097152 20 3643.30 2618.75 2707.99 62.17
4194304 10 7861.61 6294.99 6681.63 76.55
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 4
# ( 508 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 1.96 1.03 0.76 0.00
4 1000 14.77 8.21 8.35 21.36
8 1000 14.84 8.32 8.33 21.81
16 1000 14.91 8.40 8.39 22.38
32 1000 8.02 3.74 3.58 0.00
64 1000 8.80 4.03 4.32 0.00
128 1000 8.88 4.03 4.30 0.00
256 1000 9.09 4.34 4.30 0.00
512 1000 10.23 5.12 5.00 0.00
1024 1000 49.12 22.65 22.98 0.00
2048 1000 51.45 19.53 20.89 0.00
4096 1000 85.29 44.52 47.02 13.30
8192 1000 97.68 57.76 61.46 35.04
16384 1000 100.50 48.92 51.10 0.00
32768 1000 133.16 75.88 80.46 28.81
65536 640 208.02 125.01 135.63 38.80
131072 320 320.20 203.18 216.79 46.02
262144 160 552.87 387.50 418.35 60.47
524288 80 1083.17 784.31 837.23 64.30
1048576 40 2667.68 1831.70 1957.51 57.29
2097152 20 5108.64 3597.90 3820.80 60.46
4194304 10 10313.20 8468.91 9063.70 79.65
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 8
# ( 504 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 1.98 1.08 0.79 0.00
4 1000 16.31 9.35 8.99 21.74
8 1000 16.34 9.35 8.99 21.34
16 1000 17.11 9.50 9.67 21.35
32 1000 10.83 5.23 4.93 0.00
64 1000 11.79 5.46 5.78 0.00
128 1000 12.00 5.60 5.79 0.00
256 1000 13.05 6.05 5.68 0.00
512 1000 15.45 7.46 7.79 0.00
1024 1000 88.62 54.16 56.41 38.92
2048 1000 91.67 56.92 59.94 42.03
4096 1000 117.08 68.63 72.45 33.13
8192 1000 155.92 100.22 105.75 47.34
16384 1000 169.91 105.55 111.33 42.19
32768 1000 206.10 124.65 132.13 38.36
65536 640 282.44 175.18 188.25 43.02
131072 320 451.45 293.52 318.08 50.35
262144 160 814.90 586.73 637.70 64.22
524288 80 1525.50 1116.74 1217.77 66.43
1048576 40 2812.62 2110.67 2290.47 69.35
2097152 20 5562.60 4368.10 4736.86 74.78
4194304 10 11493.40 9449.10 10221.51 80.00
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 16
# ( 496 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 1.97 0.99 0.78 0.00
4 1000 23.98 13.42 13.79 23.40
8 1000 23.31 13.23 13.12 23.03
16 1000 23.58 13.33 13.14 21.64
32 1000 17.37 8.55 8.40 0.00
64 1000 18.41 9.06 9.36 0.07
128 1000 19.42 9.59 9.79 0.00
256 1000 20.89 10.66 10.55 3.04
512 1000 25.39 13.18 13.29 8.11
1024 1000 104.53 64.62 68.77 41.98
2048 1000 111.75 69.24 73.65 42.28
4096 1000 153.60 88.38 90.89 28.24
8192 1000 204.74 119.83 124.31 31.69
16384 1000 231.00 133.90 140.69 30.98
32768 1000 283.16 163.01 172.74 30.45
65536 640 401.26 234.56 247.10 32.54
131072 320 614.08 399.35 427.43 49.76
262144 160 1542.62 1202.69 1307.54 74.00
524288 80 1799.06 1249.39 1350.63 59.30
1048576 40 3540.18 2597.12 2869.92 67.14
2097152 20 7238.65 5102.35 5448.03 60.79
4194304 10 14614.80 11747.81 12562.39 77.18
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 32
# ( 480 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 2.02 1.11 0.75 0.00
4 1000 26.12 14.94 15.33 27.10
8 1000 26.55 15.09 15.36 25.44
16 1000 26.55 15.13 15.47 26.21
32 1000 56.41 30.47 31.51 17.71
64 1000 59.09 31.82 32.48 16.05
128 1000 60.26 31.61 32.79 12.63
256 1000 62.16 32.38 33.32 10.61
512 1000 66.90 35.50 36.33 13.57
1024 1000 132.19 87.74 92.36 51.88
2048 1000 143.53 95.89 101.80 53.20
4096 1000 233.79 134.12 141.19 29.41
8192 1000 296.50 175.57 179.41 32.59
16384 1000 319.48 187.27 196.92 32.86
32768 1000 368.47 213.33 221.82 30.06
65536 640 492.14 291.76 309.23 35.20
131072 320 684.80 447.48 480.06 50.57
262144 160 1126.11 755.92 811.77 54.40
524288 80 2924.45 2255.89 2435.89 72.55
1048576 40 4575.87 3269.03 3582.63 63.52
2097152 20 8135.84 6039.50 6651.46 68.48
4194304 10 15998.10 12681.91 13849.74 76.06
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 64
# ( 448 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 2.04 1.15 0.76 0.00
4 1000 29.01 17.49 17.86 35.52
8 1000 29.03 17.67 17.89 36.53
16 1000 30.20 17.80 18.35 32.39
32 1000 73.38 38.22 39.07 10.02
64 1000 74.97 39.25 39.98 10.66
128 1000 76.75 39.92 41.26 10.76
256 1000 78.64 40.89 43.21 12.64
512 1000 83.92 43.38 45.37 10.64
1024 1000 154.92 105.69 111.82 55.97
2048 1000 172.74 118.62 124.87 56.66
4096 1000 274.13 156.72 164.95 28.82
8192 1000 347.89 206.35 214.72 34.08
16384 1000 370.66 217.45 225.99 32.21
32768 1000 413.57 237.15 251.72 29.91
65536 640 545.39 319.63 337.08 33.02
131072 320 753.33 488.38 519.44 48.99
262144 160 1737.74 1251.31 1299.16 62.56
524288 80 2977.36 2254.06 2367.65 69.45
1048576 40 4748.08 3493.15 3702.74 66.11
2097152 20 8336.81 6419.29 6859.48 72.05
4194304 10 16571.00 13475.70 14752.53 79.02
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 128
# ( 384 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 2.04 1.09 0.78 0.00
4 1000 30.45 19.34 19.24 41.99
8 1000 31.35 19.46 19.93 40.37
16 1000 31.67 19.67 20.53 41.52
32 1000 87.47 44.39 46.79 7.95
64 1000 88.55 45.19 46.93 7.60
128 1000 90.29 46.80 48.85 10.96
256 1000 93.93 47.70 48.93 5.52
512 1000 99.08 51.59 53.88 11.87
1024 1000 195.37 139.19 146.23 61.58
2048 1000 208.54 142.09 150.35 55.80
4096 1000 309.73 173.60 182.41 25.37
8192 1000 390.39 230.71 242.45 34.14
16384 1000 407.32 236.79 248.74 31.45
32768 1000 450.14 256.92 271.70 28.89
65536 640 585.12 341.30 359.84 32.24
131072 320 825.88 513.12 549.32 43.06
262144 160 1953.49 1313.58 1388.67 53.92
524288 80 3207.60 2371.51 2500.62 66.56
1048576 40 5129.15 3738.55 4064.95 65.79
2097152 20 9047.15 6680.85 6916.02 65.79
4194304 10 18619.20 13818.79 15136.36 68.29
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 256
# ( 256 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 2.07 1.13 0.78 0.00
4 1000 31.49 19.95 19.94 42.12
8 1000 32.18 20.12 20.58 41.37
16 1000 32.56 20.31 21.06 41.81
32 1000 112.31 60.96 61.83 16.96
64 1000 111.23 59.58 61.60 16.16
128 1000 109.10 55.40 57.23 6.17
256 1000 115.20 60.47 62.23 12.04
512 1000 118.41 60.85 62.80 8.36
1024 1000 209.41 142.46 148.86 55.03
2048 1000 237.63 163.81 172.69 57.25
4096 1000 344.82 192.93 198.02 23.30
8192 1000 427.11 250.26 262.61 32.66
16384 1000 446.48 258.61 267.91 29.87
32768 1000 490.09 280.44 295.29 29.00
65536 640 622.08 363.85 383.41 32.65
131072 320 892.98 532.36 558.42 35.42
262144 160 2272.04 1694.26 1788.40 67.69
524288 80 3432.19 2461.62 2507.14 61.29
1048576 40 5334.58 3799.22 4057.78 62.16
2097152 20 9105.75 6808.46 7253.48 68.33
4194304 10 17817.90 14020.30 15345.43 75.25
#-----------------------------------------------------------------------------
# Benchmarking Iallreduce
# #processes = 512
#-----------------------------------------------------------------------------
#bytes #repetitions t_ovrl[usec] t_pure[usec] t_CPU[usec] overlap[%]
0 1000 2.14 1.09 0.77 0.00
4 1000 33.53 21.75 21.94 46.32
8 1000 34.28 21.93 21.92 43.66
16 1000 35.75 22.59 22.60 41.77
32 1000 132.41 70.23 73.37 15.25
64 1000 129.62 67.00 68.07 8.01
128 1000 130.41 67.86 69.37 9.83
256 1000 131.01 66.39 69.32 6.78
512 1000 138.63 73.31 77.14 15.31
1024 1000 323.16 253.45 263.74 73.57
2048 1000 342.64 263.96 279.01 71.80
4096 1000 424.28 242.21 253.95 28.30
8192 1000 504.31 309.63 321.41 39.43
16384 1000 518.99 312.61 327.53 36.99
32768 1000 560.26 333.12 347.17 34.57
65536 640 687.74 412.09 435.21 36.66
131072 320 994.97 611.65 644.15 40.49
262144 160 2419.55 1789.48 1893.83 66.73
524288 80 3542.32 2525.26 2685.30 62.12
1048576 40 5559.43 3904.75 4200.27 60.61
2097152 20 9645.30 7030.95 7644.80 65.80
4194304 10 19036.89 14211.01 15473.82 68.81
| 
