VMA On K1 Powerlinux

VMA On K1 PowerLinux

VMA是一个消息加速器messaging accelerator(MPI-Massage passing Interface);The VMA library is delivered as a user-space library
install the default VMA version compiled with MLNX_OFED

环境背景

一台FP5280G2 做server 一台FP5280G2 做client
2块Mellanox的卡 CCAT_LC_PCIE16X_100G_XR_D 用光纤线直连。

测试架构

安装mellanox 套件 mlnx_ofed

安装依赖库

yum install python-devel lsof redhat-rpm-config rpm-build gcc kernel-devel-4.14.0-49.el7a.ppc64le
yum install tcl gcc-gfortran tcsh tk pciutils createrepo
./mlnxofedinstall --add-kernel-support --skip-distro-check
/etc/init.d/openibd restart

安装AT11

手动安装如下安装包
advance-toolchain-atX.X-runtime
advance-toolchain-atX.X-devel
advance-toolchain-atX.X-perf
advance-toolchain-atX.X-mcore-libs
下载AT 11地址https://developer.ibm.com/linuxonpower/advance-toolchain/

安装sockperf

yum install sockperf-3.5-1.el7.ppc64le.rpm
下载地址
http://mirror.hmc.edu/epel/7/ppc64le/Packages/s/

编译安装libVMA

yum install Autoconf Automake libtool unzip patch libnl-devel gcc
yum -y install libnl3-devel libnl3-route-devel gcc-c++-4.8.5-28.el7.ppc64le libibverbs-devel
1. ./autogen.sh
2. ./configure --with-ofed=/usr --prefix=/usr --libdir=/opt/at11.0/lib64 --includedir=/opt/at11.0/include --docdir=/usr/share/doc/libvma --sysconfdir=/etc
3. make
4.make install
配置网卡
nmcli conn add type ethernet con-name enP51p1s0f1 ifname enP51p1s0f1 ip4 192.168.100.101/24
运行测试工具
yum -y install numactl
systemctl stop firewalld
Server：
VMA_MTU=200 VMA_RX_BUFS=30000 VMA_RX_POLL_OS_RATIO=0 VMA_SELECT_POLL_OS=0 VMA_SPEC=latency LD_PRELOAD=/opt/at11.0/lib64/libvma.so numactl --cpunodebind=8 taskset -c 156 sockperf sr -i 192.168.100.100 -p 12345 --tcp --msg-size 14
Client：
VMA_MTU=200 VMA_RX_BUFS=30000 VMA_RX_POLL_OS_RATIO=0 VMA_SELECT_POLL_OS=0 VMA_SPEC=latency LD_PRELOAD=/opt/at11.0/lib64/libvma.so numactl --cpunodebind=8 taskset -c 172 sockperf pp -i 192.168.100.100 -p 12345 --tcp --msg-size 14

调优

设置大页面

# echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages
# echo 4096 > /sys/devices/system/node/node8/hugepages/hugepages-16384kB/nr_hugepages
#mount -t hugetlbfs nodev /mnt/hugevim /etc/modprobe.d/mlnx.conf
options mlx4_core log_num_mgm_entry_size=-1 VMA_MEM_ALLOC_TYPE=1
/etc/init.d/openibd restart

关闭SMT

ppc64_cpu –smt=off

设置CPU模式

cpupower frequency-set --governor performance
vim /etc/tuned/tuned-main.conf
sleep_interval = 1 ===> change to 100
update_interval = 10 ===> change to 10000

vim /usr/lib/tuned/network-latency/tuned.conf
[bootloader]
cmdline = audit=0 idle=poll nosoftlockup mce=ignore_ce

减少系统调用

echo 100000000 > /proc/sys/kernel/sched_min_granularity_ns
echo 50000000 > /proc/sys/kernel/sched_migration_cost_ns
echo 0 > /proc/sys/vm/swappiness
sysctl -w vm.swappiness=0
sysctl -w vm.zone_reclaim_mode=0
echo never > /sys/kernel/mm/transparent_hugepage/enabled

更改网卡MTU的值

echo “200” > /sys/class/net/enP51p1s0f1/mtu
MLNX设置
mlnx_tune -p LOW_LATENCY_VMA
软件版本

Testing Envrionment

Hardware	Two FP5280G2 with Two CX5 MCX516A-CDAT
OS Version	rhel-alt-server-7.5-ppc64le
Kernel Version	4.14.0-49.el7a.ppc64le
CX5 OFED	MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5alternate-ppc64le
VMA Version	libvma-8.7.1 https://github.com/Mellanox/libvma
Sockperf	sockperf-3.5-1

yum –y install numactl

Client：

VMA_MTU=200 VMA_RX_BUFS=30000 VMA_RX_POLL_OS_RATIO=0 VMA_SELECT_POLL_OS=0 VMA_SPEC=latency LD_PRELOAD=/opt/at11.0/lib64/libvma.so numactl --cpunodebind=8 taskset -c 172 sockperf pp -i 192.168.100.100 -p 12345 --tcp --msg-size 14

Server:

VMA_MTU=200 VMA_RX_BUFS=30000 VMA_RX_POLL_OS_RATIO=0 VMA_SELECT_POLL_OS=0 VMA_SPEC=latency LD_PRELOAD=/opt/at11.0/lib64/libvma.so numactl --cpunodebind=8 taskset -c 156 sockperf sr -i 192.168.100.100 -p 12345 --tcp --msg-size 14

测试结果

VMA成功在哦K1 PowerLinux上部署，TCP协议下不同字节大小的延迟如数据如下。

调优参考：
https://community.mellanox.com/docs/DOC-2797