IB_Switch交换机SB7890配置及Band网卡

1.创建软件存放目录

[root@localhost ~]#

mkdir -p /soft

2.上传OS镜像，本案例以RHEL7.7为例

mount -o loop /soft/rhel-server-7.7-x86_64-dvd.iso /mnt/

3.配置yum源

[root@localhost ~]#

vi /etc/yum.conf

[rheldvd]

name=rhel7.7

baseurl=file:///mnt

enabled=1

gpgcheck=0

[main]

………………………………

4.IB交接机驱动下载或上传已有驱动包至soft目录

驱动包：

MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz

5.解压驱动并安装

cd /soft

tar -xvzf

MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64.tgz

6.安装驱动

yum install pciutils perl gtk2 atk cairo gcc-gfortran tcsh openssl-libs libusbx lsof tcl fuse-libs tk -y

cd /soft/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel7.7-x86_64

./mlnxofedinstall

7.Load HCA驱动

/etc/init.d/openibd restart

8.启动子网管理并设置开机启动

/etc/init.d/opensmd start   ——对于当前SB7890交换不自子网管理，因此只能在OS端开启子网管理服务，若多个节点建议开1——2个节点即可，不建议全部开启；

/sbin/chkconfig opensmd on

/sbin/chkconfig opensmd --list

9.重启OS

reboot

10.查看IB是否正常识别

ifconfig -a

正常识别如下：

ib0: flags=4163  mtu 2044

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

infiniband 00:00:10:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

RX packets 2  bytes 232 (232.0 B)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 0  bytes 0 (0.0 B)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ib1: flags=4099  mtu 4092

Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).

infiniband 00:00:18:87:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)

RX packets 0  bytes 0 (0.0 B)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 0  bytes 0 (0.0 B)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

11.Band IB配置网络

01. Create bond0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-bond0

Note: the only meaningful bonding policy in IPoIB is High-Availability (bonding mode number 1,

or "active-backup").

To learn more about the Linux bonding options click here .

Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"

MTU=2044

02.Update/Create ib0 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib0

Here is an example:

DEVICE=ib0

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=yes

03.Update/Create ib1 Interface on the server. Update the following file:

touch /etc/sysconfig/network-scripts/ifcfg-ib1

Here is an example:

DEVICE=ib1

TYPE=InfiniBand

NM_CONTROLLED=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

PRIMARY=no

04.Create/Update the bond.conf file.

touch /etc/modprobe.d/bond.conf

Configure this file as follows:

alias bond0 bonding # Adapter bonding driver

options bond0 max_bonds=2 miimon=100 mode=1 # 100ms fail-over timer. Mode 1 = Active/Backup

05. Restart the network services and the driver.

#

/etc/init.d/network restart

#

/etc/init.d/openibd restart

06.Verify the bond configuration:

#

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)

Primary Slave: ib0 (primary_reselect always)

Currently Active Slave: ib0

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 100

Down Delay (ms): 100

Slave Interface: ib1

MII Status:

up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: a0:04:03:00:fe:80

Slave queue ID: 0

Slave Interface: ib0

MII Status:

up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 254

Permanent HW addr: a0:04:02:20:fe:80

Slave queue ID: 0

07.Fail Over MAC

Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added

to the bond interface configuration in case of Ethernet interface.

The value active(1) for fail_over_mac policy indicates that the MAC address of the bond should

always be the MAC address of the currently active slave. The MAC address of the slaves is not

changed; instead, the MAC address of the bond changes during a failover.

Changing to fail_over_mac=1 solves this problem as if this not active, the bond MAC jumps from

one interface to another according to the active interface.

Add the bonding options (BONDING_OPT)? "fail_over_mac=1" to /etc/sysconfig/network- scripts/ifcfg-bond0. Note: the default of this parameter is disabled (0).

Here is an example:

DEVICE=bond0

IPADDR=11.11.11.1

NETWORK=11.11.11.0

NETMASK=255.255.255.0

BROADCAST=11.11.11.255

USERCTL=no

BOOTPROTO=none

NM_CONTROLLED=yes

BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100

fail_over_mac=1"

MTU=2044