mellanox SRIOV

a. Make sure that SR-IOV is enabled in the BIOS of the specific server.

b. vi /boot/grub/grub.conf or vi /boot/grub2/grub.cfg ==> intel_iommu=on
对于我们：
step to update grub.cfg in CentOS
vi /etc/default/grub ==> intel_iommu=on for example :GRUB_CMDLINE_LINUX=" console=tty0 net.ifnames=0 biosdevname=0 rootdelay=90 nomodeset root=UUID=7c3de344-bb4f-4c95-8127-3b2f1e435e8d intel_iommu=on "
update-grub

c. mlnxofedinstall (http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers 不同网卡，不同操作系统，需要的安装包不同。
比如我们的环境下下载此安装包MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu16.04-x86_64，解压，进入目录执行./mlnxofedinstall)

d. Enable SR-IOV on the Firmware

1. mst start
2. mlxconfig -d /dev/mst/mt4115_pciconf0 q
3. mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4 
4. 查看对于的pci
5. reboot

e. Enable SR-IOV on the MLNX_OFED Driver

1.ibstat
      ibdev2netdev 查看active状态的interface
2.cat /sys/class/net/ens785f1/device/sriov_totalvfs   只读文件，vf的数量由I.3设置。若没有此参数，则代表intel_iommu=on 没添加
3.查看当前的VFS：mlx5_num_vfs 和sriov_numvfs的区别在于：无论是否intel_iommu为on，mlx5_num_vfs一直存在，但sriov_numvfs由intel_iommu决定
\# cat /sys/class/infiniband/mlx5_1/device/mlx5_num_vfs
\# cat /sys/class/net/ens785f1/device/sriov_numvfs
\# cat /sys/class/net/ens785f1/device/mlx5_num_vfs
4.set所需的VFS,kernel version不同，方法不同：
echo 4 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
echo 4 > /sys/class/net/ens785f1/device/sriov_numvfs
echo 4 > /sys/class/net/ens785f1/device/mlx5_num_vfs 

参考
https://community.mellanox.com/docs/DOC-2386
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
https://community.mellanox.com/docs/DOC-1502