Mellanox 网卡使用SR | 您所在的位置:网站首页 › sriov无法开启 › Mellanox 网卡使用SR |
Mellanox 网卡使用SR-IOV
一、硬件环境准备
物理服务器为:浪潮 网卡型号:Mellanox Technologies MT27800 1、进入BIOS浪潮服务器为例,开机画面中按“DEL”键; 等待系统进入BIOS 后再进行下一步; 成功进入BIOS的画面
2、主板开启SR-IOV 移动光标到“Advanced”标签页面,再选择“PCI Subsystem Settings”,选中按回车; 再将SR-IOV支持开关设置为“Enabled”;
处理器开启虚拟化支持; 将光标移动到“Processor”标签页,再选中“Processor Configration”; 将“VMX”与“SMX”的值设置为“Enabled”;
再从”Processor ”下找到“IIO Configuration”,进入后将“VT-d”功能打开;
操作系统开启iommu (Centos8.1 为例) [root@localhost ~]# vim /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on iommu=pt" GRUB_DISABLE_RECOVERY="true" GRUB_ENABLE_BLSCFG=true [root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg 开启完成后需要重启机器; 驱动环境准备 1、取得下载合适的安装包进行驱动的安装这里以MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz 为例: 2、解压[root@localhost ~]# tar xzvf MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz 1、查看readme 安装相应的依赖包: [root@localhost ~]#yum install perl pciutils python gcc-gfortran libxml2-python tcsh libnl.i686 libnl expat glib2 tcl libstdc++ bc tk gtk2 atk cairo numactl pkgconfig ethtool 2、驱动安装 [root@localhost ~]#./mlnxofedinstall 3、安装完成后重启机器; 3、配置opensm开启虚拟化[root@localhost docs]# vim /etc/opensm/opensm.conf 添加内容后保存退出: virt_enabled 2 参数的解释: parameter has the following configuration options:
0: Ignore Virtualizations - No virtualization support 1: Disable Virtualization - Disable virtualization on all Virtualization supporting ports 2: Enable Virtualization - Enable (virtualization on all Virtualization supporting ports) 4、系统上开启SRIOV[root@localhost docs] # mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices
[root@localhost docs]# mst status MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded
MST devices: ------------ /dev/mst/mt4119_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:af:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 00 5、查询设备状态信息:[root@localhost docs]# mlxconfig -d /dev/mst/mt4119_pciconf0 q
Device #1: ----------
Device type: ConnectX5 Name: MCX516A-CCA_Ax Description: ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6 Device: /dev/mst/mt4119_pciconf0 Configurations: Next Boot MEMIC_BAR_SIZE 0 MEMIC_SIZE_LIMIT _256KB(1) HOST_CHAINING_MODE DISABLED(0) HOST_CHAINING_DESCRIPTORS Array[0..7] HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7] FLEX_PARSER_PROFILE_ENABLE 0 FLEX_IPV4_OVER_VXLAN_PORT 0 ROCE_NEXT_PROTOCOL 254 ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7] ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7] PF_BAR2_SIZE 0 NON_PREFETCHABLE_PF_BAR False(0) VF_VPD_ENABLE False(0) STRICT_VF_MSIX_NUM False(0) VF_NODNIC_ENABLE False(0) NUM_OF_VFS 4 PF_BAR2_ENABLE False(0) SRIOV_EN True(1) PF_LOG_BAR_SIZE 5 VF_LOG_BAR_SIZE 1 NUM_PF_MSIX 63 NUM_VF_MSIX 11 INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0) 上面的两个参数值需要注意,一个是虚拟机出来的VF数量,一个是开启sriov 功能的开关; 如果需要调整请使用: mlxconfig -d /dev/mst/mt41139pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4 Device #1: ---------- Device type: ConnectI PCI device: /dev/mst/mt4119_pciconf0 Configurations: Current New SRIOV_EN 0 1 NUM_OF_VFS 0 4 FPP_EN 0 1 INT_LOG_MAX_PAYLOAD_SIZE 0 0 Apply new Configuration? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations. 设置完成后需要重启机器; 6、适配设备[root@localhost docs]# mlxfwreset --device /dev/mst/mt4119_pciconf0 reset
Minimal reset level for device, /dev/mst/mt4119_pciconf0:
3: Driver restart and PCI reset Continue with reset?[y/N] y -I- Sending Reset Command To Fw -Done -I- Stopping Driver -Done -I- Resetting PCI -Done -I- Starting Driver -Done -I- Restarting MST -Done -I- FW was loaded successfully 7、设置网卡的vf参数值.[root@localhost docs]# echo 4 >/sys/class/net/ens47f1/device/sriov_numvfs 8、检查是否成功;常用命令检查即可,如ifconfig .ip a ,lspci等; |
CopyRight 2018-2019 实验室设备网 版权所有 |