How to Install VPP The following describes how to install VPP on Ubuntu 18.04. For a complete set of instructions click on the button at the bottom of the page. Update the OS It is a good idea to first update and upgrade the OS before starting; run the following command to update the OS: $ sudo bash # apt-get update Point to the Repository Create a file /etc/apt/sources.list.d/99fd.io.list with contents that point to the version needed. In this example we point to the latest release. deb [trusted=yes] https://packagecloud.io/fdio/release/ubuntu bionic main Get the key: # curl -L https://packagecloud.io/fdio/release/gpgkey | sudo apt-key add - Install the Mandatory Packages Install the mandatory packages by running the following commands: # sudo apt-get update # sudo apt-get install vpp vpp-plugin-core vpp-plugin-dpdk Install the Optional Packages Install the optional packages by running the following command: # sudo apt-get install vpp-api-python python3-vpp-api vpp-dbg vpp-dev Uninstall the Packages Uninstall the packages by running the following command: # sudo apt-get remove --purge vpp*
root@srv6:~# cat /etc/apt/sources.list.d/99fd.io.list deb [trusted=yes] https://packagecloud.io/fdio/release/ubuntu bionic main root@srv6:~#
ARm64
root@gobgp:~# apt-get install vpp vpp-plugin-core vpp-plugin-dpdk Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libmbedcrypto1 libmbedtls10 libmbedx509-0 libvppinfra The following NEW packages will be installed: libmbedcrypto1 libmbedtls10 libmbedx509-0 libvppinfra vpp vpp-plugin-core vpp-plugin-dpdk 0 upgraded, 7 newly installed, 0 to remove and 141 not upgraded. Need to get 6,723 kB of archives. After this operation, 35.4 MB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 libmbedcrypto1 arm64 2.8.0-1 [121 kB] Get:3 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 libmbedx509-0 arm64 2.8.0-1 [31.4 kB] Get:4 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 libmbedtls10 arm64 2.8.0-1 [55.8 kB] Get:2 https://packagecloud.io/fdio/release/ubuntu bionic/main arm64 libvppinfra arm64 20.05.1-release [112 kB] Get:5 https://packagecloud.io/fdio/release/ubuntu bionic/main arm64 vpp arm64 20.05.1-release [2,114 kB] Get:6 https://packagecloud.io/fdio/release/ubuntu bionic/main arm64 vpp-plugin-core arm64 20.05.1-release [1,621 kB] Get:7 https://packagecloud.io/fdio/release/ubuntu bionic/main arm64 vpp-plugin-dpdk arm64 20.05.1-release [2,667 kB] Fetched 6,723 kB in 14s (475 kB/s) Selecting previously unselected package libmbedcrypto1:arm64. (Reading database ... 94440 files and directories currently installed.) Preparing to unpack .../0-libmbedcrypto1_2.8.0-1_arm64.deb ... Unpacking libmbedcrypto1:arm64 (2.8.0-1) ... Selecting previously unselected package libmbedx509-0:arm64. Preparing to unpack .../1-libmbedx509-0_2.8.0-1_arm64.deb ... Unpacking libmbedx509-0:arm64 (2.8.0-1) ... Selecting previously unselected package libmbedtls10:arm64. Preparing to unpack .../2-libmbedtls10_2.8.0-1_arm64.deb ... Unpacking libmbedtls10:arm64 (2.8.0-1) ... Selecting previously unselected package libvppinfra. Preparing to unpack .../3-libvppinfra_20.05.1-release_arm64.deb ... Unpacking libvppinfra (20.05.1-release) ... Selecting previously unselected package vpp. Preparing to unpack .../4-vpp_20.05.1-release_arm64.deb ... Unpacking vpp (20.05.1-release) ... Selecting previously unselected package vpp-plugin-core. Preparing to unpack .../5-vpp-plugin-core_20.05.1-release_arm64.deb ... Unpacking vpp-plugin-core (20.05.1-release) ... Selecting previously unselected package vpp-plugin-dpdk. Preparing to unpack .../6-vpp-plugin-dpdk_20.05.1-release_arm64.deb ... Unpacking vpp-plugin-dpdk (20.05.1-release) ... Setting up libvppinfra (20.05.1-release) ... Setting up libmbedcrypto1:arm64 (2.8.0-1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Setting up libmbedx509-0:arm64 (2.8.0-1) ... Setting up vpp (20.05.1-release) ... * Applying /etc/sysctl.d/10-console-messages.conf ... kernel.printk = 4 4 1 7 * Applying /etc/sysctl.d/10-ipv6-privacy.conf ... net.ipv6.conf.all.use_tempaddr = 2 net.ipv6.conf.default.use_tempaddr = 2 * Applying /etc/sysctl.d/10-kernel-hardening.conf ... kernel.kptr_restrict = 1 * Applying /etc/sysctl.d/10-link-restrictions.conf ... fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /etc/sysctl.d/10-lxd-inotify.conf ... fs.inotify.max_user_instances = 1024 * Applying /etc/sysctl.d/10-magic-sysrq.conf ... kernel.sysrq = 176 * Applying /etc/sysctl.d/10-network-security.conf ... net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.tcp_syncookies = 1 * Applying /etc/sysctl.d/10-ptrace.conf ... kernel.yama.ptrace_scope = 1 * Applying /etc/sysctl.d/10-zeropage.conf ... vm.mmap_min_addr = 32768 * Applying /usr/lib/sysctl.d/50-default.conf ... net.ipv4.conf.all.promote_secondaries = 1 net.core.default_qdisc = fq_codel * Applying /etc/sysctl.d/80-router.conf ... net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 * Applying /etc/sysctl.d/80-vpp.conf ... vm.nr_hugepages = 1024 vm.max_map_count = 3096 vm.hugetlb_shm_group = 0 kernel.shmmax = 2147483648 * Applying /etc/sysctl.d/90-mpls-router.conf ... net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 net.ipv4.conf.all.rp_filter = 0 net.ipv4.tcp_l3mdev_accept = 1 net.ipv4.udp_l3mdev_accept = 1 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.conf ... Created symlink /etc/systemd/system/multi-user.target.wants/vpp.service → /lib/systemd/system/vpp.service. Setting up libmbedtls10:arm64 (2.8.0-1) ... Setting up vpp-plugin-dpdk (20.05.1-release) ... Setting up vpp-plugin-core (20.05.1-release) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... root@gobgp:~# uname -a Linux gobgp 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:10:24 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux root@gobgp:~# c
root@gobgp:~# cat /lib/systemd/system/vpp.service [Unit] Description=vector packet processing engine After=network.target [Service] Type=simple ExecStartPre=-/sbin/modprobe uio_pci_generic ExecStart=/usr/bin/vpp -c /etc/vpp/startup.conf ExecStopPost=/bin/rm -f /dev/shm/db /dev/shm/global_vm /dev/shm/vpe-api Restart=always # Uncomment the following line to enable VPP coredumps on crash # You still need to configure the rest of the system to collect them, see # https://fdio-vpp.readthedocs.io/en/latest/troubleshooting/reportingissues/reportingissues.html#core-files # for details #LimitCORE=infinity [Install] WantedBy=multi-user.target root@gobgp:~#
ExecStartPre=-/sbin/modprobe uio_pci_generic
变为
ExecStartPre=-/sbin/modprobe vfio-pci
root@gobgp:~# modprobe uio_pci_generic root@gobgp:~# modprobe vfio-pci root@gobgp:~# lsmod | grep io_pci vfio_pci 53248 0 vfio_virqfd 16384 1 vfio_pci vfio 40960 2 vfio_iommu_type1,vfio_pci uio_pci_generic 16384 0 uio 20480 1 uio_pci_generic root@gobgp:~#
/etc/vpp/startup.conf
root@gobgp:~# mkdir /var/log/vpp -p
root@gobgp:~#
打开dpdk 设置驱动，这里使用vfio-pci，后面还有一篇文章针对igb_uio的 dpdk { 这里使用vfio-pci # 注意， vmxnet3类型的只能使用vfio-pci，且不要在配置文件里接管接口 uio-driver vfio-pci ... }
root@gobgp:~# ethtool -i enp7s0 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:07:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no You also have python3 installed, you can run 'python3' instead. root@gobgp:~# python3 dpdk-devbind.py --bind vfio-pci 07:00.0 Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci Error: unbind failed for 0000:07:00.0 - Cannot open /sys/bus/pci/drivers//unbind root@gobgp:~#
root@gobgp:~# python3 dpdk-devbind.py --status Network devices using kernel driver =================================== 0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci,uio_pci_generic *Active* Other Network devices ===================== 0000:07:00.0 'Virtio network device 1041' unused=vfio-pci,uio_pci_generic No 'Baseband' devices detected ============================== No 'Crypto' devices detected ============================ No 'Eventdev' devices detected ============================== No 'Mempool' devices detected ============================= No 'Compress' devices detected ============================== No 'Misc (rawdev)' devices detected =================================== root@gobgp:~#
root@gobgp:~# python3 dpdk-devbind.py --bind vfio-pci 01:00.0 Warning: routing table indicates that interface 0000:01:00.0 is active. Not modifying root@gobgp:~#
绑定设备 # ./usertools/dpdk-devbind.py --bind=igb_uio 0000:00:07.0 Routing table indicates that interface 0000:00:07.0 is active. Not modifying # ./usertools/dpdk-devbind.py --force --bind=igb_uio 0000:00:07.0 # ./usertools/dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:00:07.0 'Virtio network device 1000' drv=igb_uio unused=virtio_pci Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio *Active* 第一条我使用命令将PCI为0000:00:07.0的设备绑定到igb_uio的dpdk驱动上，显示该设备是active，不允许修改；第二条命令加了一个--force之后成功绑定；第三条命令显示当前设备的状态，可以看到有一个DPDK兼容的驱动设备，还有一个内核驱动的设备。 注：绑定时一定要避免正在使用ssh登陆的网卡，否则绑定之后网络将会中断，只能重启主机来恢复环境了。 解绑设备 # ./usertools/dpdk-devbind.py -u 0000:00:07.0 # ./usertools/dpdk-devbind.py --status-dev net Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio *Active* Other Network devices ===================== 0000:00:07.0 'Virtio network device 1000' unused=virtio_pci,igb_uio # ./usertools/dpdk-devbind.py --bind=virtio-pci 0000:00:07.0 # ./usertools/dpdk-devbind.py --status-dev net Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci,igb_uio *Active* 0000:00:07.0 'Virtio network device 1000' if=eth1 drv=virtio-pci unused=virtio_pci,igb_uio 第一条命令将igb_uio驱动的设备进行解绑，第二条命令查看状态可以看到该设备当前没有绑定到任何驱动上，第三条命令将该设备绑回virtio-pci的内核驱动，第四条命令可以看到已经绑回去。
root@gobgp:~# dmesg |tail [ 4.625463] VFIO - User Level meta-driver version: 0.3 [ 161.664269] vfio-pci: probe of 0000:07:00.0 failed with error -22 [ 161.664306] vfio-pci: probe of 0000:07:00.0 failed with error -22 root@gobgp:~#
iommu,嵌套或者开vfio非安全模式就可以了
系统驱动如果支持MSI-X，就无法使用驱动uio_pci_generic，必须使用vfio-pci。 在没有OIMMU的系统上，vfio驱动也能被用到 内核4.15.0-33-generic kernel (Ubuntu 18.04)非iommu模式下。
绑定uio_pci_generic
root@gobgp:~# python3 dpdk-devbind.py --force --bind=vfio-pci 0000:07:00.0 Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci Error: unbind failed for 0000:07:00.0 - Cannot open /sys/bus/pci/drivers//unbind root@gobgp:~# python3 dpdk-devbind.py --force --bind=uio_pci_generic 0000:07:00.0 Error: Driver 'uio_pci_generic' is not loaded. root@gobgp:~# modprobe uio_pci_generic root@gobgp:~# python3 dpdk-devbind.py --force --bind=uio_pci_generic 0000:07:00.0 root@gobgp:~#
root@gobgp:~# python3 dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:07:00.0 'Virtio network device 1041' drv=uio_pci_generic unused=vfio-pci Network devices using kernel driver =================================== 0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci,uio_pci_generic *Active* root@gobgp:~#
root@gobgp:~# systemctl status vpp ● vpp.service - vector packet processing engine Loaded: loaded (/lib/systemd/system/vpp.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-08-21 10:33:45 +08; 56s ago Process: 2308 ExecStopPost=/bin/rm -f /dev/shm/db /dev/shm/global_vm /dev/shm/vpe-api (code=exited, status=0/SUC Process: 2303 ExecStart=/usr/bin/vpp -c /etc/vpp/startup.conf (code=exited, status=1/FAILURE) Process: 2297 ExecStartPre=/sbin/modprobe uio_pci_generic (code=exited, status=0/SUCCESS) Main PID: 2303 (code=exited, status=1/FAILURE) Aug 21 10:33:45 gobgp systemd[1]: vpp.service: Failed with result 'exit-code'. Aug 21 10:33:45 gobgp systemd[1]: vpp.service: Service hold-off time over, scheduling restart. Aug 21 10:33:45 gobgp systemd[1]: vpp.service: Scheduled restart job, restart counter is at 5. Aug 21 10:33:45 gobgp systemd[1]: Stopped vector packet processing engine. Aug 21 10:33:45 gobgp systemd[1]: vpp.service: Start request repeated too quickly. Aug 21 10:33:45 gobgp systemd[1]: vpp.service: Failed with result 'exit-code'. Aug 21 10:33:45 gobgp systemd[1]: Failed to start vector packet processing engine. root@gobgp:~# /usr/bin/vpp -c /etc/vpp/startup.conf vlib_call_all_config_functions: unknown input `uio-driver uio_pci_generic' root@gobgp:~#
root@gobgp:~# /usr/bin/vpp -c /etc/vpp/startup.conf /usr/bin/vpp[2333]: clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory /usr/bin/vpp[2333]: vlib_pci_bind_to_uio: Skipping PCI device 0000:01:00.0 as host interface enp1s0 is up /usr/bin/vpp[2333]: dpdk: EAL init args: -c 2 -n 4 --in-memory --file-prefix vpp -b 0000:01:00.0 --master-lcore 1
root@gobgp:~# vppctl _______ _ _ _____ ___ __/ __/ _ (_)__ | | / / _ / _ _/ _// // / / / _ | |/ / ___/ ___/ /_/ /____(_)_/\___/ |___/_/ /_/ vpp# show version vpp v20.05.1-release built by root on ff1827471f2b at 2020-07-15T20:14:36 vpp#
root@gobgp:~# vppctl show pci Address Sock VID:PID Link Speed Driver Product Name Vital Product Data 0000:01:00.0 0 1af4:1041 2.5 GT/s x1 virtio-pci 0000:07:00.0 0 1af4:1041 2.5 GT/s x1 uio_pci_generic root@gobgp:~# vppctl show interface addr GigabitEthernet7/0/0 (dn): local0 (dn): root@gobgp:~#
源码编译