https://kernelgo.org/vfio-introduction.html
[root@localhost ~]# dmesg -T | grep -i iommu [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:74:03.0 to group 0, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:b4:03.0 to group 1, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:7a:01.0 to group 2, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:ba:01.0 to group 3, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:7a:00.0 to group 4, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:ba:00.0 to group 5, default domain type 3 [Mon Aug 24 02:26:42 2020] iommu: Adding device 0000:7a:02.0 to group 6, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:ba:02.0 to group 7, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:00.0 to group 8, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:08.0 to group 9, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:0c.0 to group 10, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:10.0 to group 11, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:11.0 to group 12, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:00:12.0 to group 13, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:03:00.0 to group 14, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:04:00.0 to group 15, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:04:01.0 to group 16, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:7c:00.0 to group 17, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:80:00.0 to group 18, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:80:08.0 to group 19, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:80:0c.0 to group 20, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:80:10.0 to group 21, default domain type 3 [Mon Aug 24 02:26:43 2020] iommu: Adding device 0000:bc:00.0 to group 22, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:02:00.0 to group 23, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:74:02.0 to group 24, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:81:00.0 to group 25, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:05:00.0 to group 26, default domain type 3 ------------group 26 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:7d:00.0 to group 27, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:84:00.0 to group 28, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:7d:00.1 to group 29, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:08:00.0 to group 30, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:7d:00.2 to group 31, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:7d:00.3 to group 32, default domain type 3 [Mon Aug 24 02:26:45 2020] iommu: Adding device 0000:06:00.0 to group 33, default domain type 3 [Mon Aug 24 02:26:47 2020] iommu: Adding device 0000:74:04.0 to group 34, default domain type 3 [Mon Aug 24 02:26:49 2020] iommu: Adding device 0000:b4:02.0 to group 35, default domain type 3 [Mon Aug 24 02:26:51 2020] iommu: Adding device 0000:b4:04.0 to group 36, default domain type 3 [root@localhost ~]#
[root@localhost ~]# find /sys/kernel/iommu_groups/ -type l | grep '05:00.0' /sys/kernel/iommu_groups/26/devices/0000:05:00.0
[root@localhost dpdk-19.11]# modprobe vfio-pci [root@localhost dpdk-19.11]# ls /dev/vfio/* /dev/vfio/vfio [root@localhost dpdk-19.11]# ./usertools/dpdk-devbind.py --bind=vfio-pci 0000:05:00.0 ---------绑定后还是26 [root@localhost dpdk-19.11]# ls /dev/vfio/* /dev/vfio/26 /dev/vfio/vfio [root@localhost dpdk-19.11]#
[root@localhost dpdk-19.11]# readlink -f /sys/bus/pci/devices/0000:05:00.0/iommu_group /sys/kernel/iommu_groups/26 [root@localhost dpdk-19.11]#
[root@localhost dpdk-19.11]# ls /dev/vfio/* -al crw-------. 1 root root 242, 0 Aug 24 21:41 /dev/vfio/26 crw-rw-rw-. 1 root root 10, 196 Aug 24 21:40 /dev/vfio/vfio [root@localhost dpdk-19.11]#
Linux内核设备驱动充分利用了“一切皆文件”的思想，VFIO驱动也不例外，VFIO中为了方便操作device, group, container等对象将它们和对应的设备文件进行绑定。 VFIO驱动在加载的时候会创建一个名为/dev/vfio/vfio的文件，而这个文件的句柄关联到了vfio_container上，用户态进程打开这个文件就可以初始化和访问vfio_container。 当我们把一个设备直通给虚拟机时，首先要做的就是将这个设备从host上进行解绑，即解除host上此设备的驱动，然后将设备驱动绑定为“vfio-pci”， 在完成绑定后会新增一个/dev/vfio/$groupid的文件，其中$groupid为此PCI设备的iommu group id， 这个id号是在操作系统加载iommu driver遍历扫描host上的PCI设备的时候就已经分配好的，可以使用readlink -f /sys/bus/pci/devices/$bdf/iommu_group来查询。 类似的，/dev/vfio/$groupid这个文件的句柄被关联到vfio_group上，用户态进程打开这个文件就可以管理这个iommu group里的设备。 然而VFIO中并没有为每个device单独创建一个文件，而是通过VFIO_GROUP_GET_DEVICE_FD这个ioctl来获取device的句柄，然后再通过这个句柄来管理设备。
