kata virtio-9p-pci qemu-system-aarch64

利用Qemu-4.0虚拟ARM64实验平台

https://chasinglulu.github.io/2019/07/27/%E5%88%A9%E7%94%A8Qemu-4-0%E8%99%9A%E6%8B%9FARM64%E5%AE%9E%E9%AA%8C%E5%B9%B3%E5%8F%B0/

 

How to launch ARM aarch64 VM with QEMU from scratch.

The below instructions will allow for bringing up an ARM VM from scratch.

It is worth mentioning that our work includes automating all of the below, resulting in a more streamlined option to building and launching the VM. See the article on QEMU aarch64 VMs for more details.

To launch an aarch64 VM we first need to install a few dependencies, including QEMU and the qemu-efi-aarch64 package, which includes the efi firmware.

apt-get install qemu-system-arm
apt-get install qemu-efi-aarch64
apt-get install qemu-utils

Create the flash images with the correct sizes.

dd if=/dev/zero of=flash1.img bs=1M count=64
dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=flash0.img conv=notrunc

Download the image you want to boot.

For our example we use an Ubuntu installer.

wget http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/mini.iso

Create the empty Ubuntu image file we will install Ubuntu into.

We will use 20 gigabytes for this file.

qemu-img create ubuntu-image.img 20G

Start QEMU with the installer.

qemu-system-aarch64 -nographic -machine virt,gic-version=max -m 512M -cpu max -smp 4 
-netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet 
-drive file=ubuntu-image.img,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 
-drive file=mini.iso,if=none,id=drive1,cache=writeback -device virtio-blk,drive=drive1,bootindex=1 
-drive file=flash0.img,format=raw,if=pflash -drive file=flash1.img,format=raw,if=pflash 

Follow the instructions to install Ubuntu to the ubuntu-image.img file.

Once the install is finished you can exit QEMU with -a x.

Then restart QEMU without the installer image with the following command.

qemu-system-aarch64 -nographic -machine virt,gic-version=max -m 512M -cpu max -smp 4 
-netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet 
-drive file=ubuntu-image.img,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 
-drive file=flash0.img,format=raw,if=pflash -drive file=flash1.img,format=raw,if=pflash 

qemu启动参数需要添加：

-fsdev local,security_model=passthrough,id=fsdev0,path=/tmp/share -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

然后提示：
'virtio-9p-pci' is not a valid device model name

需要重新编译qemu，编译时添加额外的configure参数：
--enable-virtfs

编译安装qemu后，再运行，可以了。

在guest中挂载host共享的目录：

mkdir /tmp/host_files
mount -t 9p -o trans=virtio,version=9p2000.L hostshare /tmp/host_files

提示：
mount: unknown filesystem type '9p'

需要在kernel中添加9p的支持：
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y (Optional)
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y

然后就可以使用了，如果再加上virtio console的支持，那么既有shell，又可以传输文件，基本可以代替adb的常用功能了。

参考：

qemu启动命令：http://www.linux-kvm.org/page/9p_virtio

kernel配置：http://wiki.qemu.org/Documentation/9psetup

qemu添加configure选项：https://groups.google.com/forum/#!topic/coreos-dev/MjhL3tOOAVM

mkdir /tmp/host_files
mount -t 9p -o trans=virtio,version=9p2000.L hostshare /tmp/host_files

/usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel"   -initrd /usr/share/kata-containers/binary/kata-containers.img   -fsdev local,security_model=passthrough,id=fsdev0,path=./nfsroot   -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

/usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none

 https://futurewei-cloud.github.io/ARM-Datacenter/qemu/how-to-launch-aarch64-vm/

root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none  -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=./mysock
qemu-system-aarch64: -chardev socket,id=charconsole0,path=./mysock: Failed to connect socket ./mysock: Connection refused
root@ubuntu:/home/ubuntu# ls
dirA  dirB  dirC  mysock  nfsroot  qemu.sh  qemu.sh.bak  server.py  txt
root@ubuntu:/home/ubuntu# 

qemu-system-aarch64: root=/dev/pmem0p1: Could not open 'root=/dev/pmem0p1': No such file or directory
root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel  systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none  

root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/pmem0p1 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel  systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none  

root@ubuntu:/home/ubuntu# ps -elf | grep 20462
7 S root     20462 20399  4  80   0 - 801176 sys_po 20:25 ?       00:00:02 /usr/share/kata-containers/binary/qemu-system-aarch64 -name sandbox-3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc -uuid 2939253a-c41f-4b7d-bf0c-3fc4ca555d85 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -qmp unix:/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=257176M -device pcie-pci-bridge,bus=pcie.0,id=pcie-bridge-0,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/binary/kata-containers.img,size=184549376 -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/shared,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -append console=hvc0 console=hvc1 iommu.passthrough=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug panic=1 nr_cpus=64 agent.use_vsock=false systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug agent.debug_console -pidfile /run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/pid -D /run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/qemu.log -smp 1,cores=1,threads=1,sockets=64,maxcpus=64
1 S root     20466     2  0  80   0 -     0 vhost_ 20:25 ?        00:00:00 [vhost-20462]
0 S root     20513 20131  0  80   0 -  1097 pipe_w 20:26 pts/1    00:00:00 grep --color=auto 20462
root@ubuntu:/home/ubuntu# pidof qemu
root@ubuntu:/home/ubuntu# pidof qemu-system-aarch64
20462
root@ubuntu:/home/ubuntu# 

root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none -device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3 --enable-kvm 
qemu-system-aarch64: -device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3: device is modern-only, use disable-legacy=on
root@ubuntu:/home/ubuntu# 

qemu-system-x86_64 
      -nographic 
      -nodefaults 
      -L . 
      -machine virt,accel=kvm,kernel_irqchip,nofw 
      -smp sockets=1,cpus=4,cores=2,maxcpus=8 
      -cpu host 
      -m 2G,slots=3,maxmem=16G 
      -kernel /home/sebastien/workloads/linux_4_17/linux-4.17.14/arch/x86/boot/compressed/vmlinux.bin 
      -append 'console=hvc0 single iommu=false root=/dev/ram0 pci=lastbus=0' 
      -device virtio-serial-pci,id=virtio-serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev stdio,id=charconsole0 
      -device sysbus-debugcon,iobase=0x402,chardev=debugcon -chardev file,path=/tmp/debug-log,id=debugcon 
      -monitor telnet:127.0.0.1:55555,server,nowait 
      -qmp unix:/run/virtcontainers/pods/wH9OFIZOLD5jtSs3PNht/monitor.sock,server,nowait 
      -device pcie-root-port,id=rp1,addr=6,bus=pcie.0,multifunction=on

root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64  -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off  -smp 2 -m 4096M  -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88   -nographic   -append "root=/dev/pmem0p1 rw rootfstype=ext4 console=ttyAMA0  ignore_loglevel  systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug"   -initrd /usr/share/kata-containers/binary/kata-containers.img    -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none  -device virtio-serial-pci,id=virtio-serial0    -monitor telnet:127.0.0.1:55555,server,nowait

-monitor dev

Redirect the monitor to host device dev (same devices as the serial port). The default device is vc in graphical mode and stdio in non graphical mode. Use -monitor none to disable the default monitor.

 

-qmp dev

Like -monitor but opens in ’control’ mode.

 

-qmp-pretty dev

Like -qmp but uses pretty JSON formatting.

 

-mon [chardev=]name[,mode=readline|control][,pretty[=on|off]]

Setup monitor on chardev name. pretty turns on JSON pretty printing easing human reading and debugging.

 

-debugcon dev

Redirect the debug console to host device dev (same devices as the serial port). The debug console is an I/O port which is typically port 0xe9; writing to that I/O port sends output to this device. The default device is vc in graphical mode and stdio in non graphical mode.

# telnet 监听

qemu-system-i386 -monitor telnet:127.0.0.1:55555,server,nowait;

echo system_powerdown |telnet 127.0.0.1 55555

# tcp 监听

qemu-system-i386 -monitor tcp:127.0.0.1:55555,server,nowait;

echo 'info kvm' |nc -N 127.0.0.1 55555

echo system_powerdown |nc -N 127.0.0.1 55555

 

# unix

# -qmp JSON 协议

qemu-system-i386 monitor unix:qemu-monitor-socket,server,nowai

socat -,echo=0,icanon=0 unix-connect:qemu-monitor-socket

 

echo "info status" | socat - unix-connect:qemu-monitor-socket

echo "info status" | socat - unix-connect:qemu-monitor-socket | tail --lines=+2 | grep -v '^(qemu)'

# 查看版本

info version

 

# qemu -vnc :1,password

# 给 VNC 设置密码

change vnc password

 

# qemu -S

# 默认不启动 cpu, 手动启动

c