[root@bogon kata]# cd /opt/gopath/src/github.com/kata-containers/kata-containers/src/ [root@bogon src]# make && make install make: *** No targets specified and no makefile found. Stop. [root@bogon src]# ls agent runtime trace-forwarder [root@bogon src]# cd runtime/ [root@bogon runtime]# ls arch CODEOWNERS data go.mod hack Makefile pkg README.md VERSION cli containerd-shim-v2 golang.mk go.sum LICENSE netmon protocols vendor virtcontainers [root@bogon runtime]# pwd /opt/gopath/src/github.com/kata-containers/kata-containers/src/runtime [root@bogon runtime]# make && make install
[root@bogon src]# /usr/local/bin/kata-runtime -version : 2.0.0-rc0 commit : ed711b5ab1cd36390690f52b2cb2f2e204b4adf3 OCI specs: 1.0.1-dev [root@bogon src]#
[root@bogon src]# ls /usr/local/bin/ containerd-shim-kata-v2 kata-collect-data.sh kata-monitor kata-runtime [root@bogon src]#
firecracker
1、加载vhost_vsock
[root@bogon src]# lsmod | grep vhost_vsock [root@bogon src]# modprobe vhost_vsock [root@bogon src]# lsmod | grep vhost_vsock vhost_vsock 262144 0 vmw_vsock_virtio_transport_common 262144 1 vhost_vsock vsock 262144 2 vhost_vsock,vmw_vsock_virtio_transport_common vhost 262144 1 vhost_vsock [root@bogon src]#
[root@bogon runtime]# ls /usr/local/bin/ containerd-shim-kata-v2 kata-collect-data.sh kata-monitor kata_qemu kata-runtime [root@bogon runtime]# kata-runtime kata-check ERRO[0000] Near line 27 (last key parsed 'hypervisor.firecracker.valid_hypervisor_paths'): expected value but found ' ' instead arch=arm64 name= pid=64031 source=runtime Near line 27 (last key parsed 'hypervisor.firecracker.valid_hypervisor_paths'): expected value but found ' ' instead [root@bogon runtime]#
添加firecrakcer
[root@bogon kata]# ls /usr/bin/firecracker /usr/bin/firecracker [root@bogon kata]#
[root@bogon runtime]# kata-runtime kata-env Near line 27 (last key parsed 'hypervisor.firecracker.valid_hypervisor_paths'): expected value but found '/' instead [root@bogon runtime]# kata-runtime kata-env /usr/share/defaults/kata-containers/configuration-fc.toml: file /usr/share/kata-containers/vmlinux.container does not exist [root@bogon runtime]#
[root@bogon kata]# docker run --rm -it --runtime kata-runtime ubuntu Unable to find image 'ubuntu:latest' locally latest: Pulling from library/ubuntu a25fe3630538: Pull complete 326fa3abf061: Pull complete ff1b87601e0a: Pull complete Digest: sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537 Status: Downloaded newer image for ubuntu:latest docker: Error response from daemon: OCI runtime create failed: Invalid command "create": unknown. [root@bogon kata]#
2.0没有shim proxy
[root@bogon kata]# ls /usr/local/bin/ containerd-shim-kata-v2 kata-collect-data.sh kata-monitor kata_qemu kata-runtime [root@bogon kata]#
[root@bogon kata]# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/nginx:alpine sh ctr: io.containerd.kata.v2: failed to listen to abstract unix socket "/containerd-shim/default/sh/shim.sock": listen unix /containerd-shim/default/sh/shim.sock: bind: address already in use : exit status 1: unknown [root@bogon kata]#
[root@bogon kata]# netstat -ax | grep containerd unix 2 [ ACC ] STREAM LISTENING 33325958 @/containerd-shim/default/sh/shim.sock@ unix 2 [ ACC ] STREAM LISTENING 33460449 /run/containerd/containerd.sock.ttrpc unix 2 [ ACC ] STREAM LISTENING 33460451 /run/containerd/containerd.sock unix 3 [ ] STREAM CONNECTED 33423160 /run/containerd/containerd.sock unix 3 [ ] STREAM CONNECTED 33405690 /run/containerd/containerd.sock [root@bogon kata]#
[root@bogon kata]# netstat -pax | grep containerd unix 2 [ ACC ] STREAM LISTENING 33325958 12307/containerd-sh @/containerd-shim/default/sh/shim.sock@ unix 2 [ ACC ] STREAM LISTENING 33460449 13248/containerd /run/containerd/containerd.sock.ttrpc unix 2 [ ACC ] STREAM LISTENING 33460451 13248/containerd /run/containerd/containerd.sock unix 3 [ ] STREAM CONNECTED 33468754 13248/containerd unix 3 [ ] STREAM CONNECTED 33423160 13248/containerd /run/containerd/containerd.sock unix 2 [ ] DGRAM 33458712 12307/containerd-sh unix 3 [ ] STREAM CONNECTED 33405690 13248/containerd /run/containerd/containerd.sock [root@bogon kata]# kill -9 12307 [root@bogon kata]# netstat -pax | grep containerd unix 2 [ ACC ] STREAM LISTENING 33460449 13248/containerd /run/containerd/containerd.sock.ttrpc unix 2 [ ACC ] STREAM LISTENING 33460451 13248/containerd /run/containerd/containerd.sock unix 3 [ ] STREAM CONNECTED 33468754 13248/containerd unix 3 [ ] STREAM CONNECTED 33423160 13248/containerd /run/containerd/containerd.sock unix 3 [ ] STREAM CONNECTED 33405690 13248/containerd /run/containerd/containerd.sock [root@bogon kata]#
[root@bogon output]# curl --unix-socket /tmp/firecracker.socket -i http://localhost/ HTTP/1.1 200 Server: Firecracker API Connection: keep-alive Content-Type: application/json Content-Length: 91 {"id":"anonymous-instance","started":false,"vmm_version":"0.22.1","app_name":"Firecracker"}[root@bogon output]#
等一等
[root@bogon kata]# curl --unix-socket /tmp/firecracker.socket -i http://localhost/ HTTP/1.1 200 Server: Firecracker API Connection: keep-alive Content-Type: application/json Content-Length: 90 {"id":"anonymous-instance","started":true,"vmm_version":"0.22.1","app_name":"Firecracker"}[root@bogon kata]# [root@bogon kata]# [root@bogon kata]#
Kata Containers on Kubernetes and Kata Firecracker VMM support
Enjoy the speed of containers while still keeping the security of virtual machines?? what does this mean and how different is this approach when compared with existing container architecture?? Traditional containers use Linux control groups, cgroups, for managing and allocating resources and namespaces to provide container isolation. Shared kernel between the host operating system and the guest containers, leaving the other container workloads in a cluster vulnerable if one container is comprised. This issue is one of the big drivers behind Kata Containers originated form Intel’s Clear Containers project started in 2015.
In Kata, each container has its own lightweight virtual machine and mini-kernel, providing container isolation via hardware virtualization facilitated by Kata’s six components: Agent, Runtime, Proxy, Shim, Kernel and packaging of QEMU. It is designed to be architecture agnostic, run on multiple hypervisors and be compatible with the OCI specification for Docker containers and CRI for Kubernetes.
The recent addition of CRI (Container Runtime Interface) to Kubernetes means Kata Containers can be controlled by any OCI (Open Container Initiative) compatible CRI implementation, CRI-O (lightweight alternative to using Docker as the runtime for kubernetes) being the main one. Clear Containers can now receive container annotations to let it know when and how to run pod VMs or container workloads within those pods. In Kubernetes clusters with CRI-O and cc-runtime as the default container runtime, the launch of a pod results in the creation of a VM. Then, when a container is added to that pod, it is launched as a container inside the pod’s VM.
Kata Containers on Kubernetes
The key difference between the Kata approach and other container engines is that Kata uses hardware-backed isolation as the boundary for each container or collection of containers in a Kubernetes container pod. As of today Kata can be installed with CRI-O or Containerd and cannot be leveraged with dockershim. As the architecture uses QEMU (hardware virtualization), Kata will only work if nested-virtualization support is available on the virtual machines.
kata-runtime: The Open Containers Initiative (OCI) compliant runtime which is called by an upper level orchestrator (such as Docker or a Kubernetes CRI-shim). With Kata installed we can start pods which will make use of Kata Containers.
CRI-O: CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes. It is a lightweight alternative to using Docker as the runtime for Kubernetes.
As seen above docker is replaced with CRI-O and all the client operations can be done using “CRICTL”.
kata-proxy and kata-shim: proxy binary runs on the host to help conversion of multiplex messages into demultiplex messages and executed per pod. shim binary runs on the host to handle stdio and signaling for container processes.
Along with the above binaries, artifacts associated with the VM itself like : containers.img and vmlinuz are included.
Enabling Kubelet to use CRI-O as runtime:
Extra arguments can be provided to Kubelet configration to use CRI-O as a CRI replacing conventional Docker runtime.
Initializing Kubernetes with CRI-O:
Kubeadm with the above configuration can be used to bootstrap a Kubernetes cluster with CRI-O. Crictl replaces docker cli.
CNI can be installed the same way users do with Kubeadm and docker.
Running Kata Containers as Kubernetes Pods:
The CRI-O project supports the ability to provide a secondary runtime to handle untrusted workloads using Kubernetes annotations. In CRI-O this is called the untrusted-runtime. This means, in an environment with workloads of various levels of trust, CRI-O allows your Kubernetes cluster to be composed of a mix of runc and cc-runtime based pods. Note, if this `untrusted runtime` is not provided in the CRI-O configuration, then all workloads will make use of the trusted runtime, which defaults to runc.
By default all the kubernetes control-plane components are created on runc as a specific annotation is not set by the Kubeadm while creating the containers. As seen below the 18 Kubernetes control plane components are defaulted to runc as specific annotation to use cri-o was not mentioned in the manifests.
No control-plane container/pod is identified as a Kata workload and will not be listed in “kata-runtime list”.
Creating a sample container with the ‘kubernetes.cri-o’ set in the manifest above the
The above manifest creates test-kata vm-containers on Kubernetes using CRI-O and Kata container runtime. The same will not be listed on runc as a specific annotation to use cri-o was set.
kata-runtime list output:
As Kata uses QEMU to create light-weight VM’s the pods created with the manifest above are listed in the QEMU processes as seen below.
As seen above Kubernetes can leverage CRI-O & Kata-Runtime to create pod-vm’s where Kubernetes can accept/operate them as pods along with providing all the properties/features that comes with a virtual-machine.
Firecracker VMM + Kata Containers
Firecracker form AWS is a Virtual Machine Manager (VMMs) used to create and manage microVMs with a minimalistic and simple design reducing memory overhead. Firecracker is exclusively designed for running transient and short-lived processes like functions and serverless workloads which require a faster start and higher density with minimal resource utilization. Firecracker was developed using the language Rust as a way to enhance the backend implementation of AWS Lambda and AWS Fargate.
Kata Containers vs Firecracker:
Kata executes containers within QEMU based virtual machines. Firecracker is a cloud-native alternative to QEMU that is purpose-built for running containers safely and efficiently, and nothing more. Firecracker is being positioned as a next-generation of Kata that would be more focused on modern workloads. Firecracker also allows the use of container runtimes like Containerd to manage containers as microVMs. This allows Docker and container orchestration frameworks like Kubernetes to use Firecracker. However, initial integration with Kubernetes is limited to external APIs.
Creating a MicroVM using Firecracker
Firecracker VMM supports a totally api-driven operations schema which can easily tie with the serverless and lambda architecture. Once Firecracker-API starts to serve requests users can submit guest-kernel, guest-filesystem, configuration, starting and stopping the micro-vm through API calls.
Below is a simple example showing how a VM can be created:
Providing kernel_image_path and file-system:
Configuring resources and starting the VM:
With this a micro-vm will be created on the host and the whole architecture uses Rust.
The Firecracker process exposes REST API via a UNIX socket, which can be used to manage the lifecycle of a microVM. Users can only access them through UART/serial console because they don’t even run SSH. Apart from the serial console, these microVMs may be connected to a virtual NIC, a block device and a one-button keyboard.
Running Kata containers utilizing Firecracker VMM/Hypervisor
The 1.5.0-rc2 release of Kata Containers introduces support for the Firecracker hypervisor. This is an initial release and evolving. Docker should be configured to use kata-runtime and vhost_vsock is leveraged to enable communication between a hypervisor and virtual machine.
{
"runtimes": {
"kata": {
"path": "/opt/kata/bin/kata-runtime"
}
},
"storage-driver": "devicemapper"
}
With this docker configuration users can now run kata containers utilizing firecracker. A flag can be passed with docker cli to run containers as shown below:
docker run --runtime=kata -itd --name=kata-test alpine sh
In this case QEMU is not used as a hypervisor, Kata uses Firecracker hypervisor to create VM’s.
With this support users can use Firecracker with Kubernetes to create micro-vm’s once the project evolves.
Kata works well in an environment where users need the efficiency of a container stack with a higher level of security than running containers side by side in a single kernel. Multiple scenarios such as network functions virtualization, edge computing, development and testing, and containers as a service can greatly make use of Kata. NFV on Kubernetes can benefit significantly as Kata can provide all the features that a VM can have to containers which means NFV (like: multiple interfaces, DPDK, SRIOV etc.) can be made easy in container space in the future using this approach. In addition, Kata’s small footprint and high level of security will make it well suited to edge deployments where resources are limited.