Making a Custom MicroVM for AWS Firecracker!

 

First, let’s bake a kernel!

The firecracker team was nice enough to give us a .config to use as the basis of our kernel and its a great place to start.

#Move to a new directory and grab the config!
$ wget https://raw.githubusercontent.com/firecracker-microvm/firecracker/master/resources/microvm-kernel-config$ mv microvm-kernel-config .config

Now let’s grab the kernel sources we will be using. Note it’s best to choose a kernel version close to the distro’s native kernel but not an exact requirement.

#Grab sources
$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/linux-4.20.9.tar.xz#Untar
$ tar xvf linux-4.20.9.tar.xz#Move the .config to the root of the kernel source folders
$ cd linux-4.20.9/
$ cd ../.config ./

Next up is compiling the kernel, often an annoying and tricky part full of dragons and bad times. But assuming you have the right .config, proper libraries installed and a good source snapshot, it will work.

#while in linux-4.20.9/#Compile the kernel and use -j to increase threads to 32
$ make -j32

Errors while compiling are common, especially missing library, so try to find anything you need in a repo and install with dnf/apt or whatever your system package manager is before trying again.

Next if you get a successful build the kernel will be located at “linux-4.20.9/arch/x86/boot/bzImage”. (Yes, this will be the 64 bit kernel if you use the config.)

Now extract the kernel from the bzImage!

#Copy then extract linux-4.20.9/arch/x86/boot/bzImage
$ cp ./arch/x86/boot/bzImage ./
$ binwalk --extract bzImage

Running binwalk in “--extract” will create a folder named “_FILENAME.extraced” that contains the output. In my case the only file generated is the decompressed kernel named “44E9”. This “44E9” file is exactly whats needed.

$ cd _bzImage.extracted/
$ ls
44E9$ file 44E9
44E9: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=3c3a47380d9e44d2183f676a54bb86f9e37273eb, stripped$ binwalk 44E9 | head
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------
0             0x0             ELF, 64-bit LSB executable, AMD x86-64
4309805       0x41C32D        Boot section Start 0x0 End 0x-59BA38B8
12583136      0xC000E0        Linux kernel version "4.20.9 (root@loc
12617408      0xC086C0        ELF, 64-bit LSB relocatable, AMD x86-6
12852480      0xC41D00        CRC32 polynomial table, little endian

And now we have the kernel! Save it somewhere for later or try to boot it without a filesystem to see if you succeed. If you see a kernel panic following the line “VFS: Cannot open root device “(null)” or unknown-block(0,0): error -6", dont worry, it simply needs a filesystem to use.

[    0.185186] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
[    0.186122] Please append a correct "root=" boot option; here are the available partitions:
[    0.187213] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.188265] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.20.9 #2
[    0.189008] Call Trace:
[    0.189339]  dump_stack+0x4d/0x6a
[    0.189785]  panic+0xf6/0x244
[    0.190165]  mount_block_root+0x206/0x2ad
[    0.190671]  ? do_early_param+0x90/0x90

 

Now time to create that root filesystem! This is easiest to do by creating a raw image and use QEMU to install alpine to that image. Then with kpartx we can mount the disk image and copy/edit files we need to our new rootfs.

Create the ext4 filesystem, format it and mount.

#Create the new root filesystem with QEMU tools
$ qemu-img create -f raw ./rootfs.ext4 250M#Format
$ mkfs.ext4 rootfs.ext4#Mount for later
$ mount rootfs.ext4 /mnt

Boot a qemu vm with a new disk (use raw disk!) and perform a normal alpine install. Once the installation is complete, shut down the vm. Using kpartx we mount the fresh alpine install. This image will contain both a boot loader and a root filesystem but all we need is the filesystem. DD is an option also.

#Understand filetype
$ file qemu-alpine.raw
qemu-alpine.raw: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x1a,3,16), end-CHS (0x2a1,2,14), startsector 2048, 204800 sectors; partition 2 : ID=0x83, start-CHS (0x2a1,2,15), end-CHS (0x250,3,7), startsector 206848, 305152 sectors#Create loop devices
$ kpartx -a -v qemu-alpine.raw
add map loop5p1 (253:5): 0 204800 linear 7:5 2048
add map loop5p2 (253:6): 0 305152 linear 7:5 206848#make mountpoint
$mkdir /mnt2#Mount loop. Usually loop0p1 == "/boot" & loop0p2 == "/" or rootfs
$ mount /dev/mapper/loop0p2 /mnt2

Lastly lets copy from the Alpine install to our new rootfs.

#populate rootfs
$ cp -r /mnt/* /mnt2/#Cleanup
$ umount /mnt
$ umount /mnt2

 

Booting

One important thing to do is remember to modify your firecracker boot-method to tell the kernel what device will be the filesystem is so it can continue the boot process and not panic like before. To do this we add the boot argument “root=/dev/vda”.

#Modified kernel call
curl --unix-socket /tmp/firecracker.socket -i 
-X PUT 'http://localhost/boot-source'   
-H 'Accept: application/json'           
-H 'Content-Type: application/json'     
-d '{
"kernel_image_path": "./kernel.bin",
"boot_args": "console=ttyS0 reboot=k panic=1 pci=off 'root=/dev/vda'"
}'curl --unix-socket /tmp/firecracker.socket -i 
-X PUT 'http://localhost/drives/rootfs' 
-H 'Accept: application/json'           
-H 'Content-Type: application/json'     
-d '{
"drive_id": "rootfs",
"path_on_host": "./rootfs.ext4",
"is_root_device": true,
"is_read_only": false
}'

Then do an “InstanceStart” and see what happens! If everything is working then it should get you a login prompt like below. Note the 4.20.9 kernel.

 

Some additional info to validate your kernel:

For the sake of discussion let’s use binwalk see how our kernel looks vs the hello-world vm’s kernel. This helps sort why your kernel might not be booting. To start it must be a non-compressed, elf binary with a boot section.

First hello-vmlinux.bin, note the boot section at 0x0

$ file hello-vmlinux.bin
hello-vmlinux.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=8647c7e2e72433a17adcc6f485c42764f187d2ff, not stripped#
$ binwalk hello-vmlinux.bin | headDECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------
0             0x0             ELF, 64-bit LSB executable, AMD x86-
4185862       0x3FDF06        Boot section Start 0x0 End 0x-59BA38B8
12583104      0xC000C0        Linux kernel version "4.14.55-84.37.am
12617856      0xC08880        ELF, 64-bit LSB relocatable, AMD x86-
12846848      0xC40700        CRC32 polynomial table, little endian
13033080      0xC6DE78        LZO compressed data
13463595      0xCD702B        Unix path: /x86/entry/vdso/vma.c

Now our custom kernel before unpacking:

$ file bzImage
bzImage: Linux kernel x86 boot executable bzImage, version 4.20.9 (root@localhost.localdomain) #4 SMP Sun Mar 3 14:53:10 GMT 2019, RO-rootFS, swap_dev 0x4, Normal VGA$binwalk bzImage
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------
17641         0x44E9          gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00)

And after unpacking with “binwalk --extract” a good kernel looks like:

$ file 44E9
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=3c3a47380d9e44d2183f676a54bb86f9e37273eb, stripped#Use binwalk on the kernel image
$ binwalk 44E9 | headDECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------
0             0x0             ELF, 64-bit LSB executable, AMD x86-4309805       0x41C32D        Boot section Start 0x0 End 0x-59BA38B8
12583136      0xC000E0        Linux kernel version "4.20.9 (root@loc
12617408      0xC086C0        ELF, 64-bit LSB relocatable, AMD x86-12852480      0xC41D00        CRC32 polynomial table, little endian