gvisor gofer vfs2

gVisor accesses the filesystem through a file proxy, called the Gofer. The gofer runs as a separate process, that is isolated from the sandbox. Gofer instances communicate with their respective sentry using the 9P protocol. For another explanation see What is gVisor?.

Sandbox overlay

To isolate the host filesystem from the sandbox, you can set a writable tmpfs overlay on top of the entire filesystem. All modifications are made to the overlay, keeping the host filesystem unmodified.

Note: All created and modified files are stored in memory inside the sandbox.

To use the tmpfs overlay, add the following runtimeArgs to your Docker configuration (/etc/docker/daemon.json) and restart the Docker daemon:

cat /etc/docker/daemon.json 
{
    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm",
                "--overlay"
            ]
        }
    }
}

 

root@cloud:~# systemctl daemon-reload
root@cloud:~# systemctl restart docker 
root@cloud:~# mount | grep docker
root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000
cebede556c6355f20bc7914113aaa254f66045bf85da8c19e7c9f73e9287ec57
root@cloud:~# mount | grep docker
overlay on /var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/X4BDV77JW4QTM43662GL73AOXI:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/diff,workdir=/var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/work,xino=off)
nsfs on /run/docker/netns/bd5cad826299 type nsfs (rw)
root@cloud:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
cebede556c63        alpine              "sleep 1000"        20 seconds ago      Up 19 seconds                           test
root@cloud:~# mount | grep docker | grep cebede556c63 
root@cloud:~# docker rm -f test
test
root@cloud:~# mount | grep docker
root@cloud:~# 

root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000
5bafb1757274aeaa7d56c590e7efbaf1cd2ae17d024351d955d2cc4aa3749761
root@cloud:~# docker exec -it test sh
/ # touch hello.txt
/ # exit
root@cloud:~# mount | grep docker 
overlay on /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/TOMZM5ZQL7BKVLGCOK6ESPTBC4:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/diff,workdir=/var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/work,xino=off)
nsfs on /run/docker/netns/f69a72493d7b type nsfs (rw)
root@cloud:~# find /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/merged -name hello.txt
root@cloud:~# find /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7 -name hello.txt
root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt
root@cloud:~# 

 host上没有hello.txt

Shared root filesystem

The root filesystem is where the image is extracted and is not generally modified from outside the sandbox. This allows for some optimizations, like skipping checks to determine if a directory has changed since the last time it was cached, thus missing updates that may have happened. If you need to docker cp files inside the root filesystem, you may want to enable shared mode. Just be aware that file system access will be slower due to the extra checks that are required.

Note: External mounts are always shared.

To use set the root filesystem shared, add the following runtimeArgs to your Docker configuration (/etc/docker/daemon.json) and restart the Docker daemon:

{
    "runtimes": {
        "runsc": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--file-access=shared"
            ]
       }
    }
}

/etc/docker/daemon.json 

{
    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm",
                "--file-access=shared",
                "--overlay"
            ]
        }
    }

root@cloud:~# systemctl daemon-reload
root@cloud:~# systemctl restart docker 
root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000
9dd8e1dbeeb2d2f6368e30ad6682b976ba0f703ec53d0f157ff12a94fd1dec3b
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/9dd8e1dbeeb2d2f6368e30ad6682b976ba0f703ec53d0f157ff12a94fd1dec3b/log.json: no such file or directory): /var/lib/docker/runtimes/runsc-kvm did not terminate successfully: unknown.
root@cloud:~# 

root@cloud:~# vi /etc/docker/daemon.json 
                "--overlay"
{
    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm",
                "--file-access=shared"
            ]
        }
    }
}

root@cloud:~# systemctl daemon-reload
root@cloud:~# systemctl restart docker 
root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000
bf9cf4e5d74155c909159977b45d743dce47ccdf6382307f9b96ae7115c43cc5
root@cloud:~# docker exec -it test sh
/ # touch hello.txt
/ # exit
root@cloud:~# mount | grep docker
overlay on /var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/34I2F4BGSXK7U3IDYMVJF5DK7G:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/diff,workdir=/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/work,xino=off)
nsfs on /run/docker/netns/1cb697a103c8 type nsfs (rw)
root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt
/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/merged/hello.txt
/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/diff/hello.txt
root@cloud:~# 

host上可以访问hello.txt

root@cloud:~# vi /etc/docker/daemon.json 
{
    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm"
            ]
        }
    }
}

root@cloud:~# systemctl daemon-reload
root@cloud:~# systemctl restart docker 
root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000
9f59a059edc0a17b9ea1bc98c9d126300a57fcb4e50355597c39f44f3e2a1ad8
root@cloud:~# mount | grep docker
overlay on /var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BQYREFK7IYSGF2GTV26WSADNYS:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/diff,workdir=/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/work,xino=off)
nsfs on /run/docker/netns/f2e7880a002b type nsfs (rw)
root@cloud:~# docker exec -it test sh
/ # touch hello.txt
/ # exit
root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt
/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/merged/hello.txt
/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/diff/hello.txt
root@cloud:~#

$ mkfifo f    # make a named pipe
$ cat f&      # try to open it
$ touch       # further operations like this will hang

W0715 08:38:16.338198       1 log.go:338] Sentry detected 1 stuck task(s):
    Task tid: 1 (0x1), entered RunSys state 3m35.03s ago.
Search for '(*Task).run(0x..., 0x<tid>)' in the stack dump to find the offending goroutine:
......
goroutine 125 [semacquire, 3 minutes]:
sync.runtime_SemacquireMutex(0xc0002684b4, 0x0, 0x0)
    GOROOT/src/runtime/sema.go:71 +0x47
sync.(*RWMutex).Lock(0xc0002684ac)
    GOROOT/src/sync/rwmutex.go:103 +0x88
gvisor.dev/gvisor/pkg/sentry/fsimpl/gofer.(*filesystem).renameMuRUnlockAndCheckCaching(0xc000268400, 0xc000304e40)
    pkg/sentry/fsimpl/gofer/filesystem.go:637 +0x8e
gvisor.dev/gvisor/pkg/sentry/fsimpl/gofer.(*filesystem).StatAt(0xc000268400, 0x14dc520, 0xc0002a8000, 0xc00065a900, 0x7ff, 0x100000000fff, 0x0, 0x1, 0x81ed00000000, 0x1000000018801ef, ...)
    pkg/sentry/fsimpl/gofer/filesystem.go:1354 +0x25f
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).StatAt(0xc000340900, 0x14dc520, 0xc0002a8000, 0xc0003821e0, 0xc000305640, 0xc000305198, 0x0, 0x0, 0x0, 0x0, ...)
    pkg/sentry/vfs/vfs.go:569 +0x124
gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.fstatat(0xc0002a8000, 0x7f5affffff9c, 0x564f0686bab0, 0x7f5af0767710, 0xc000000000, 0x0, 0x0)
    pkg/sentry/syscalls/linux/vfs2/stat.go:104 +0x46f
gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.Stat(0xc0002a8000, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0, 0x0, ...)
    pkg/sentry/syscalls/linux/vfs2/stat.go:34 +0xb1
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0, ...)
    pkg/sentry/kernel/task_syscall.go:170 +0x501
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0)
    pkg/sentry/kernel/task_syscall.go:305 +0x96
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0)
    pkg/sentry/kernel/task_syscall.go:265 +0xf1
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0xc0002a8000, 0x0, 0x0)
    pkg/sentry/kernel/task_syscall.go:240 +0x6c9
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x0, 0xc0002a8000, 0x0, 0x0)
    pkg/sentry/kernel/task_run.go:259 +0x178c
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc0002a8000, 0x1)
    pkg/sentry/kernel/task_run.go:92 +0x221
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start
    pkg/sentry/kernel/task_start.go:318 +0x126