gVisor accesses the filesystem through a file proxy, called the Gofer. The gofer runs as a separate process, that is isolated from the sandbox. Gofer instances communicate with their respective sentry using the 9P protocol. For another explanation see What is gVisor?.
Sandbox overlay
To isolate the host filesystem from the sandbox, you can set a writable tmpfs overlay on top of the entire filesystem. All modifications are made to the overlay, keeping the host filesystem unmodified.
Note: All created and modified files are stored in memory inside the sandbox.
To use the tmpfs overlay, add the following runtimeArgs to your Docker configuration (/etc/docker/daemon.json) and restart the Docker daemon:
cat /etc/docker/daemon.json { "runtimes": { "runsc-ptrace": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=ptrace" ] }, "runsc-kvm": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=kvm", "--overlay" ] } } }
root@cloud:~# systemctl daemon-reload root@cloud:~# systemctl restart docker root@cloud:~# mount | grep docker root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000 cebede556c6355f20bc7914113aaa254f66045bf85da8c19e7c9f73e9287ec57 root@cloud:~# mount | grep docker overlay on /var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/X4BDV77JW4QTM43662GL73AOXI:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/diff,workdir=/var/lib/docker/overlay2/e3aa8b015aace1dabd480a01aba69b5a54d44f2b0fd0e18542f7e8896c1952b3/work,xino=off) nsfs on /run/docker/netns/bd5cad826299 type nsfs (rw) root@cloud:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cebede556c63 alpine "sleep 1000" 20 seconds ago Up 19 seconds test root@cloud:~# mount | grep docker | grep cebede556c63 root@cloud:~# docker rm -f test test root@cloud:~# mount | grep docker root@cloud:~#
root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000 5bafb1757274aeaa7d56c590e7efbaf1cd2ae17d024351d955d2cc4aa3749761 root@cloud:~# docker exec -it test sh / # touch hello.txt / # exit root@cloud:~# mount | grep docker overlay on /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/TOMZM5ZQL7BKVLGCOK6ESPTBC4:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/diff,workdir=/var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/work,xino=off) nsfs on /run/docker/netns/f69a72493d7b type nsfs (rw) root@cloud:~# find /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7/merged -name hello.txt root@cloud:~# find /var/lib/docker/overlay2/00a72e36719737dea6f78c3e520c790c83d1b7703dd285de4289fda5b19b7ab7 -name hello.txt root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt root@cloud:~#
host上没有hello.txt
Shared root filesystem
The root filesystem is where the image is extracted and is not generally modified from outside the sandbox. This allows for some optimizations, like skipping checks to determine if a directory has changed since the last time it was cached, thus missing updates that may have happened. If you need to docker cp files inside the root filesystem, you may want to enable shared mode. Just be aware that file system access will be slower due to the extra checks that are required.
Note: External mounts are always shared.
To use set the root filesystem shared, add the following runtimeArgs to your Docker configuration (/etc/docker/daemon.json) and restart the Docker daemon:
{
"runtimes": {
"runsc": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--file-access=shared"
]
}
}
}/etc/docker/daemon.json { "runtimes": { "runsc-ptrace": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=ptrace" ] }, "runsc-kvm": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=kvm", "--file-access=shared", "--overlay" ] } }
root@cloud:~# systemctl daemon-reload root@cloud:~# systemctl restart docker root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000 9dd8e1dbeeb2d2f6368e30ad6682b976ba0f703ec53d0f157ff12a94fd1dec3b docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/9dd8e1dbeeb2d2f6368e30ad6682b976ba0f703ec53d0f157ff12a94fd1dec3b/log.json: no such file or directory): /var/lib/docker/runtimes/runsc-kvm did not terminate successfully: unknown. root@cloud:~#
root@cloud:~# vi /etc/docker/daemon.json "--overlay" { "runtimes": { "runsc-ptrace": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=ptrace" ] }, "runsc-kvm": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=kvm", "--file-access=shared" ] } } }
root@cloud:~# systemctl daemon-reload root@cloud:~# systemctl restart docker root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000 bf9cf4e5d74155c909159977b45d743dce47ccdf6382307f9b96ae7115c43cc5 root@cloud:~# docker exec -it test sh / # touch hello.txt / # exit root@cloud:~# mount | grep docker overlay on /var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/34I2F4BGSXK7U3IDYMVJF5DK7G:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/diff,workdir=/var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/work,xino=off) nsfs on /run/docker/netns/1cb697a103c8 type nsfs (rw) root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt /var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/merged/hello.txt /var/lib/docker/overlay2/28cf157e6d16dec0b35ac97aa9f02767a5b8fa68ee1ef03277c9be35005b0483/diff/hello.txt root@cloud:~#
host上可以访问hello.txt
root@cloud:~# vi /etc/docker/daemon.json { "runtimes": { "runsc-ptrace": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=ptrace" ] }, "runsc-kvm": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=kvm" ] } } }
root@cloud:~# systemctl daemon-reload root@cloud:~# systemctl restart docker root@cloud:~# docker run --runtime=runsc-kvm --rm --name=test -d alpine sleep 1000 9f59a059edc0a17b9ea1bc98c9d126300a57fcb4e50355597c39f44f3e2a1ad8 root@cloud:~# mount | grep docker overlay on /var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BQYREFK7IYSGF2GTV26WSADNYS:/var/lib/docker/overlay2/l/YF6WOKNIAKC7ELH2VYQOQA4ER2,upperdir=/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/diff,workdir=/var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/work,xino=off) nsfs on /run/docker/netns/f2e7880a002b type nsfs (rw) root@cloud:~# docker exec -it test sh / # touch hello.txt / # exit root@cloud:~# find /var/lib/docker/overlay2/ -name hello.txt /var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/merged/hello.txt /var/lib/docker/overlay2/87b65072da34bf0823dfc8fa7ed9433278374465f83b7e78c3fa113e38d6dbad/diff/hello.txt root@cloud:~#
$ mkfifo f # make a named pipe $ cat f& # try to open it $ touch # further operations like this will hang
W0715 08:38:16.338198 1 log.go:338] Sentry detected 1 stuck task(s): Task tid: 1 (0x1), entered RunSys state 3m35.03s ago. Search for '(*Task).run(0x..., 0x<tid>)' in the stack dump to find the offending goroutine: ...... goroutine 125 [semacquire, 3 minutes]: sync.runtime_SemacquireMutex(0xc0002684b4, 0x0, 0x0) GOROOT/src/runtime/sema.go:71 +0x47 sync.(*RWMutex).Lock(0xc0002684ac) GOROOT/src/sync/rwmutex.go:103 +0x88 gvisor.dev/gvisor/pkg/sentry/fsimpl/gofer.(*filesystem).renameMuRUnlockAndCheckCaching(0xc000268400, 0xc000304e40) pkg/sentry/fsimpl/gofer/filesystem.go:637 +0x8e gvisor.dev/gvisor/pkg/sentry/fsimpl/gofer.(*filesystem).StatAt(0xc000268400, 0x14dc520, 0xc0002a8000, 0xc00065a900, 0x7ff, 0x100000000fff, 0x0, 0x1, 0x81ed00000000, 0x1000000018801ef, ...) pkg/sentry/fsimpl/gofer/filesystem.go:1354 +0x25f gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).StatAt(0xc000340900, 0x14dc520, 0xc0002a8000, 0xc0003821e0, 0xc000305640, 0xc000305198, 0x0, 0x0, 0x0, 0x0, ...) pkg/sentry/vfs/vfs.go:569 +0x124 gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.fstatat(0xc0002a8000, 0x7f5affffff9c, 0x564f0686bab0, 0x7f5af0767710, 0xc000000000, 0x0, 0x0) pkg/sentry/syscalls/linux/vfs2/stat.go:104 +0x46f gvisor.dev/gvisor/pkg/sentry/syscalls/linux/vfs2.Stat(0xc0002a8000, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0, 0x0, ...) pkg/sentry/syscalls/linux/vfs2/stat.go:34 +0xb1 gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0, ...) pkg/sentry/kernel/task_syscall.go:170 +0x501 gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0) pkg/sentry/kernel/task_syscall.go:305 +0x96 gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0xc0002a8000, 0x4, 0x564f0686bab0, 0x7f5af0767710, 0x7f5af0767710, 0x564f06822010, 0x1, 0xf97dcc70, 0x0, 0x0) pkg/sentry/kernel/task_syscall.go:265 +0xf1 gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0xc0002a8000, 0x0, 0x0) pkg/sentry/kernel/task_syscall.go:240 +0x6c9 gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x0, 0xc0002a8000, 0x0, 0x0) pkg/sentry/kernel/task_run.go:259 +0x178c gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc0002a8000, 0x1) pkg/sentry/kernel/task_run.go:92 +0x221 created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start pkg/sentry/kernel/task_start.go:318 +0x126