docker

背景

最近在某云申请了一个免费试用的云服务器，默认配置是：CPU - 1Core， MEM - 1G, 使用 docker-compose 启动服务组，docker container 反复重启。。。

排查问题

使用 docker log 进入容器查看，未发现问题。。

使用 docker stats 查看容器使用资源的情况，发现问题:  内存溢出！！

ubuntu@VM-0-3-ubuntu:~$ docker stats
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x55fb06c3ca5c, 0x16)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/panic.go:617 +0x74 fp=0x7fff4a9245d0 sp=0x7fff4a9245a0 pc=0x55fb056605d4
runtime.sysMap(0xc000000000, 0x4000000, 0x55fb08bfefb8)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mem_linux.go:170 +0xc9 fp=0x7fff4a924610 sp=0x7fff4a9245d0 pc=0x55fb0564b8e9
runtime.(*mheap).sysAlloc(0x55fb08be5aa0, 0x2000, 0x55fb08be5ab0, 0x1)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:633 +0x1cf fp=0x7fff4a9246b8 sp=0x7fff4a924610 pc=0x55fb0563e6ff
runtime.(*mheap).grow(0x55fb08be5aa0, 0x1, 0x0)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1222 +0x44 fp=0x7fff4a924710 sp=0x7fff4a9246b8 pc=0x55fb05658cf4
runtime.(*mheap).allocSpanLocked(0x55fb08be5aa0, 0x1, 0x55fb08bfefc8, 0x0)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1150 +0x381 fp=0x7fff4a924748 sp=0x7fff4a924710 pc=0x55fb05658be1
runtime.(*mheap).alloc_m(0x55fb08be5aa0, 0x1, 0x2a, 0x6e43a318)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:977 +0xc6 fp=0x7fff4a924798 sp=0x7fff4a924748 pc=0x55fb05658236
runtime.(*mheap).alloc.func1()
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1048 +0x4e fp=0x7fff4a9247d0 sp=0x7fff4a924798 pc=0x55fb0568935e
runtime.(*mheap).alloc(0x55fb08be5aa0, 0x1, 0x55fb0501002a, 0x7fff4a924870)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mheap.go:1047 +0x8c fp=0x7fff4a924820 sp=0x7fff4a9247d0 pc=0x55fb0565850c
runtime.(*mcentral).grow(0x55fb08be68a0, 0x0)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcentral.go:256 +0x97 fp=0x7fff4a924868 sp=0x7fff4a924820 pc=0x55fb0564b367
runtime.(*mcentral).cacheSpan(0x55fb08be68a0, 0x7ff734a23000)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcentral.go:106 +0x301 fp=0x7fff4a9248c8 sp=0x7fff4a924868 pc=0x55fb0564ae71
runtime.(*mcache).refill(0x7ff734a23008, 0x2a)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/mcache.go:135 +0x88 fp=0x7fff4a9248e8 sp=0x7fff4a9248c8 pc=0x55fb0564a908
runtime.(*mcache).nextFree(0x7ff734a23008, 0x55fb08bdb92a, 0x7ff734a23008, 0x7ff734a23000, 0x8)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:786 +0x8a fp=0x7fff4a924920 sp=0x7fff4a9248e8 pc=0x55fb0563ef3a
runtime.mallocgc(0x180, 0x55fb0792fe20, 0x1, 0x55fb08bff020)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:939 +0x780 fp=0x7fff4a9249c0 sp=0x7fff4a924920 pc=0x55fb0563f870
runtime.newobject(0x55fb0792fe20, 0x4000)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/malloc.go:1068 +0x3a fp=0x7fff4a9249f0 sp=0x7fff4a9249c0 pc=0x55fb0563fc7a
runtime.malg(0x521c300008000, 0x55fb08be8110)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:3220 +0x33 fp=0x7fff4a924a30 sp=0x7fff4a9249f0 pc=0x55fb05669a83
runtime.mpreinit(...)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/os_linux.go:311
runtime.mcommoninit(0x55fb08bdfd60)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:618 +0xc6 fp=0x7fff4a924a68 sp=0x7fff4a924a30 pc=0x55fb056633f6
runtime.schedinit()
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/proc.go:540 +0x78 fp=0x7fff4a924ac0 sp=0x7fff4a924a68 pc=0x55fb05663088
runtime.rt0_go(0x7fff4a924bc8, 0x2, 0x7fff4a924bc8, 0x0, 0x7ff734055b97, 0x2, 0x7fff4a924bc8, 0x200008000, 0x55fb0568b3d0, 0x0, ...)
        /build/docker.io-YkakXX/docker.io-19.03.6/go/src/runtime/asm_amd64.s:195 +0x11e fp=0x7fff4a924ac8 sp=0x7fff4a924ac0 pc=0x55fb0568b4fe
ubuntu@VM-0-3-ubuntu:~$ 

　　

解决方法

一是增加系统内存，二是优化进程，使其占用内存降低。

考虑到云服务器只有1G内存，目前考虑升级硬件配置。

关于docker使用物理内存的情况，网上搜罗了一下，大体是说需要保证2G的物理内存保障docker的运行。。 

I can also easily see this error, with docker 1.8.2, kernel 4.3-rc4 on a virtual machine with 1G RAM.
I suppose it's a similar issue as moby/moby#14460. However, with docker 1.8.2 the crash seems to still happen.
As a workaround, it would be possible to raise system memory to 2G or higher. That'll work according to my tests.
In contrast to other issues like #489, it's interesting that kernel OOM killer is not necessarily involved in this case. So it might make sense to limit memory utilization on the docker side.

参考：

https://github.com/coreos/bugs/issues/908

https://blog.csdn.net/loveliness_peri/article/details/88310473