docker不稳定 short running containers with -rm failed to destroy

正常运行以下命令

sudo docker run --rm busybox echo helloworld

 

/var/log/upstart/docker.log 日志如下：

2014/08/07 00:12:02 POST /v1.13/containers/create
[339dd1d9] +job create()
[339dd1d9] -job create() = OK (0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/attach?stderr=1&stdout=1&stream=1
[339dd1d9] +job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] +job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
2014/08/07 00:12:02 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/start
[339dd1d9] +job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] +job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job allocate_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] -job start(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] +job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job release_interface(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
[339dd1d9] -job attach(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 POST /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/wait
[339dd1d9] +job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job wait(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 GET /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0/json
[339dd1d9] +job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_inspect(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)
2014/08/07 00:12:03 DELETE /v1.13/containers/5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0?v=1
[339dd1d9] +job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0)
[339dd1d9] -job container_delete(5abf93d6f3f8fc6166f12fb0682d3f4d8a056cb7a07b9ef52f109525137192c0) = OK (0)

我把这个写入crontab中，每分钟执行一次，有几率出现如下错误（如果更加频繁执行docker run那出现错误的几率更大，目前一分钟一次的出错概率大概是2天一个error）

Error response from daemon: Cannot destroy container b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: Driver aufs failed to remove root filesystem b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e: rename /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e /var/lib/docker/aufs/mnt/b12b979ec303276ad8c7fedcdf687c7c3760c822297fb3b7f39f52954b90057e-removing: device or resource busy

Error response from daemon: Cannot destroy container b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: Unable to remove filesystem for b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: remove /var/lib/docker/containers/b103443f2f2d74862e4aa37132043b9ca4aa1ade7bbac37a514c051f859d0473: directory not empty

原因：基本还是docker不稳定造成的，并行快速的运行container的时候，有几率出现如如上错误

与之有关的issue如下：

https://github.com/docker/docker/issues/4095

https://github.com/docker/docker/issues/3968

https://github.com/docker/docker/issues/3823

我机器的信息如下：

root@localhost:~# docker info
Containers: 4
Images: 12
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 20
Execution Driver: native-0.2
Kernel Version: 3.8.0-44-generic
WARNING: No swap limit support
root@localhost:~# docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

root@hadoop-yarn-three:~# uname -a

Linux localhost 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

如果哪位发现docker解决了这个issue了，请帖出docker version和机器信息，多谢~

@Tom哥，说 rhel6.5 内核版本2.6.32，docker version 1.1.2, 没有这个问题

@IT人， 说ubuntu 14.04  centos7.0  7.1，没有这个问题

我还没有验证过。