背景：平时运行的好好的k8s集群，早上过来一看发现好多红色指标，马上查 下集群状况，发现一个节点跪了。
1
可以确定work01是真的跪了了，那么如何查找问题呢？ 我们用它：journalctl工具是CentOS-7才有的工具
Systemd 统一管理所有 Unit 的启动日志。带来的好处就是 ，可以只用journalctl一个命令，查看所有日志（内核日志和 应用日志）。日志的配置文件/etc/systemd/journald.conf
journalctl用法：
查看所有日志（默认情况下 ，只保存本次启动的日志）： journalctl
查看内核日志（不显示应用日志）： journalctl -k
查看系统本次启动的日志： journalctl -b
查看上一次启动的日志（需更改设置）：
在该[Journal]部分下，将该Storage=选项设置为“persistent”以启用持久记录：
vim /etc/systemd/journald.conf
. . .
[Journal]
Storage=persistent
查看httpd服务的日志信息：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
我们使用的是：
追踪日志
要主动追踪当前正在编写的日志，大家可以使用-f标记。同样功能类似为tail -f，只要不终止，会一直监控
journalctl -f
也许最有用的过滤方式是你感兴趣的单位。我们可以使用这个-u选项来过滤我们可以使用这个-u选项来过滤
journalctl -u
所以我们最终使用的命令是：
journalctl -f -u kubelet
1
2
3
4
5
6
7
8
9
[root@***-***-work01 ~]# journalctl -f -u kubelet
-- Logs begin at Wed 2018-09-05 11:16:12 CST. --
Sep 05 11:43:30 ***-***-work01 kubelet[5764]: E0905 11:43:30.325784 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.048633 5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.304730 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:34 ***-***-work01 kubelet[5764]: E0905 11:43:34.325829 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:36 ***-***-work01 kubelet[5764]: E0905 11:43:36.293212 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:38 ***-***-work01 kubelet[5764]: E0905 11:43:38.301224 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:40 ***-***-work01 kubelet[5764]: E0905 11:43:40.330581 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:42 ***-***-work01 kubelet[5764]: E0905 11:43:42.289427 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:44 ***-***-work01 kubelet[5764]: E0905 11:43:44.311478 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:46 ***-***-work01 kubelet[5764]: E0905 11:43:46.296680 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:48 ***-***-work01 kubelet[5764]: E0905 11:43:48.320159 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:50 ***-***-work01 kubelet[5764]: E0905 11:43:50.295335 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.049269 5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.332253 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:54 ***-***-work01 kubelet[5764]: E0905 11:43:54.319333 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:56 ***-***-work01 kubelet[5764]: E0905 11:43:56.298336 5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
NOTE:因为我的原因，很明显，work01节点是机器直接跪了，
最后重新启动就OK了。
systemctl restart docker
————————————————
版权声明：本文为CSDN博主「common_util」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/shenhonglei1234/article/details/82421742