etcd,Docker问题汇总

• 单节点etcd publish error

正在愉快的进行jenkins流程，突然发现etcd连接不上去了。重新reboot后发现日志publish error

Oct 31 10:22:42 k8s-master etcd: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379,http://etcd:4001
Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag 
Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag 
Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag 
Oct 31 10:22:42 k8s-master etcd: etcd Version: 3.1.3
Oct 31 10:22:42 k8s-master etcd: Git SHA: 21fdcc6
Oct 31 10:22:42 k8s-master etcd: Go Version: go1.7.4
Oct 31 10:22:42 k8s-master etcd: Go OS/Arch: linux/amd64
Oct 31 10:22:42 k8s-master etcd: setting maximum number of CPUs to 1, total number of available CPUs is 1
Oct 31 10:22:42 k8s-master etcd: the server is already initialized as member before, starting as etcd member...
Oct 31 10:22:42 k8s-master etcd: listening for peers on http://localhost:2380
Oct 31 10:22:42 k8s-master etcd: listening for client requests on 0.0.0.0:2379
Oct 31 10:22:42 k8s-master etcd: listening for client requests on 0.0.0.0:4001
Oct 31 10:22:42 k8s-master etcd: recovered store from snapshot at index 210021
Oct 31 10:22:42 k8s-master etcd: name = master
Oct 31 10:22:42 k8s-master etcd: data dir = /var/lib/etcd/default.etcd
Oct 31 10:22:42 k8s-master etcd: member dir = /var/lib/etcd/default.etcd/member
Oct 31 10:22:42 k8s-master etcd: heartbeat = 100ms
Oct 31 10:22:42 k8s-master etcd: election = 1000ms
Oct 31 10:22:42 k8s-master etcd: snapshot count = 10000
Oct 31 10:22:42 k8s-master etcd: advertise client URLs = http://etcd:2379,http://etcd:4001
Oct 31 10:22:42 k8s-master etcd: ignored file 0000000000000001-0000000000012700.wal.broken in wal
Oct 31 10:22:42 k8s-master etcd: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 215587
Oct 31 10:22:42 k8s-master etcd: 8e9e05c52164694d became follower at term 19
Oct 31 10:22:42 k8s-master etcd: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 19, commit: 215587, applied: 210021, lastindex: 215587, lastterm: 19]
Oct 31 10:22:42 k8s-master etcd: enabled capabilities for version 3.1
Oct 31 10:22:42 k8s-master etcd: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
Oct 31 10:22:42 k8s-master etcd: set the cluster version to 3.1 from store
Oct 31 10:22:42 k8s-master etcd: starting server... [version: 3.1.3, cluster version: 3.1]
Oct 31 10:22:49 k8s-master etcd: publish error: etcdserver: request timed out
Oct 31 10:22:56 k8s-master etcd: publish error: etcdserver: request timed out
Oct 31 10:23:03 k8s-master etcd: publish error: etcdserver: request timed out
Oct 31 10:23:10 k8s-master etcd: publish error: etcdserver: request timed out
Oct 31 10:23:17 k8s-master etcd: publish error: etcdserver: request timed out
Oct 31 10:23:24 k8s-master etcd: publish error: etcdserver: request timed out

查了一下，原因可能是因为主机压力比较大没有在5s内将信息写回照成不一致。

解决办法是删除data dir: /var/lib/etcd/default.etcd 中的数据，然后启动成功

不好的地方是，所有的信息基本全丢，flanneld信息又需要再写一次

etcdctl mk /atomic.io/network/config '{ "Network": "10.0.0.0/16" }'

各种pv,pvc,pod啥信息也没有了。。。。。

• Docker Service无法启动

在一台nodes节点上因为连接不上etcd导致启动失败，因为flannel组件需要连接etcd获取网段，docker服务依赖于flanned Service

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

分析命令主要包括

systemctl list-unit-files 列出所有可用的Unit
systemctl list-units 列出所有正在运行的Unit
systemctl --failed 列出所有失败单元
systemctl mask httpd.service 禁用服务
systemctl unmask httpd.service
systemctl kill httpd 杀死服务

解决办法，使用systemctl unmask flanneld.service禁止flanneld服务，然后删除

/usr/lib/systemd/system/flanneld.service
/etc/systemd/system/docker.service.requires/flanneld.service，

使用systemctl daemon-reload重新加载服务配置文件，最后systemctl start docker.service，

发现docker启动成功了