k8s故障记录

一、etcd故障修改

　　1、利用已存活的etcd做个备份

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379" snapshot save snapshot.db

　　2、从集群中剔除有问题的节点

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379,https://192.168.112.111:2379,https://192.168.112.112:2379" member list

　　3、删除掉坏的节点

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379,https://192.168.112.111:2379,https://192.168.112.112:2379" member remove 6627a32423113ab8

　　4、修改坏掉节点的配置文件

　　如果是旧机器，请先清除etcd的数据存储目录里面的文件。

-initial-cluster-state 由"new" 改成"existing"

　　5、在启动新节点之前，必须把新节点接入集群

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379,https://192.168.112.111:2379,https://192.168.112.112:2379"  member add etcd-3 --peer-urls=https://192.168.112.112:2380

　　6、启动节点

systemctl start etcd

　　7、再次检查集群状态

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379,https://192.168.112.111:2379,https://192.168.112.112:2379" endpoint health
https://192.168.112.111:2379 is healthy: successfully committed proposal: took = 17.633542ms
https://192.168.112.112:2379 is healthy: successfully committed proposal: took = 18.544015ms
https://192.168.112.110:2379 is healthy: successfully committed proposal: took = 19.061029ms

二、harbor私有仓库引起rancher集群报错

　　1、在master节点上起了一个harbor仓库，rancher访问不了