kubernetes部署kube-controller-manager服务

本文档介绍部署高可用 kube-controller-manager 集群的步骤。

该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

以下是非认证的的配置文件（用127.0.0.1连接kube-apiserver）：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager 
  --address=127.0.0.1 
  --master=http://127.0.0.1:8080 
  --allocate-node-cidrs=true 
  --service-cluster-ip-range=10.254.0.0/16 
  --cluster-cidr=172.30.0.0/16 
  --cluster-name=kubernetes 
  --leader-elect=true 
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem 
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem 
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem 
  --root-ca-file=/etc/kubernetes/ca/ca.pem 
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

这个简单很多，但是以后可能会取消，如果要使用认证授权的方式看下面：

------------------------------------------------------------------

为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：

与 kube-apiserver 的安全端口通信时;
在安全端口(https，10252) 输出 prometheus 格式的 metrics；

创建 kube-controller-manager 证书和私钥

cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.111.10",
      "192.168.111.11",
      "192.168.111.12"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "ChongQing",
        "L": "ChongQing",
        "O": "system:kube-controller-manager",
        "OU": "yunwei"
      }
    ]
}
EOF


cfssl gencert -ca=/etc/kubernetes/ca/ca.pem 
  -ca-key=/etc/kubernetes/ca/ca-key.pem 
  -config=/etc/kubernetes/ca/ca-config.json 
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

创建和分发 kubeconfig 文件

kubeconfig 文件包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书；

kubectl config set-cluster kubernetes
--certificate-authority=/etc/kubernetes/ca/ca.pem
--embed-certs=true
--server=https://192.168.111.9:8443
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager
--client-certificate=/etc/kubernetes/ca/kube-controller-manager.pem
--client-key=/etc/kubernetes/ca/kube-controller-manager-key.pem
--embed-certs=true
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager
--cluster=kubernetes
--user=system:kube-controller-manager
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

分发 kubeconfig 到所有 master 节点：

# scp kube-controller-manager.kubeconfig 192.168.111.11:/etc/kubernetes/
# scp kube-controller-manager.kubeconfig 192.168.111.12:/etc/kubernetes/

创建和分发 kube-controller-manager systemd unit 文件：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager 
  --port=0 
  --secure-port=10252 
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig 
  --service-cluster-ip-range=10.254.0.0/16 
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem 
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem 
  --root-ca-file=/etc/kubernetes/ca/ca.pem 
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem 
  --leader-elect=true 
  --feature-gates=RotateKubeletServerCertificate=true 
  --controllers=*,bootstrapsigner,tokencleaner 
  --horizontal-pod-autoscaler-use-rest-clients=true 
  --horizontal-pod-autoscaler-sync-period=10s 
  --tls-cert-file=/etc/kubernetes/ca/kube-controller-manager.pem 
  --tls-private-key-file=/etc/kubernetes/ca/kube-controller-manager-key.pem 
  --use-service-account-credentials=true 
  --alsologtostderr=true 
  --logtostderr=false 
  --log-dir=/var/log/kubernetes 
  --v=2
Restart=on
Restart=on-failure
RestartSec=5
User=k8s

[Install]
WantedBy=multi-user.target
EOF

-------------------------------------------------

启动kube-controller-manager

# systemctl daemon-reload&&for SERVICES in kube-controller-manager;do systemctl enable $SERVICES; systemctl start $SERVICES; systemctl status $SERVICES; done

查看输出的 metric
注意：以下命令在 kube-controller-manager 节点上执行。

kube-controller-manager 监听 10252 端口，接收 https 请求：

# ss -lnpt|grep kube-controll

# curl http://127.0.0.1:10252/metrics |head

测试 kube-controller-manager 集群的高可用
停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。
查看当前的 leader
kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml