011.Kubernetes二进制master节点部署kube-controller-manager

一 部署kube-controller-manager

1.1 高可用kube-controller-manager介绍

本实验部署一个三实例 kube-controller-manager 的集群，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：

• 与 kube-apiserver 的安全端口通信;
• 在安全端口(https，10252) 输出 prometheus 格式的 metrics。

1.2 创建kube-controller-manager证书和私钥

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [
    "127.0.0.1",
    "172.24.8.71",
    "172.24.8.72",
    "172.24.8.73",
    "172.24.8.100"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "system:kube-controller-manager",
      "OU": "System"
    }
  ]
}
EOF		#创建kube-controller-manager的CA证书请求文件

解释：

hosts 列表包含所有 kube-controller-manager 节点 IP；

CN 和 O 均为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 
-ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json 
-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager	#生成密钥和证书

提示：本步骤操作仅需要在master01节点操作。

1.3 分发证书和私钥

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${master_ip}"
    scp kube-controller-manager*.pem root@${master_ip}:/etc/kubernetes/cert/
  done

提示：本步骤操作仅需要在master01节点操作。

1.4 创建和分发kubeconfig

kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书：

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# kubectl config set-cluster kubernetes 
  --certificate-authority=/opt/k8s/work/ca.pem 
  --embed-certs=true 
  --server=${KUBE_APISERVER} 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@master01 work]# kubectl config set-credentials system:kube-controller-manager 
  --client-certificate=kube-controller-manager.pem 
  --client-key=kube-controller-manager-key.pem 
  --embed-certs=true 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@master01 work]# kubectl config set-context system:kube-controller-manager 
  --cluster=kubernetes 
  --user=system:kube-controller-manager 
  --kubeconfig=kube-controller-manager.kubeconfig

[root@master01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${master_ip}"
    scp kube-controller-manager.kubeconfig root@${master_ip}:/etc/kubernetes/
  done

提示：本步骤操作仅需要在master01节点操作。

1.5 创建kube-controller-manager的systemd

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \
  --secure-port=10257 \
  --bind-address=127.0.0.1 \
  --profiling \
  --cluster-name=kubernetes \
  --controllers=*,bootstrapsigner,tokencleaner \
  --kube-api-qps=1000 \
  --kube-api-burst=2000 \
  --leader-elect \
  --use-service-account-credentials\
  --concurrent-service-syncs=2 \
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --client-ca-file=/etc/kubernetes/cert/ca.pem \
  --requestheader-allowed-names="system:metrics-server" \
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
  --experimental-cluster-signing-duration=87600h \
  --horizontal-pod-autoscaler-sync-period=10s \
  --concurrent-deployment-syncs=10 \
  --concurrent-gc-syncs=30 \
  --node-cidr-mask-size=24 \
  --service-cluster-ip-range=${SERVICE_CIDR} \
  --cluster-cidr=${CLUSTER_CIDR} \
  --pod-eviction-timeout=6m \
  --terminated-pod-gc-threshold=10000 \
  --root-ca-file=/etc/kubernetes/cert/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

提示：本步骤操作仅需要在master01节点操作。

1.6 分发systemd

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${master_ip}"
    scp kube-controller-manager.service.template root@${master_ip}:/etc/systemd/system/kube-controller-manager.service
  done						#分发system

提示：本步骤操作仅需要在master01节点操作。

二 启动并验证

2.1 启动kube-controller-manager 服务

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${master_ip}"
    ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
    ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
  done

提示：本步骤操作仅需要在master01节点操作。

2.2 检查kube-controller-manager 服务

[root@master01 ~]# cd /opt/k8s/work
[root@master01 work]# source /root/environment.sh
[root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${master_ip}"
    ssh root@${master_ip} "systemctl status kube-controller-manager|grep Active"
  done

提示：本步骤操作仅需要在master01节点操作。

2.3 查看输出的 metrics

[root@master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://127.0.0.1:10257/metrics | head

提示：本步骤操作仅需要在master01节点操作。

2.4 查看权限

[root@master01 ~]# kubectl describe clusterrole system:kube-controller-manager

ClusteRole system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中。

当在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。

[root@master01 ~]# kubectl get clusterrole | grep controller

如deployment controller：

[root@master01 ~]# kubectl describe clusterrole system:controller:deployment-controller

提示：本步骤操作仅需要在master01节点操作。

2.5 查看当前leader

[root@master01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml

kubelet 认证和授权：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization