K8S从入门到放弃系列-(5)kubernetes集群之kube-apiserver部署

摘要：

1、kube-apiserver为是整个k8s集群中的数据总线和数据中心，提供了对集群的增删改查及watch等HTTP Rest接口

2、kube-apiserver是无状态的，虽然客户端如kubelet可通过启动参数"--api-servers"指定多个api-server，但只有第一个生效，并不能达到高可用的效果，关于kube-apiserver高可用方案，我们在后面介绍，本章，之介绍如何安装。

创建k8s集群各组件运行用户

安全性考虑，我们创建单独的用户运行k8s中各组件

[root@k8s-master01 ~]# ansible k8s-master -m group -a 'name=kube'
[root@k8s-master01 ~]# ansible k8s-master -m user -a 'name=kube group=kube comment="Kubernetes user" shell=/sbin/nologin createhome=no'

1）创建kube-apiserver证书请求文件

apiserver TLS 认证端口需要的证书

[root@k8s-master01 ~]# vim /opt/k8s/certs/kube-apiserver-csr.json
  {
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.10.0.18",
    "10.10.0.19",
    "10.10.0.20",
    "10.254.0.1",
    "localhost",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

hosts字段列表中，指定了master节点ip，本地ip，10.254.0.1为集群service ip一般为设置的网络段中第一个ip

2）生成 kubernetes 证书和私钥

[root@k8s-master01 certs]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem 
     -ca-key=/etc/kubernetes/ssl/ca-key.pem 
     -config=/opt/k8s/certs/ca-config.json 
     -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
2019/04/23 16:56:52 [INFO] generate received request
2019/04/23 16:56:52 [INFO] received CSR
2019/04/23 16:56:52 [INFO] generating key: rsa-2048
2019/04/23 16:56:52 [INFO] encoded CSR
2019/04/23 16:56:52 [INFO] signed certificate with serial number 22317568679091080825926949538404731378745389881
2019/04/23 16:56:52 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3）查看证书生成

[root@k8s-master01 certs]# ll kube-apiserver*
-rw-r--r-- 1 root root 1277 Apr 23 16:56 kube-apiserver.csr
-rw-r--r-- 1 root root  489 Apr 23 16:56 kube-apiserver-csr.json
-rw------- 1 root root 1675 Apr 23 16:56 kube-apiserver-key.pem
-rw-r--r-- 1 root root 1651 Apr 23 16:56 kube-apiserver.pem

4）证书分发

[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver.pem dest=/etc/kubernetes/ssl'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver-key.pem dest=/etc/kubernetes/ssl'

5）配置kube-apiserver客户端使用的token文件

kubelet 启动时向 kube-apiserver发送注册信息，在双向的TLS加密通信环境中需要认证，手工为kubelet生成证书/私钥在node节点较少且数量固定时可行，采用TLS Bootstrapping 机制，可使大量的node节点自动完成向kube-apiserver的注册请求。

原理：kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求，kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token.csv 一致，如果一致则自动为 kubelet生成证书和秘钥。

[root@k8s-master01 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
fb8f04963e38858eab0867e8d2296d6b
[root@k8s-master01 ~]# vim /opt/k8s/cfg/bootstrap-token.csv
fb8f04963e38858eab0867e8d2296d6b,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
##分发token文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/bootstrap-token.csv dest=/etc/kubernetes/config/'

6）生成 apiserver RBAC 审计配置文件

官方说明：https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

[root@k8s-master01 ~]# vim /opt/k8s/cfg/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
##分发审计文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/audit-policy.yaml dest=/etc/kubernetes/config/'

7）编辑kube-apiserver核心文件

apiserver 启动参数配置文件,注意创建参数中涉及的日志目录，并授权kube用户访问

[root@k8s-master01 ~]# vim /opt/k8s/cfg/kube-apiserver.conf

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=10.10.0.18 --bind-address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--secure-port=6443"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://10.10.0.18:2379,https://10.10.0.19:2379,https://10.10.0.20:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"

# Add your own!
KUBE_API_ARGS=" --allow-privileged=true 
                --anonymous-auth=false 
                --alsologtostderr 
                --apiserver-count=3 
                --audit-log-maxage=30 
                --audit-log-maxbackup=3 
                --audit-log-maxsize=100 
                --audit-log-path=/var/log/kube-audit/audit.log 
                --audit-policy-file=/etc/kubernetes/config/audit-policy.yaml 
                --authorization-mode=Node,RBAC 
                --client-ca-file=/etc/kubernetes/ssl/ca.pem 

                --token-auth-file=/etc/kubernetes/config/bootstrap-token.csv 
                --enable-bootstrap-token-auth 
                --enable-garbage-collector 
                --enable-logs-handler 
                --endpoint-reconciler-type=lease 
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem 
                --etcd-certfile=/etc/kubernetes/ssl/etcd.pem 
                --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem 
                --etcd-compaction-interval=0s 
                --event-ttl=168h0m0s 
                --kubelet-https=true 
                --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem 
                --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem 
                --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem 
                --kubelet-timeout=3s 
                --runtime-config=api/all=true 
                --service-node-port-range=30000-50000 
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem 
                --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem 
                --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem 
                --v=2"

##分发参数配置文件，同时把参数中出现的IP修改为对应的本机IP
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/kube-apiserver.conf dest=/etc/kubernetes/config/'
##创建日志目录并授权
[root@k8s-master01 ~]# ansible k8s-master -m file -a 'path=/var/log/kube-audit state=directory owner=kube group=kube'

个别参数解释：

1. KUBE_API_ADDRESS：向集群成员通知apiserver消息的IP地址。这个地址必须能够被集群中其他成员访问。如果IP地址为空，将会使用--bind-address，如果未指定--bind-address，将会使用主机的默认接口地址
2. KUBE_API_PORT：用于监听具有认证授权功能的HTTPS协议的端口。如果为0，则不会监听HTTPS协议。 （默认值6443)
3. KUBE_ETCD_SERVERS：连接的etcd服务器列表
4. KUBE_ADMISSION_CONTROL：控制资源进入集群的准入控制插件的顺序列表
5. apiserver-count：集群中apiserver数量
6. KUBE_SERVICE_ADDRESSES: CIDR IP范围，用于分配service 集群IP。不能与分配给节点pod的任何IP范围重叠

kube-apiserver启动脚本配置文件kube-apiserver.service

[root@k8s-master01 ~]# vim /opt/k8s/unit/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config/kube-apiserver.conf
User=kube
ExecStart=/usr/local/bin/kube-apiserver 
        $KUBE_LOGTOSTDERR 
        $KUBE_LOG_LEVEL 
        $KUBE_ETCD_SERVERS 
        $KUBE_API_ADDRESS 
        $KUBE_API_PORT 
        $KUBELET_PORT 
        $KUBE_ALLOW_PRIV 
        $KUBE_SERVICE_ADDRESSES 
        $KUBE_ADMISSION_CONTROL 
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
## 分发apiserver启动脚本文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/kube-apiserver.service dest=/usr/lib/systemd/system/'

8）启动kube-apiserver 服务

[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable kube-apiserver'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start kube-apiserver'

 9）授予 kubernetes 证书访问 kubelet API 的权限

[root@k8s-master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

后面部署好集群，在执行 kubectl exec、run、logs 等命令时，apiserver 会转发到 kubelet。这里定义 RBAC 规则，授权 apiserver 调用 kubelet API，否则会报类似以下错误：

Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-8477bdff5d-2lf7k)