k8s证书更新

目录

• 来自github
• 操作步骤
  • 前置工作
  • 进入目录
  • 修改配置文件
• 出现的问题
  • 关于kube-proxy
  • 关于calico和calico-controller-manage
  • 关于dns

来自github

https://github.com/misrn/k8s-tls

操作步骤

前置工作

1. 备份原有证书目录
    mv /etc/kubernetes /etc/kubernetes.bak
2. 集群说明
    版本: 1.13.3
    节点: 
           集群一共有三个节点(三个节点都是master,worker)
           192.168.122.11  node1   
           192.168.122.12  node2
           192.168.122.13  node3

进入目录

此操作我node1上执行的

1. cd /usr/src
2. git clone https://github.com/fandaye/k8s-tls.git && cd k8s-tls/
3. chmod +x ./run.sh

修改配置文件

• apiserver.json

{
     "CN": "kube-apiserver",
     "hosts": [
       "192.168.122.11",  # master兼node
       "192.168.122.12",  # master兼node
       "192.168.122.13",  # master兼node
       "10.233.0.1",  # k8s 中 kubernetes svc IP地址
       "127.0.0.1",    
       "172.17.0.1",   # docker0网卡ip地址
       "node1",
       "node2",
       "node3"
       "kubernetes",
       "kubernetes.default",
       "kubernetes.default.svc",
       "kubernetes.default.svc.cluster",
       "kubernetes.default.svc.cluster.local"     
     ],
     "key": {
         "algo": "rsa",
         "size": 2048
     }
 }

注意: 如果不清楚要加哪些IP，可以参考正常运行的集群中apiserver.crt例如:
openssl x509 -in apiserver.crt -text -noout

• config.yaml

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: v1.13.3
networking:
  podSubnet: 10.233.64.0/18  # 修改为自己集群pod的地址段
apiServerCertSANs:
- node1
- node2
- node3
- 192.168.122.11
- 192.168.122.12
- 192.168.122.13
apiServerExtraArgs:
  endpoint-reconciler-type: "lease"
etcd:
  endpoints:
  - https://192.168.122.11:2379
  - https://192.168.122.12:2379
  - https://192.168.122.13:2379
token: "deed3a.b3542929fcbce0f0"
tokenTTL: "0"

目前就只需要修改这两个文件

• 运行run.sh及复制配置文件

./run.sh
# 运行该脚本后，会生成/etc/kubernetes/pki目录
# 拷贝kubelet.env文件和manifests目录到各个master节点

• 进入/etc/kubernetes/pki/编辑node.sh文件（该脚本需要在每个机器上运行）

说明:  目前在node1上
修改其中的两个参数  
ip="192.168.122.11"  # node1对应的IP 如果在其他node上执行，对应修改IP就行
NODE="node1"  # 如果在其他node上执行，对应修改为hostname就行

• 更新~/.kube/config文件

cp /etc/kubernetes/admin.conf ~/.kube/config

出现的问题

证书更新后重启了docker和kubelet，apiserver controller-manage schedu 运行正常，但是kube-proxy dns kube-proxy calico 都有均出现了问题

• kube-proxy
  出现无法连接到127.0.0.1:6443 报证书错误
  分析: 本机的127.0.01:6443端口是通的，所以排除apiserver服务本身的问题。
  用kubectl查看了kube-proxy相关信息(sa secret clusterrole等等)发现使用的还是未过期时的，所以基本找到问题了，需要构建这些有关kube-proxy信息
• calico 和 calico-controller-manage
  calico-controller-manage:
  calico: network plugin unauthri
  kubectl查看到的calico相关信息虽然更新了，但还是出现了认证失败，所以还是需要重建相关资源
• dns
  无法连接到10.233.0.1 (这个错误是因为网络不通，所以根本问题是在kube-proxy和calico)

关于kube-proxy

从上面的分析来看，需要重新构建kube-proxy资源，以下是相关资源yaml文件

• sa.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-proxy
  namespace: kube-system

• ClusterRoleBinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node-proxier
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-proxier
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-proxy

• configmap.yaml

apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    clusterCIDR: 10.233.64.0/18
    configSyncPeriod: 15m0s
    conntrack:
      max: null
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: rr
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: ipvs
    nodePortAddresses: []
    oomScoreAdj: -999
    portRange: ""
    resourceContainer: /kube-proxy
    udpIdleTimeout: 250ms
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system

kube-proxy资源重建后又出现报错iptables-restore: line 7 failed in kube-proxy, google上说升级kube-proxy版本就OK了，于是将版本升级到1.16.3。嗯果然可以了

关于calico和calico-controller-manage

calico资源相关yaml文件

• calico-sa.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"

• calico-cr.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      - watch
      - list
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      - patch
  - apiGroups:
    - policy
    resourceNames:
    - privileged
    resources:
    - podsecuritypolicies
    verbs:
    - use

• calico-crb.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  etcd_endpoints: "https://192.168.122.11:2379,https://192.168.122.12:2379,https://192.168.122.13:2379"
  etcd_ca: "/calico-secrets/ca_cert.crt"
  etcd_cert: "/calico-secrets/cert.crt"
  etcd_key: "/calico-secrets/key.pem"
  cluster_type: "kubespray,bgp"
  calico_backend: "bird"

• calico-ds.yaml

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        # Mark pod as critical for rescheduling (Will have no effect starting with kubernetes 1.12)
        kubespray.etcd-cert/serial: "C9370ED3FE6243D3"
        prometheus.io/scrape: 'true'
        prometheus.io/port: "9091"
    spec:
      priorityClassName: system-node-critical
      hostNetwork: true
      serviceAccountName: calico-node
      tolerations:
        - effect: NoExecute
          operator: Exists
        - effect: NoSchedule
          operator: Exists
        # Mark pod as critical for rescheduling (Will have no effect starting with kubernetes 1.12)
        - key: CriticalAddonsOnly
          operator: "Exists"
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      initContainers:
        # This container installs the Calico CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: quay.io/calico/cni:v3.4.0-amd64
          command: ["/install-cni.sh"]
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # CNI binaries are already on the host
            - name: UPDATE_CNI_BINARIES
              value: "false"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG_FILE
              value: "/host/etc/cni/net.d/calico.conflist.template"
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
      containers:
        # Runs calico/node container on each Kubernetes node. This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: quay.io/calico/node:v3.4.0-amd64
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cluster_type
            # Set noderef for node controller.
            - name: CALICO_K8S_NODE_REF
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "RETURN"
            - name: FELIX_HEALTHHOST
              value: "localhost"
            # Prior to v3.2.1 iptables didn't acquire the lock, so Calico's own implementation of the lock should be used,
            # this is not required in later versions https://github.com/projectcalico/calico/issues/2179
# should be set in etcd before deployment
# # Configure the IP Pool from which Pod IPs will be chosen.
# - name: CALICO_IPV4POOL_CIDR
# value: "192.168.0.0/16"
            - name: CALICO_IPV4POOL_IPIP
              value: "Off"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_PROMETHEUSMETRICSENABLED
              value: "false"
            - name: FELIX_PROMETHEUSMETRICSPORT
              value: "9091"
            - name: FELIX_PROMETHEUSGOMETRICSENABLED
              value: "true"
            - name: FELIX_PROMETHEUSPROCESSMETRICSENABLED
              value: "true"
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: FELIX_HEALTHENABLED
              value: "true"
            - name: FELIX_IGNORELOOSERPF
              value: "False"
          securityContext:
            privileged: true
          resources:
            limits:
              cpu: 300m
              memory: 500M
            requests:
              cpu: 150m
              memory: 64M
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /liveness
              port: 9099
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -bird-ready
              - -felix-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /var/run/calico
              name: var-run-calico
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - mountPath: /calico-secrets
              name: etcd-certs
            - name: xtables-lock
              mountPath: /run/xtables.lock
              readOnly: false
      volumes:
        # Used by calico/node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        # Used to install CNI.
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Mount in the etcd TLS secrets.
        - name: etcd-certs
          hostPath:
            path: "/etc/calico/certs"
        # Mount the global iptables lock file, used by calico/node
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 20%
    type: RollingUpdate

关于dns

不再贴配置了，/etc/kubernetes.bak下有对应文件

OK。。。k8s证书更新到此结束啦