安全机制

Kubernetes的安全框架（总的来说就是各种形式访问API Server）

访问K8S集群的资源需要过三关：认证、鉴权、准入控制
普通用户若要安全访问集群API Server，往往需要证书、Token或者用户名+密码；Pod访问API Server，需要ServiceAccount

K8S安全控制框架主要由下面3个阶段进行控制，每一个阶段都支持插件方式，通过API Server配置来启用插件。
1.Authentication
2.Authorization
3.Admission Control

[root@master01 yaml_doc]# kubectl get sa
NAME      SECRETS   AGE
default   1         11d
[root@master01 yaml_doc]# 
[root@master01 yaml_doc]# kubectl get secrets
NAME                   TYPE                                  DATA   AGE
default-token-sj2lw    kubernetes.io/service-account-token   3      15d
#token是用secret资源类型存储的 会生成一个跟ServiceAccount：pod-reader同名的secret  （在同一个命名空间下）
[root@master01 yaml_doc]# kubectl describe secret default-token-sj2lw
Name:         default-token-sj2lw
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 1c1e6043-0430-11ea-9860-1866dafb2f54

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1359 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tc2oybHciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjMWU2MDQzLTA0MzAtMTFlYS05ODYwLTE4NjZkYWZiMmY1NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.b1CuohIrZi_ubpkoYQAQKzCELgWeGSbLAEiCTyN7mIZj1CVro4SkftoaTYisyJnhmMKw8PQyvDunKOLa-tkE1C-XeAVh_rcno80NXOD9Uz33CijUPNZlHNwx0CimjGS_AA5yHoUXmLxAsm-cOq-SkWaawtUfxnRAQVv2TKEjfmha9AaLPw6Gj4-DgBPPhV8pWpBNnuuwDR-tKyyWUTAIE8wGDDhIYyIgKgzo_q7MCOupQpYUiUzwu83B5DTmimHXVcbZ3ffkR1X198KIS3lUu_fR997Pn2ZoxK5mjZ1PmzK6I0gUhSfU7wI8_AUy67sKmRUEJ2ORcAVkk-KQ_PaXUw

1、Authentication #传输安全，认证  相当于进公司大门

传输安全：告别8080，迎接6443
[root@master01 yaml_doc]# netstat -anput | grep 8080 | grep LISTEN    #不对外
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      169560/kube-apiserv 
[root@master01 yaml_doc]# 
[root@master01 yaml_doc]# netstat -anput | grep 6443 | grep LISTEN
tcp        0      0 10.192.27.100:6443      0.0.0.0:*               LISTEN      169560/kube-apiserv 
[root@master01 yaml_doc]# 

认证
三种客户端身份认证：
•HTTPS 证书认证：基于CA证书签名的数字证书认证
[root@master01 yaml_doc]# ls /root/k8s/k8s-cert/  #不同的组件通过证书访问API Server
admin.csr       admin-key.pem  ca-config.json  ca-csr.json  ca.pem       kube-proxy.csr       kube-proxy-key.pem  server.csr       server-key.pem
admin-csr.json  admin.pem      ca.csr          ca-key.pem   k8s-cert.sh  kube-proxy-csr.json  kube-proxy.pem      server-csr.json  server.pem
[root@master01 yaml_doc]# 

•HTTP Token认证：通过一个Token来识别用户
[root@master01 yaml_doc]# cat /opt/kubernetes/cfg/token.csv   #kubelet-bootstrap认证使用
0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@master01 yaml_doc]# 

•HTTP Base认证：用户名+密码的方式认证

2、Authorization  #授权  相当于授权进入那个办公区

RBAC（Role-Based Access Control，基于角色的访问控制）：负责完成授权（Authorization）工作。

3、Admission Control #准入控制  相当于验证你的权限

AdminssionControl实际上是一个准入控制器插件列表，发送到APIServer的请求都需要经过这个列表中的每个准入控制器插件的检查，检查不通过，则拒绝请求。
1.11版本以上推荐使用的插件：
--enable-admission-plugins= 
NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds, ResourceQuota

[root@master01 yaml_doc]# ps -ef | grep apiserver
root      85972  18645  0 10:36 pts/0    00:00:00 grep --color=auto apiserver
root     169560      1  3 11月11 ?      10:17:24 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.192.27.100:2379,https://10.192.27.115:2379,https://10.192.27.116:2379 --bind-address=10.192.27.100 --secure-port=6443 --advertise-address=10.192.27.100 --allow-privileged=true  --service-cluster-ip-range=10.0.0.0/24 
准入控制字段设置：--enable-admission-plugins=NamespaceLifecycle（命令控制）,LimitRanger（限制pod使用的资源CPU 内存）, 
ServiceAccount（会在每个pod中导入默认的serviceaccount，使pod有权限访问API server）,ResourceQuota（高级的配额管理）,NodeRestriction（node加入进来以最小的权限运行） 
 
 --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem
[root@master01 yaml_doc]# 

着重介绍第二个授权阶段

•角色
Role：授权特定命名空间的访问权限
ClusterRole：授权所有命名空间的访问权限 #集群层面
•角色绑定
RoleBinding：将角色绑定到主体（即subject）
ClusterRoleBinding：将集群角色绑定到主体
•主体（subject）
User：用户  #给使用人自己用的
Group：用户组
ServiceAccount：服务账号  #给程序用的

k8s实现多租户不同部门不同项目之间的隔离一般通过命名空间
[root@master01 yaml_doc]#  kubectl get ns  #默认命名空间
NAME          STATUS   AGE
default       Active   11d
kube-public   Active   11d
kube-system   Active   11d
[root@master01 yaml_doc]# kubectl create ns ctnrs  #创建命名空间
namespace/ctnrs created
[root@master01 yaml_doc]#  kubectl get ns
NAME          STATUS   AGE
ctnrs         Active   2s
default       Active   11d
kube-public   Active   11d
kube-system   Active   11d
[root@master01 yaml_doc]# 


两个node节点配置一下 --pod-infra-container-image=10.192.27.111/library/pause-amd64:3.0 然后重启服务
[root@node01 cfg]# ps -ef | grep kubelet
root     166978      1  9 11:10 ?        00:00:00 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.192.27.115 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=10.192.27.111/library/pause-amd64:3.0

[root@master01 yaml_doc]# kubectl run nginx --image=10.192.27.111/library/nginx:latest -n ctnrs
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx created


[root@master01 yaml_doc]# kubectl get all -n ctnrs
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-644d56598-n4hnv   1/1     Running   0          3m19s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           3m19s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-644d56598   1         1         1       3m19s
[root@master01 yaml_doc]# 


[root@master01 yaml_doc]# kubectl scale deploy/nginx --replicas=3 -n ctnrs  #调整副本数
deployment.extensions/nginx scaled
[root@master01 yaml_doc]# kubectl get pods -n ctnrs
NAME                    READY   STATUS              RESTARTS   AGE
nginx-644d56598-bqkg4   0/1     ContainerCreating   0          2s
nginx-644d56598-dtcn9   0/1     ContainerCreating   0          2s
nginx-644d56598-n4hnv   1/1     Running             0          4m47s
[root@master01 yaml_doc]# 

来一个例子吧

#创建一个role角色
[root@master01 yaml_doc]# vim rbac-role.yaml  
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ctnrs
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]  #只能操作pod资源
  verbs: ["get", "watch", "list"] #只读

[root@master01 yaml_doc]# kubectl create -f rbac-role.yaml 
role.rbac.authorization.k8s.io/pod-reader created
[root@master01 yaml_doc]# kubectl get role -n ctnrs
NAME         AGE
pod-reader   16s
[root@master01 yaml_doc]#

######################测试方式一 用户测试###########################
[root@master01 yaml_doc]# vim rbac-rolebinding.yaml #绑定角色
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: ctnrs
subjects: #指定用户
- kind: User
  name: jane  #用户名
  apiGroup: rbac.authorization.k8s.io
roleRef:  #绑定哪个角色
  kind: Role
  name: pod-reader #角色名
  apiGroup: rbac.authorization.k8s.io
[root@master01 yaml_doc]# kubectl create -f rbac-rolebinding.yaml 
rolebinding.rbac.authorization.k8s.io/read-pods created


[root@master01 yaml_doc]# kubectl get role,rolebinding -n ctnrs
NAME                                        AGE
role.rbac.authorization.k8s.io/pod-reader   9m1s

NAME                                              AGE
rolebinding.rbac.authorization.k8s.io/read-pods   2m19s
[root@master01 yaml_doc]# 


#生成证书  以上用户如何识别身份
[root@master01 yaml_doc]# mkdir jane
[root@master01 yaml_doc]# cd jane/
[root@master01 jane]# vim rabc-user.sh
cat > jane-csr.json <<EOF
{
  "CN": "jane",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes jane-csr.json | cfssljson -bare jane  
#设置集群IP 、证书、上下文等配置 生成用户jane的访问集群APIserver的配置文件
kubectl config set-cluster kubernetes 
  --certificate-authority=ca.pem 
  --embed-certs=true 
  --server=https://10.192.27.117:6443 
  --kubeconfig=jane-kubeconfig

kubectl config set-credentials jane 
  --client-key=jane-key.pem 
  --client-certificate=jane.pem 
  --embed-certs=true 
  --kubeconfig=jane-kubeconfig

kubectl config set-context default 
  --cluster=kubernetes 
  --user=jane 
  --kubeconfig=jane-kubeconfig

#使用默认上下文
kubectl config use-context default --kubeconfig=jane-kubeconfig  
[root@master01 jane]# cp /root/k8s/k8s-cert/ca* .           #用到根证书（kube-apiserver是的证书） 生成用户客户端证书
[root@master01 jane]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  rabc-user.sh
[root@master01 jane]# bash rabc-user.sh 
2019/09/17 20:54:24 [INFO] generate received request
2019/09/17 20:54:24 [INFO] received CSR
2019/09/17 20:54:24 [INFO] generating key: rsa-2048
2019/09/17 20:54:25 [INFO] encoded CSR
2019/09/17 20:54:25 [INFO] signed certificate with serial number 30365589649039563082592558185944057795826217362
2019/09/17 20:54:25 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
Cluster "kubernetes" set.
User "jane" set.
Context "default" created.
Switched to context "default".
[root@master01 jane]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  jane.csr  jane-csr.json  jane-key.pem  jane-kubeconfig  jane.pem  rabc-user.sh
[root@master01 jane]# 
[root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get pods -n ctnrs  #根据上面的权限（只有ctnrs命令空间的pod只读权限） 查看指定的内容
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7cdbd8cdc9-7rc5x   1/1     Running   0          29m
nginx-7cdbd8cdc9-l4zlj   1/1     Running   0          27m
nginx-7cdbd8cdc9-s8ggw   1/1     Running   0          27m
[root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get svc -n ctnrs 
Error from server (Forbidden): services is forbidden: User "jane" cannot list resource "services" in API group "" in the namespace "ctnrs"
[root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get svc    #default空间没有权限
Error from server (Forbidden): services is forbidden: User "jane" cannot list resource "services" in API group "" in the namespace "default"
[root@master01 jane]# kubectl --kubeconfig=jane-kubeconfig get pods    #default空间没有权限
Error from server (Forbidden): pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "default"
[root@master01 jane]# 

#########################测试方式二  ServiceAccount：服务账号测试  #给程序用的 ###########
#使用web UI  https://10.192.27.115:30001/ 或者116
[root@master01 jane]# kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP   4d11h
kubernetes-dashboard   NodePort    10.0.0.141   <none>        443:30001/TCP   13d

[root@master01 jane]# kubectl get secret -n kube-system
NAME                               TYPE                                  DATA   AGE
coredns-token-bkjq9                kubernetes.io/service-account-token   3      4d11h  
dashboard-admin-token-x2fmd        kubernetes.io/service-account-token   3      13d  #超级管理员
default-token-gkwdg                kubernetes.io/service-account-token   3      14d
kubernetes-dashboard-certs         Opaque                                11     13d
kubernetes-dashboard-key-holder    Opaque                                2      13d
kubernetes-dashboard-token-2qxt6   kubernetes.io/service-account-token   3      13d


[root@master01 jane]# vim sa.yaml   
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-reader  #创建sa对应下 这个可以改
  namespace: ctnrs

---

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sa-read-pods
  namespace: ctnrs
subjects:
- kind: ServiceAccount
  name: pod-reader #对应上SA名字 这个可以改
roleRef:
  kind: Role
  name: pod-reader #上面创建的的角色
  apiGroup: rbac.authorization.k8s.io
[root@master01 jane]# kubectl apply -f sa.yaml 
serviceaccount/pod-reader created
rolebinding.rbac.authorization.k8s.io/sa-read-pods created
[root@master01 jane]# kubectl get sa -n ctnrs
NAME         SECRETS   AGE
default      1         42m
pod-reader   1         23s
[root@master01 jane]# 
[root@master01 jane]# kubectl describe sa pod-reader -n ctnrs
Name:                pod-reader
Namespace:           ctnrs
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"pod-reader","namespace":"ctnrs"}}
Image pull secrets:  <none>
Mountable secrets:   pod-reader-token-5vmrr
Tokens:              pod-reader-token-5vmrr
Events:              <none>
[root@master01 jane]# kubectl describe secret pod-reader -n ctnrs  #token是用secret资源类型存储的 会生成一个跟ServiceAccount：pod-reader同名的secret  （在同一个命名空间下）
Name:         pod-reader-token-5vmrr
Namespace:    ctnrs
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: pod-reader
              kubernetes.io/service-account.uid: 7b1a812f-d94c-11e9-8a2e-000c29586be2

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1359 bytes
namespace:  5 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjdG5ycyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwb2QtcmVhZGVyLXRva2VuLTV2bXJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InBvZC1yZWFkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3YjFhODEyZi1kOTRjLTExZTktOGEyZS0wMDBjMjk1ODZiZTIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Y3RucnM6cG9kLXJlYWRlciJ9.JyiOaiL79AVAs59QRQY7fyPn2Ukc3b52NejeuUZK5JlHJN7rE6mLqZnJtEm2ef5Yp07rVpf0IAqWzEOcPRC6uKzYKT544BxQxoEHOP1GNaWFDh6JHIeMHP1Ak-4FqmI9q7a7YyCYrvR7RtMuTdUr8G0Fo2gyNzlIkp1UlJH1-LpoAR0BG5MBtFzj4trxlGSIzVHtAHoIXXcG5Ke0mEHWEt0zWEFUJ5r-y-X99HDR3n5yER43djywjpncxzJ426Rm-R3azjpHUZ3LP4urkIwSWocIxuVcGAay5EgX9sHcZb5a-1fT3IZPX77ATK9LDcZfG0UoPYgDjbUPw4BqLq9HGg

[root@master01 jane]# 

使用web UI  https://10.192.27.115:30001/ 或者116