Kubernetes系统安全-准入控制(admission control)
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.准入控制(admission control)概述
1>.常见的准入控制器(Admission Controllers)
AlwaysAdmit(DEPRECATED):
该准入控制器已被废弃,总是允许，所有的请求(包括不符合定义)的规则都允许，如果有不符合的规则请求会日志记录而已，和临时关闭selinux的功能有点像。
AlwaysPullImages:
无论你拉取镜像的规则定义成什么类型，总是会去拉取镜像，也就是强制避免使用本地镜像，即无视本地镜像定义的拉取规则。
AlwaysDeny(DEPRECATED):
该准入控制器已被废弃,和AlwaysAdmit相反，所有请求(包括符合定义)的规则拒绝。
DefaultStorageClass:
指定默认的存储类。
DefaultTolerationSeconds
DenyEscalatingExec
EventRateLimit(alpha)
ExtendedResourceToleration
ImagePolicyWebhook
Initializers(alpha)
LimitPodHardAnitiAffinityTopology
LimitRanger:
允许在名称空间中创建一个LimitRange资源，任何Pod都需要指定一个资源上限和下限的范围，若未指定会使用默认值。
MutaingAdmission Webhook(beta in 1.9)
NamespaceAutoProvision
NamespaceExists:
检查名称空间是否存在的控制器。
NamespaceLifecycle
NodeRestriction
OwnerReferencesPermissionEnforcement
PodNodeSelector
PersistentVolumeClaimResize
PodPreset
PodSecurityPolicy:
为Pod定义安全策略。
PodTolerationRestriction
Priority
ResourceQuota:
允许在名称空间中创建一个ResourceQuota资源,可以明确指定Pod使用资源的配额，比如内存不允许超过20G，允许使用的总的PVC数量等K8S集群资源。
SecurityContextDeny
ServiceAccount:
比如创建的Pod内部默认有一个存储卷，每个存储卷对应了一个secret用于ApiServer认证,这就是ServiceAccount来定义的。
Storage Object in Use Protection
ValidatingAdmission Webhook(alpha in 1.8;beta in 1.9)
2>.打开或者关闭控制器
Kubernetes API server标志enable adminimission plugins接受在修改集群中的对象之前要调用的许可控制插件的逗号分隔列表。使用"--enable-admission-plugins="指定，如下图所示。 Kubernetes API服务器标志disable admission plugins接受一个逗号分隔的要禁用的许可控制插件列表，即使它们在默认启用的插件列表中。使用"--disable-admission-plugins="。
二.LimitRange and LimitRanger
Pod 对象虽然支持使用requests和limits进行可用计算资源配置，但它们却非强制选项; LimitRange的主要目的是确保请求和/限制根据其规范自动关联到容器; 如果容器是在具有默认内存限制的命名空间中创建的，并且该容器未指定其自己的内存限制，则会为该容器分配默认内存限制; 由limit range对象定义的限制范围枚举pod和container级别的命名空间中的计算资源约束，并指定pod或container可以消耗的资源量。 对项目中的每个LimitRange对象评估所有资源创建和修改请求，如果资源未设置显式值，并且约束支持默认值，则将默认值应用于资源。 LimitRanger将观察传入的请求，并确保它不违反命名空间中LimitRange对象中枚举的任何约束: 所有资源创建和修改请求都是根据命名空间中的每个LimitRange对象计算的; 如果资源违反任何枚举约束，则资源将被拒绝。 如果资源未设置显式值，并且约束支持默认值，则将默认值应用于资源。 LimitRanger还可用于将默认资源请求应用于未指定任何的pod；当前，默认LimitRanger将0.1 CPU需求应用于默认命名空间中的所有pod。
1>.创建LimitRange资源
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-limit-range namespace: yinzhengjie-admission-control spec: limits: - default: cpu: 1000m defaultRequest: cpu: 1000m min: cpu: 500m max: cpu: 2000m maxLimitRequestRatio: cpu: 4 type: Container [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl create ns yinzhengjie-admission-control namespace/yinzhengjie-admission-control created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get ns NAME STATUS AGE default Active 15d develop Active 13h kube-node-lease Active 15d kube-public Active 15d kube-system Active 15d kubernetes-dashboard Active 14h yinzhengjie-admission-control Active 7s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml limitrange/cpu-limit-range created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl describe limitrange cpu-limit-range -n yinzhengjie-admission-control Name: cpu-limit-range Namespace: yinzhengjie-admission-control Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 500m 2 1 1 4 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.使用默认的LimitRange
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml apiVersion: v1 kind: Pod metadata: name: pod-demo namespace: yinzhengjie-admission-control spec: containers: - image: ikubernetes/myapp:v1 imagePullPolicy: IfNotPresent name: myapp [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod/pod-demo created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control NAME READY STATUS RESTARTS AGE pod-demo 1/1 Running 0 18s [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe pods -n yinzhengjie-admission-control Name: pod-demo Namespace: yinzhengjie-admission-control Priority: 0 Node: node203.yinzhengjie.org.cn/172.200.1.203 Start Time: Thu, 20 Feb 2020 03:37:13 +0800 Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-demo","namespace":"yinzhengjie-admission-control"},"spec":{"conta... kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container myapp; cpu limit for container myapp Status: Running IP: 10.244.3.4 IPs: IP: 10.244.3.4 Containers: myapp: Container ID: docker://503619b6cd5b1f6e7a0f0394d9710df9d6fe6621c16fdd06466520e3df2989e7 Image: ikubernetes/myapp:v1 Image ID: docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513 Port: <none> Host Port: <none> State: Running Started: Thu, 20 Feb 2020 03:37:22 +0800 Ready: True Restart Count: 0 Limits: cpu: 1 Requests: cpu: 1 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-r5wq6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-r5wq6: Type: Secret (a volume populated by a Secret) SecretName: default-token-r5wq6 Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 25s default-scheduler Successfully assigned yinzhengjie-admission-control/pod-demo to node203.yinzhengjie.org.cn Normal Pulling 24s kubelet, node203.yinzhengjie.org.cn Pulling image "ikubernetes/myapp:v1" Normal Pulled 16s kubelet, node203.yinzhengjie.org.cn Successfully pulled image "ikubernetes/myapp:v1" Normal Created 16s kubelet, node203.yinzhengjie.org.cn Created container myapp Normal Started 16s kubelet, node203.yinzhengjie.org.cn Started container myapp [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control NAME READY STATUS RESTARTS AGE pod-demo 1/1 Running 0 2m21s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl delete -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod "pod-demo" deleted [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control No resources found in yinzhengjie-admission-control namespace. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.使用自定义的LimitRange(如果指定的资源范围不在范围内则会无法创建Pod)
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml apiVersion: v1 kind: Pod metadata: name: pod-demo namespace: yinzhengjie-admission-control spec: containers: - image: ikubernetes/myapp:v1 imagePullPolicy: IfNotPresent name: myapp resources: requests: cpu: 500m limits: cpu: 1500m [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe limitrange cpu-limit-range -n yinzhengjie-admission-control Name: cpu-limit-range Namespace: yinzhengjie-admission-control Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 500m 2 1 1 4 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod/pod-demo created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control pod-demo -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-demo 1/1 Running 0 59s 10.244.3.5 node203.yinzhengjie.org.cn <none> <none> [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe pods -n yinzhengjie-admission-control pod-demo Name: pod-demo Namespace: yinzhengjie-admission-control Priority: 0 Node: node203.yinzhengjie.org.cn/172.200.1.203 Start Time: Thu, 20 Feb 2020 03:49:42 +0800 Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-demo","namespace":"yinzhengjie-admission-control"},"spec":{"conta... Status: Running IP: 10.244.3.5 IPs: IP: 10.244.3.5 Containers: myapp: Container ID: docker://a991046c9cff1008ae93458dfd90c729794ac7a794df9e9507c92b8aa84646c3 Image: ikubernetes/myapp:v1 Image ID: docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513 Port: <none> Host Port: <none> State: Running Started: Thu, 20 Feb 2020 03:49:43 +0800 Ready: True Restart Count: 0 Limits: cpu: 1500m Requests: cpu: 500m Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-r5wq6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-r5wq6: Type: Secret (a volume populated by a Secret) SecretName: default-token-r5wq6 Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 23s default-scheduler Successfully assigned yinzhengjie-admission-control/pod-demo to node203.yinzhengjie.org.cn Normal Pulled 22s kubelet, node203.yinzhengjie.org.cn Container image "ikubernetes/myapp:v1" already present on machine Normal Created 22s kubelet, node203.yinzhengjie.org.cn Created container myapp Normal Started 22s kubelet, node203.yinzhengjie.org.cn Started container myapp [root@master200.yinzhengjie.org.cn ~]#
三.ResourceQuota
由ResourceQuota对象定义的资源配额提供限制每个命名空间聚合资源消耗的约束。
它可以按类型限制可以在命名空间中创建的对象的数量，以及该命名空间中的资源可能消耗的计算资源和存储的总量。
LimitRange用于定义单个Pod对象上计算资源的requests及limits，而ResourceQuota则负责为整个namespace设定资源配额。
当特定命名空间中存在资源配额时，将在该命名空间中强制资源配额。
ResourceQuota对象的生效依赖于ResourceQuota Admission Controller.
1>.创建resourcequota资源
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml apiVersion: v1 kind: ResourceQuota metadata: name: quota-example namespace: yinzhengjie-admission-control spec: hard: pods: "5" requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi count/deployments.apps: "2" count/deployments.extensions: "2" persistentvolumeclaims: "2" [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml resourcequota/quota-example created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get resourcequota -n yinzhengjie-admission-control NAME CREATED AT quota-example 2020-02-19T20:38:34Z [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 1500m 2 limits.memory 0 2Gi persistentvolumeclaims 0 2 pods 1 5 requests.cpu 500m 1 requests.memory 0 1Gi [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.使用自定义的resourcequota的资源限制
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 1500m 2 limits.memory 0 2Gi persistentvolumeclaims 0 2 pods 1 5 requests.cpu 500m 1 requests.memory 0 1Gi [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml apiVersion: v1 kind: Pod metadata: name: mynginx namespace: yinzhengjie-admission-control labels: app: pod-demo rel: stable spec: containers: - name: mynginx image: nginx:1.14-alpine resources: requests: cpu: 500m memory: 500Mi limits: cpu: 500m memory: 1Gi [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml pod/mynginx created [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 2 2 limits.memory 1Gi 2Gi persistentvolumeclaims 0 2 pods 2 5 requests.cpu 1 1 requests.memory 500Mi 1Gi [root@master200.yinzhengjie.org.cn ~]#
四.PodSecurityPolicy(定义Pod的安全策略)
1>.定义受限制的pod的安全策略参考配置清单
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat psp-restricted.yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: 'MustRunAsNonRoot' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: - min: 1 max: 65535 readOnlyRootFilesystem: false [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
2>.定义有特权的pod的安全策略参考配置清单
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat psp-privileged.yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: privileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' spec: privileged: true allowPrivilegeEscalation: true allowedCapabilities: - '*' volumes: - '*' hostNetwork: true hostPorts: - min: 0 max: 65535 hostIPC: true hostPID: true runAsUser: rule: 'RunAsAny' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'RunAsAny' fsGroup: rule: 'RunAsAny' [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
3>.安全策略调用参考配置文件
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat clusterrole-with-psp.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: psp:restricted rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - restricted --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: psp:privileged rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - privileged [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat clusterrolebinding-with-psp.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: restricted-psp-user roleRef: kind: ClusterRole name: psp:restricted apiGroup: rbac.authorization.k8s.io subjects: - kind: Group apiGroup: rbac.authorization.k8s.io name: system:authenticated --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: privileged-psp-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: psp:privileged subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:masters - apiGroup: rbac.authorization.k8s.io kind: Group name: system:node - apiGroup: rbac.authorization.k8s.io kind: Group name: system:serviceaccounts:kube-system [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
五.准入控制器扩展
Admission Controllers代码必须要编译进kube-apiserver,且只能在程序启动时进行配置，于是，Kubernetes又特地引入了Admission Webhooks(beta in 1.9)和Initializers(alpha)来尝试突破此限制，以允许用户单独开发主奴人控制器并运行时进行配置。 允许webhook是接收请求并对其执行操作的HTTP回调。 您可以定义两种类型的许可Webhook，验证许可Webhook和变异许可Webhook。 通过验证许可webhook，您可以拒绝执行自定义许可策略的请求 使用变异的允许webhook，您可以更改请求以强制使用自定义默认值。