admission webhook 初探（编译篇）

• 编译开源的 admission webhook
  • 编译示例 webhook
  • 制作镜像
    • Dockerfile
    • 制作命令
  • k8s 的配置
    • 启用 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook
  • 部署 webhook
    • prepare
    • 部署 webhook server
    • 部署 service
    • 在 k8s 中使能 webhook
      • 配置 ValidatingWebhookConfiguration
      • 配置 MutatingWebhookConfiguration
  • 遇到的问题
  • 参考文档

编译开源的 admission webhook

本文参考的代码 admission-webhook-example，k8s-admission-webhook 文档

admission webhook 的作用我简单的总结下，当用户的请求到达 k8s apiserver 后，apiserver 根据 MutatingWebhookConfiguration 和 ValidatingWebhookConfiguration 的配置，先调用 MutatingWebhookConfiguration 去修改用户请求的配置文件，最后会调用 ValidatingWebhookConfiguration 来验证这个修改后的配置文件是否合法。

我们可以利用 mutating 的机制，将一些特殊的配置自动加上，而不用用户来操心。同时也可以在 validating 中编写代码设置自己的规则，看请求是否合法。

因为已经研究过怎么编写 webhook 代码/怎么部署 webhook，要使用好 webhook 需要注意几个点：

• webhook 的本质是个 http server，因此，需要用代码实现这么一个 server，供 apiserver 调用
• mutating webhook 和 validating webhook 分别实现了不同的功能，这两者没有直接联系，本质上说，如果修改/验证都需要，我们需要编写两个 http server
• 编写好的 webhook，建议使用 k8s deployment 部署，同时使用 service 来暴露能力
• apiserver 是否调用 webhook，是由 k8s MutatingWebhookConfiguration/ValidatingWebhookConfiguration 来定义的
• 需要注意 apiserver 和 webhook 通信时的身份认证和权限控制问题

编译示例 webhook

https://github.com/cnych/admission-webhook-example 项目中将 mutating、validating 的功能都放到一个 http server 中了，这当然也是可以的。但是 k8s MutatingWebhookConfiguration/ValidatingWebhookConfiguration 肯定是需要都部署的。

[root@localhost lihao04]# cd /root/lihao04
[root@localhost lihao04]# git clone https://github.com/cnych/admission-webhook-example.git
[root@localhost lihao04]# cd admission-webhook-example/
[root@localhost github.com]# export GOPATH=/root/lihao04/go/
[root@localhost github.com]# mkdir -p /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# ln -s /root/lihao04/admission-webhook-example /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# export http_proxy=http://jarvis:lihao-root@10.191.67.135:3128/
[root@localhost github.com]# export https_proxy=http://jarvis:lihao-root@10.191.67.135:3128/
[root@localhost github.com]# cd /root/lihao04/go/src/github.com/cnych/admission-webhook-example
[root@localhost admission-webhook-example]# dep ensure -v
...

[root@localhost admission-webhook-example]# export GO111MODULE=on
[root@localhost admission-webhook-example]# CGO_ENABLED=0 GOOS=linux go build -mod=vendor -a -installsuffix cgo -o admission-webhook-example
go: creating new go.mod: module github.com/cnych/admission-webhook-example
go: copying requirements from Gopkg.lock
go: converting Gopkg.lock: stat github.com/json-iterator/go@f2b4162afba35581b6d4a50d3b8f34e33c144682: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/text@v0.3.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/google/gofuzz@24818f796faf91cd76ec7bddd72458fbced7a6c1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/api@12444147eb1150aa5c80d2aae532cbc5b7be73d0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/opencontainers/go-digest@v1.0.0-rc1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/golang/glog@23def4e6c14b4da8ac2ed8007337bc5eb5007998: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/reflect2@1df9eeb2bb81f327b96228865c5687bc2194af3f: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apimachinery@e386b2658ed20923da8cc9250e552f082899a1ee: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiserver@88d4601515c27f180f7efc8705e4cc18dc19100d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/net@892bf7b0c6e2f93b51166bf3882e50277fa5afc6: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/yaml.v2@v2.1.1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/gogo/protobuf@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/spf13/pflag@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/docker/distribution@34c706e759240975178df82495f147559cc0edc1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/concurrent@bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiextensions-apiserver@f584b16eb23bd2a3fd292a027d698d95db427c5d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/ghodss/yaml@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/kubernetes@efe960cdc41ee7b18d408128dfb80babb5bc746a: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/inf.v0@v0.9.0: repo version lookup disabled by -mod=vendor

制作镜像

将编译后的代码打包进 image，这样就可以使用 k8s deployment 来部署

Dockerfile

注意修改 FROM 的

FROM alpine:3.10.3

ADD admission-webhook-example /admission-webhook-example

ENTRYPOINT ["./admission-webhook-example"]

制作命令

[root@localhost admission-webhook-example]# docker build --no-cache -t docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v1 .
Sending build context to Docker daemon     43MB
Step 1/3 : FROM docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
3.10.3: Pulling from jarvis-image/alpine
89d9c30c1d48: Already exists
Digest: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a
Status: Downloaded newer image for docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
 ---> 965ea09ff2eb
Step 2/3 : ADD admission-webhook-example /admission-webhook-example
 ---> 2c65bd92673c
Step 3/3 : ENTRYPOINT ["./admission-webhook-example"]
 ---> Running in 41ac87194f86
Removing intermediate container 41ac87194f86
 ---> 7c57f014ab9a
Successfully built 7c57f014ab9a
Successfully tagged docker-registry.lihao04.virtual/admission-webhook-example:v1

k8s 的配置

启用 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook

MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook 默认是不启用的，apiserver 想调用 webhook，还得 enable 相关的能力

[root@master ~]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
  enableServiceLinks: true

因为 enable-admission-plugins 缺失 feature，我们要 enable

# 修改 /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

修改配置文件后立刻生效

[root@master manifests]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
  enableServiceLinks: true

# 确实重启过
[root@master manifests]# kubectl get pod -n kube-system|grep api
kube-apiserver-master.lihao04.virtual            1/1     Running            0          54s

部署 webhook

prepare

这个 server 是供 apiserver 调用的，它的运行代码正是我们编译的 admission-webhook-example，他使用的镜像正是我们构建的 docker-registry.lihao04.virtual/admission-webhook-example:v1

部署前，需要先创建 CertificateSigningRequest、secret 等，这个是给 apiserver 与 webhook 通信做身份认证的

[root@master lihao04]# cd /root/lihao04/admission-webhook-example/deployment
[root@master lihao04]# ./webhook-create-signed-cert.sh

部署 webhook server

[root@master deployment]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook-example-deployment
  labels:
    app: admission-webhook-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-webhook-example
  template:
    metadata:
      labels:
        app: admission-webhook-example
    spec:
      containers:
        - name: admission-webhook-example
          image: docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v2
          #image: cnych/admission-webhook-example:v1
          args:
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=4
            - 2>&1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            secretName: admission-webhook-example-certs

[root@master deployment]# kubectl apply -f deployment.yaml
deployment.apps/admission-webhook-example-deployment create

部署 service

让该 deployment(webhook server) 能被 apiserver 访问，必须套一个 service，注意，访问 service 的 443 会访问 webhook deployment 中的 pod 的 443 端口，而 pod 中的服务正是启动在 443 端口上，这个大家可以看 admission-webhook-example 源码。

[root@master deployment]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: admission-webhook-example-svc
  labels:
    app: admission-webhook-example
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: admission-webhook-example

[root@master deployment]#  kubectl create -f service.yaml

在 k8s 中使能 webhook

当 webhook server 部署好了之后，apiserver 如何知道它要访问 webhook server？

因此需要配置 k8s 集群，让 webhook 被 k8s 感知到，并能在适当的时候调用到。

因为 webhook 有两个部分的功能

• 修改
• 验证

修改在先，验证在后，因此这两个功能都需要独立配置；

配置 ValidatingWebhookConfiguration

配置名为：ValidatingWebhookConfiguration

[root@master deployment]# cat validatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: required-labels.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/validate"
      # 通过 kubectl config view --raw --flatten -o json|grep certificate-authority-data 获得
	  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
	# 对 namespace 中打了 admission-webhook-example = enabled 的进行验证，其他 namespace 不验证
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

[root@master deployment]# kubectl apply -f validatingwebhook-ca-bundle.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook-example-cfg created

测试

[root@master deployment]# cat sleep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: tutum/curl
        command: ["/bin/sleep","infinity"]
        imagePullPolicy: IfNotPresent

[root@master deployment]# kubectl apply -f sleep.yaml
Error from server (required labels are not set): error when creating "sleep.yaml": admission webhook "required-labels.qikqiak.com" denied the request: required labels are not set

可以发现，如果没有带特定的内容，会报错，达到了验证的效果

配置 MutatingWebhookConfiguration

配置名为：MutatingWebhookConfiguration

[root@master deployment]# cat mutatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: mutating-example.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/mutate"
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

[root@master deployment]# kubectl apply -f mutatingwebhook-ca-bundle.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-example-cfg created

测试

同样的 sleep 任务

[root@master deployment]# kubectl apply -f sleep.yaml
deployment.apps/sleep created

可以创建出来了，并且可以看到，增加了很多附加的 label，因为信息被添加，所以 validation 也通过了

[root@master deployment]# kubectl describe deployment sleep
Name:                   sleep
Namespace:              default
CreationTimestamp:      Fri, 25 Oct 2019 19:17:51 +0800
Labels:                 app.kubernetes.io/component=not_available
                        app.kubernetes.io/instance=not_available
                        app.kubernetes.io/managed-by=not_available
                        app.kubernetes.io/name=not_available
                        app.kubernetes.io/part-of=not_available
                        app.kubernetes.io/version=not_available
Annotations:            admission-webhook-example.qikqiak.com/status: mutated
                        deployment.kubernetes.io/revision: 1
Selector:               app=sleep
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=sleep
  Containers:
   sleep:
    Image:      tutum/curl
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sleep
      infinity
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sleep-674f75ff4d (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  57s   deployment-controller  Scaled up replica set sleep-674f75ff4d to 3

遇到的问题

1. webhook server 验证不起作用，发现是 bad certificate

[root@master deployment]# kubectl log admission-webhook-example-deployment-5594df959f-6rjql
log is DEPRECATED and will be removed in a future version. Use logs instead.
I1025 09:32:21.348823       1 main.go:50] Server started
2019/10/25 09:33:01 http: TLS handshake error from 10.244.0.0:10131: remote error: tls: bad certificate

发现，参考文档中，下面步骤有问题

cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml

参考文档

• https://www.qikqiak.com/post/k8s-admission-webhook/ (中文)
• https://banzaicloud.com/blog/k8s-admission-webhooks/ (英文)