【kserve】流量是如何一步一步的转发到kserve具体的容器接口的

背景

kserve在kubeflow0.6之后独立出来，在kfserving的基础上做了大量的重构。其重度依赖knative和istio。本文记录kserve启动后，流量如何一步一步通过istio转发给最终的serving的。

开始

首先启动一个pmml服务，为了方便理解，该pmml同时启动两个版本，并且灰度:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    predict-phantom.wenyangchou.com/group: ""
    predict-phantom.wenyangchou.com/message: ""
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  labels:
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  name: test-pmml
  namespace: phantom
spec:
  predictor:
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      device: cpu
    pmml:
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: "2"
          memory: 4Gi
      storageUri: http://dev-predict.wenyangchou.com/predict-fs/v2/file/download/predict-phantom/test-pmml/v20211225.1/model.pmml

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    predict-phantom.wenyangchou.com/group: ""
    predict-phantom.wenyangchou.com/message: ""
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  labels:
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  name: test-pmml
  namespace: phantom
spec:
  predictor:
    canaryTrafficPercent: 1
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      device: cpu
    pmml:
      resources:
        limits:
          cpu: "2"
          memory: 2Gi
        requests:
          cpu: "2"
          memory: 2Gi
      storageUri: http://dev-predict.wenyangchou.com/predict-fs/v2/file/download/predict-phantom/test-pmml/v20211225.1/model.pmml

两个版本除了资源不一样，其他都一样。启动之后如下

test-pmml         http://test-pmml.phantom.dev-predict-phantom.wenyangchou.com         True    99     1        test-pmml-predictor-default-00001   test-pmml-predictor-default-00002         3d

流量进入

此时有个http请求: http://test-pmml.phantom.wenyangchou.com/v1/models/test-pmml:predict

• 请求进入istio的service istio-system/istio-ingressgateway
• 此时流量第一步匹配gateway knative-serving/knative-ingress-gateway

MacBook-Pro:~ zhouwenyang$ kc get gateway -A
NAMESPACE         NAME                      AGE
knative-serving   knative-ingress-gateway   41d
knative-serving   knative-local-gateway     41d

• virtualservice phantom/test-pmml 进行第一步路由，该virtualservice用于路由到具体的算法服务

kc get virtualservice test-pmml -n phantom -oyaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    predict-phantom.wenyangchou.com/group: ""
    predict-phantom.wenyangchou.com/message: ""
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  labels:
    predict-phantom.wenyangchou.com/modelName: test-pmml
    predict-phantom.wenyangchou.com/modelVersion: v20211225.1
  name: test-pmml
  namespace: phantom
spec:
  gateways:
  - knative-serving/knative-local-gateway
  - knative-serving/knative-ingress-gateway
  hosts:
  - test-pmml.phantom.svc.cluster.local
  - test-pmml.phantom.dev-predict-phantom.wenyangchou.com
  http:
  - headers:
      request:
        set:
          Host: test-pmml-predictor-default.phantom.svc.cluster.local
    match:
    - authority:
        regex: ^test-pmml\.phantom(\.svc(\.cluster\.local)?)?(?::\d{1,5})?$
      gateways:
      - knative-serving/knative-local-gateway
    - authority:
        regex: ^test-pmml\.phantom\.dev-predict-phantom\.wenyangchou\.com(?::\d{1,5})?$
      gateways:
      - knative-serving/knative-ingress-gateway
    route:
    - destination:
        host: knative-local-gateway.istio-system.svc.cluster.local
        port:
          number: 80
      weight: 100

此时可以看到，虽然目前是灰度，但是这里weight还是100，所以灰度不是在这里控制的。这里观察destination

MacBook-Pro:~ zhouwenyang$ kc get svc -A

NAMESPACE              NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP                                            PORT(S)                                      AGE
istio-system           istio-ingressgateway                              NodePort       10.10.102.198   <none>                                                 15021:31507/TCP,80:30000/TCP,443:31381/TCP   41d
istio-system           istiod                                            ClusterIP      10.10.107.204   <none>                                                 15010/TCP,15012/TCP,443/TCP,15014/TCP        41d
istio-system           knative-local-gateway                             ClusterIP      10.10.194.194   <none>                                                 80/TCP                                       41d
phantom                test-pmml                                         ExternalName   <none>          knative-local-gateway.istio-system.svc.cluster.local   <none>                                       3d
phantom                test-pmml-predictor-default                       ExternalName   <none>          knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                       3d
phantom                test-pmml-predictor-default-00001                 ClusterIP      10.10.176.106   <none>                                                 80/TCP                                       3d
phantom                test-pmml-predictor-default-00001-private         ClusterIP      10.10.151.3     <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   3d
phantom                test-pmml-predictor-default-00002                 ClusterIP      10.10.115.151   <none>                                                 80/TCP                                       2d23h
phantom                test-pmml-predictor-default-00002-private         ClusterIP      10.10.163.195   <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d23h

desctination的规则是 {name}.{namespace}.{host}，优先匹配{name}.{namespace}，在匹配不到的情况下也会去匹配EXTERNAL-IP。因此这里实际匹配到的svc是 istio-system/knative-local-gateway。

根据上面的virtualservice内容信息，流量进入下层的时，请求域名变为 test-pmml-predictor-default.phantom.svc.cluster.local

• 流量第二步开始匹配gateway knative-serving/knative-local-gateway
• virtualservice进行第二步路由，路由到真正的svc

查看virtualservice:test-pmml-predictor-default-ingress

MacBook-Pro:~ zhouwenyang$ kc get virtualservice test-pmml-predictor-default-ingress -n phantom -oyaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"test-pmml-predictor-default","percent":1,"revisions":[{"revisionName":"test-pmml-predictor-default-00002","percent":1}],"stepParams":{}}]}'
    networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
    serving.knative.dev/creator: system:serviceaccount:kserve:default
    serving.knative.dev/lastModifier: system:serviceaccount:kserve:default
  creationTimestamp: "2021-12-25T02:19:58Z"
  generation: 3
  labels:
    networking.internal.knative.dev/ingress: test-pmml-predictor-default
    serving.knative.dev/route: test-pmml-predictor-default
    serving.knative.dev/routeNamespace: phantom
  name: test-pmml-predictor-default-ingress
  namespace: phantom
  ownerReferences:
  - apiVersion: networking.internal.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: test-pmml-predictor-default
    uid: a6ffa63a-d0c9-40a1-8e6e-19a9de464264
  resourceVersion: "99727359"
  uid: 303d12f7-a1d7-439b-aeb8-bd0a4b6478d6
spec:
  gateways:
  - knative-serving/knative-ingress-gateway
  - knative-serving/knative-local-gateway
  hosts:
  - prev-test-pmml-predictor-default.phantom
  - prev-test-pmml-predictor-default.phantom.dev-predict-phantom.wenyangchou.com
  - prev-test-pmml-predictor-default.phantom.svc
  - prev-test-pmml-predictor-default.phantom.svc.cluster.local
  - test-pmml-predictor-default.phantom
  - test-pmml-predictor-default.phantom.dev-predict-phantom.wenyangchou.com
  - test-pmml-predictor-default.phantom.svc
  - test-pmml-predictor-default.phantom.svc.cluster.local
  http:
  - headers:
      request:
        set:
          K-Network-Hash: 259ea7ed1c618d64a3c0c2f764261329c99609a53e020219c71d07c8d0c74edd
    match:
    - authority:
        prefix: test-pmml-predictor-default.phantom
      gateways:
      - knative-serving/knative-local-gateway
      headers:
        K-Network-Hash:
          exact: override
    retries: {}
    route:
    - destination:
        host: test-pmml-predictor-default-00002.phantom.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Knative-Serving-Namespace: phantom
            Knative-Serving-Revision: test-pmml-predictor-default-00002
      weight: 1
    - destination:
        host: test-pmml-predictor-default-00001.phantom.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Knative-Serving-Namespace: phantom
            Knative-Serving-Revision: test-pmml-predictor-default-00001
      weight: 99
  - match:
    - authority:
        prefix: test-pmml-predictor-default.phantom
      gateways:
      - knative-serving/knative-local-gateway
    retries: {}
    route:
    - destination:
        host: test-pmml-predictor-default-00002.phantom.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Knative-Serving-Namespace: phantom
            Knative-Serving-Revision: test-pmml-predictor-default-00002
      weight: 1
    - destination:
        host: test-pmml-predictor-default-00001.phantom.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Knative-Serving-Namespace: phantom
            Knative-Serving-Revision: test-pmml-predictor-default-00001
      weight: 99

这边可以看到，test-pmml-predictor-default.phantom的域名生效，转发给具体的svc，并且做了流量权重

流程图如下: