Envoy学习笔记

一、基本概念

  XDS、CDS、EDS、LDS、SDS、ADS、RDS、HDS

二、安装

三、动态文件配置

还没找到好的控制平面，为了解决动态更新CDS、LDS配置所以就采用了动态文件配置。

admin:
  access_log_path: /home/logs/envoy/admin.access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9001}
node:
  cluster: test-cluster
  id: test-id
dynamic_resources:
  cds_config:
    path: /home/envoy/cds.yaml
  lds_config:
    path: /home/envoy/lds.yaml

上面的配置文件定义动态资源的位置。cds_config 指定 cds配置文件存放位置。lds_config 指定lds配置文件存放位置。

当我们修改了其中某个配置项后可以通过 mv 命令 使envoy进行热更新 。例如： mv cds.yaml cds.yaml1 然后再mv cds.yaml1 cds.yaml。envoy 提供热启动器，但是还是建议使用mv 文件的方式来更新文件，热启动器是完全重新加载配置项，一旦某项配置不小心配置错误会导致整个监听失败。使用mv 热更新的好处是 当envoy检测到有错误配置项时它不会更新配置，这就保证了envoy不会因错误的配置导致整个监听不可用。

四、CDS

cds.yaml 文件中定义一组或多组Cluster。

resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  connect_timeout: 1s
  name: k8s.proxy       
  type: STATIC
  http2_protocol_options: {}
  load_assignment:
    cluster_name: k8s.proxy
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: xxx.xxx.xxx.xxx
              port_value: 31080
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  connect_timeout: 1s
  name: k8shttp1.proxy
  type: STATIC
  load_assignment:
    cluster_name: k8shttp1.proxy
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: xxx.xxx.xxx.xx
              port_value: 31080
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  connect_timeout: 1s
  name: ActivityService  
  type: STATIC
  circuit_breakers:
    thresholds:
    - priority: "DEFAULT"
      max_connections: 100000
      max_pending_requests: 100000
      max_requests: 100000
    - priority: "HIGH"
      max_connections: 100000
      max_pending_requests: 100000
      max_requests: 100000
  health_checks:
    timeout: 1s
    interval: 2s
    unhealthy_threshold: 1
    healthy_threshold: 1
    grpc_health_check: {}
  http2_protocol_options: {}
  load_assignment:
    cluster_name: ActivityService
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: xxx.xxx.xxx.xxx
              port_value: 10099
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  connect_timeout: 1s
  name: bigdata
  type: STRICT_DNS
  circuit_breakers:
    thresholds:
    - priority: "DEFAULT"
      max_connections: 100000
      max_pending_requests: 100000
      max_requests: 100000
    - priority: "HIGH"
      max_connections: 100000
      max_pending_requests: 100000
      max_requests: 100000
  health_checks:
    timeout: 1s
    interval: 2s
    unhealthy_threshold: 1
    healthy_threshold: 1
    grpc_health_check: {}
  http2_protocol_options: {}
  load_assignment:
    cluster_name: bigdata
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address:a.service.consul
              port_value: 10030

上面的配置中定义了两种不同的cluster，一种是STATIC 表明这种Cluster是指定静态的IP表示上游服务，一种是STRICT_DNS表明这种Cluster通过域名解析出具体的endpoint。

envoy 启动后可以通过curl localhost:9001/clusters 来查看指定的域名有几个Endpoint。

配置中还定义了健康检查、熔断。也可以通过 curl localhost:9001/clusters |grep xxx 来查看EndPoint是否健康，对于不健康的服务envoy不会将流量转发过去。健康检查只有在第一次访问该cluster才开始运行，如果没有流量则不开启健康检查。

五、LDS

lds.yaml 文件中定义了一组或多组监听配置项。

resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: k8shttps.proxy
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 443
  filter_chains:
  - transport_socket:
      name: "a.com"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
        common_tls_context:
          tls_certificates:
            certificate_chain:
              filename: /home/envoy/x.com-crt.pem
            private_key:
              filename: /home/envoy/x.com-key.pem
    filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /home/logs/envoy/k8s.https.access.log
          http_filters:
          - name: envoy.filters.http.router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: 
              - a.com
              - a.com:*
              routes:
              - match:
                  prefix: "/com.activity.server"
                route:
                  cluster: ActivityService
              - match:
                  prefix: "/AppGL"
                route:
                  cluster: k8sqahttp1.proxy
                  upgrade_configs:
                  - upgrade_type: "websocket"
                    enabled: true
              - match:
                  prefix: "/"
                route:
                  cluster: abc
                  timeout: 1200s
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                  headers:
                  - name: ":method"
                    exact_match: "HEAD"
                direct_response:
                  status: 200
                  body:
                    inline_string: "heihei"
              - match:
                  prefix: "/abc."
                  headers: 
                    name: abc
                    exact_match: abc1
                  case_sensitive: false
                route:
                  cluster: abc1
              - match:
                  prefix: "/abc."
                  case_sensitive: false
                  headers:
                    name: abc
                    exact_match: abc2
                route:
                  cluster: abc2
              - match:
                  prefix: "/hostrewrite"
                  case_sensitive: false
                route:
                  cluster: pcwang
                  host_rewrite_literal: wangpengchong.com
              - match:
                  prefix: "/AppGL"
                route:
                  cluster: k8shttp1.proxy
                  upgrade_configs:
                  - upgrade_type: "websocket"
                    enabled: true
              - match:
                  prefix: "/"
                route:
                  cluster: k8s.proxy
                  timeout: 1200s
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: k8s.proxy
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
  filter_chains:
  - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /home/logs/envoy/k8s.access.log
          http_filters:
          - name: envoy.filters.http.router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: 
              - b.com
              - b.com:*
              routes:
              - match:
                  prefix: "/AppGL"
                route:
                  cluster: k8sqahttp1.proxy
                  upgrade_configs:
                  - upgrade_type: "websocket"
                    enabled: true
              - match:
                  prefix: "/"
                route:
                  cluster: k8sqa.proxy
                  timeout: 1200s
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                  headers:
                  - name: ":method"
                    exact_match: "HEAD"
                direct_response:
                  status: 200
                  body:
                    inline_string: "heihei"
              - match:
                  prefix: "/"
                route:
                  cluster: k8s.proxy
                  timeout: 1200s

 上面的配置中定义了两组监听分别是443、80 ，443端口有配置https证书，提供https服务。

配置文件中还定义了预发布机制，根据特定域名，将该域名的所有请求都转发到预发布的Cluster中去。该机制还可以用作其他用途，可根据实际情况进行操作

还配置了较为复杂的流量转发，根据请求头中的值将流量转发到不同的Cluster中。

还配置了websocket升级机制，可将http1.1 升级为websocket，这需要客户端发送正确的升级请求。

还配置了host重写机制、超时，特定请求（健康检查）返回固定值，有需要的童鞋可以参考。

六、TCP监听

- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: AlertService
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 20010
  filter_chains:
  - filters:
      name: envoy.filters.network.tcp
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        stat_prefix: ingress_tcp
        max_connect_attempts: 5
        cluster: AlertService

七、jwk认证鉴权

- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: k8s.proxy
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10021
  filter_chains:
  - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /home/logs/envoy/beta.grpc.access.log
          http_filters:
          - name: envoy.filters.http.jwt_authn
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
              providers:
                identityserver:
                  issuer: http://xxxx
                  audiences:
                  - grpc1
                  remote_jwks:
                    http_uri:
                      uri: http://xxxx/.well-known/openid-configuration/jwks
                      cluster: identityserver
                      timeout: 5s
                    cache_duration: 600s
              rules:
              - match:
                  prefix: /
                requires:
                  provider_name: identityserver

这里展示较为简单的配置，更复杂的请参考官方文档：

https://www.envoyproxy.io/docs/envoy/v1.18.3/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#extension-envoy-filters-http-jwt-authn

八、监控

直接使用 prometheus 收集信息通grafana 展示即可