k8s filebeat sidecar模式收集ingress nginx日志并可视化展示

本文默认k8s环境以及已经部署ingress controller

公司所用ingress监控是由prometheus+grafana进行，但是监控不够全面，故使用filebeat去采集ingress日志，并自主进行可视化展示

1、ingress nginx日志数据落盘

在ingress controller中将configmap改为

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-controller
data:
  access-log-path: /var/log/nginx/access.log
  compute-full-forwarded-for: 'true'
  enable-vts-status: 'true'
  error-log-path: /var/log/nginx/error.log
  forwarded-for-header: X-Forwarded-For
  log-format-upstream: >-
    {"@timestamp":
    "$time_iso8601","remote_addr":"$remote_addr","x-forward-for":"$http_x_forwarded_for","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":$status,"vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":$upstream_status}
  use-forwarded-headers: 'true'

2、生成filebeat镜像

新建目录，目录如下

dockerfile

FROM  million12/centos-supervisor:4.0.2
WORKDIR /usr/local
ADD filebeat-7.5.0-linux-x86_64.tar.gz .
RUN ln -s filebeat-7.5.0-linux-x86_64  filebeat 
 && cd filebeat       
 && mkdir  config     
 && chmod +x filebeat 
 && cp filebeat.yml config/  
 && yum -y install logrotate crontabs


COPY supervisord.conf /etc/supervisord.conf

RUN mkdir -p /var/log/supervisor
EXPOSE 22 80
CMD ["/usr/bin/supervisord"]

因为需要使用logrotate进行日志轮转，需要安装

logrotate crontabs

supervisord.conf配置如下

[supervisord]
nodaemon=true

[program:cron]
command=/usr/sbin/crond -i

[program:filebeat]
command=/usr/local/filebeat/filebeat -c /usr/local/filebeat/config/filebeat.yml

3、修改原有ingress controller depl，将filebeat与ingress controller放到同一pod中，使用emptydir卷共享ingress日志，使filebeat能够读取，另外一个是面对日志的持续正常如何处理，这里使用logrotate，将logrotate在filebeat中配置，尽量对ingress影响小点，首先增加filebeat configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-config
data:
  filebeat.yml: |
    filebeat.inputs:
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/access.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["access"]
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/error.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["error"]
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    setup.template.settings:
      index.number_of_shards: 3
    output.elasticsearch:
      hosts: ["es-local.nxgp.svc.cluster.local:9200"]
      index: "nginx_log-%{+yyyy.MM.dd}"
      indices:
        - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "access"
        - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "error"
    setup.template.name: "nginx_log"
    setup.template.pattern: "nginx_*"
    setup.template.enabled: true
    setup.ilm.enabled: false
    setup.template.overwrite: false

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-logrotate
data:
  nginx: |
    /var/log/nginx/*.log {
      su root root
      size 50M
      notifempty
      copytruncate
      rotate 3
      missingok
      compress
      dateext
      dateformat .%Y%m%d-%H
  }

然后进行depl更新，只展示新增部分

      volumes:
        - name: ingress-log
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
            defaultMode: 420
        - name: logrotateconf
          configMap:
            name: nginx-ingress-logrotate
            items:
              - key: nginx
                path: nginx
            defaultMode: 420
      containers:
        - name: controller
          volumeMounts:
            - name: ingress-log
              mountPath: /var/log/nginx/
        - name: filebeat
          image: 'xxx/filebeat:7.5.0'
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: '1'
              memory: 1Gi
          volumeMounts:
            - name: filebeat-config
              mountPath: /usr/local/filebeat/config/
            - name: ingress-log
              mountPath: /var/log/nginx/
            - name: logrotateconf
              mountPath: /etc/logrotate.d/nginx
              subPath: nginx
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      serviceAccount: ingress-nginx
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

logrotate是按天更新，更新时间不定

轮转效果内存占用高的是还没有进行打包的







4、es可视化展示

(1)PV

(2)UV

(3)Top10(接口访问量)

(4)Top10(客户端IP访问占比)

(5)Top10(最慢接口)

(6)后端upstream占比

(7)实时流量

(8)客户端访问占比

(9)平均并发数

(10)异常状态码统计

 

(11)总流量

(12)接口异常响应码

(13)接口访问耗时占比

(14)每10秒接口访问平均耗时

(15)每10秒接口访问最大耗时

(16)状态码统计

(17)访问量趋势图

(18)超过30秒以上的接口

(19)超过30秒以上的接口出现次数