【Beats】Filebeat收集Nginx日志（十九）

　　本例介绍如何使用Filebeat收集Nginx日志，在【Beats】 Filebeat介绍及使用（十六）中，介绍了如何抓入日志，

　　前面要想实现日志数据的读取以及处理都是自己手动配置的，其实，在Filebeat中，有大量的Module，可以简化我 们的配置，直接就可以使用，如下:

Filebeat Nginx Module使用

　　Nginx日志格式如下：

　　日志格式地址：https://github.com/kubernetes/ingress-nginx/blob/nginx-0.28.0/docs/user-guide/nginx-configuration/log-format.md

log_format upstreaminfo
    '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
    '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr '
    '$upstream_response_length $upstream_response_time $upstream_status $req_id';

　　1、查看Filebeat支持模块

　　　　命令：./filebeat modules list

　　2、启动Nginx模块

　　　　启动命令：./filebeat modules enable nginx

　　　　禁用命令：./filebeat modules disable nginx

　　　　

　　3、查看modules.d目录中的文件，可以看到nginx.yml配置文件，证明nginx module模块已开启

　　　　命令：ls modules.d/

　　　　

　　4、配置modules.d/nginx.yml文件，指定access.log 和 error.log 的地址

　　　　命令：vim modules.d/nginx.yml

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/data/logs/nginx-1.16.1/access.log"]

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: true
    var.paths: ["/data/logs/nginx-1.16.1/error.log"]

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

 　　5、配置test-nginx.yml文件

# 自定义测试配置文件test-nginx.yml

# 输入
filebeat.inputs:

# 指定索引的分区数
setup.template.settings:
  index.number_of_shards: 3

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml 
  reload.enabled: false

# 输出到指定ES的配置
output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  username: "elastic"
  password: "123456"

 　　6、运行filebeat：

　　　　命令：./filebeat -e -c test-nginx.yml -d "publish"

　　7、查看ES数据如下：

{
  "_index": "filebeat-7.6.1-2020.06.21-000001",
  "_type": "_doc",
  "_id": "3gsc3XIBeBo_vUMFUe7F",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "H__D",
      "id": "9f14c4db-2f85-4740-8183-36f475ffdfed",
      "type": "filebeat",
      "ephemeral_id": "52fb4acc-a216-4d5e-9e66-aff57694c4c2",
      "version": "7.6.1"
    },
    "nginx": {
      "access": {
        "remote_ip_list": [
          "127.0.0.1"
        ]
      }
    },
    "log": {
      "file": {
        "path": "/data/logs/nginx-1.16.1/access.log"
      },
      "offset": 41957019
    },
    "source": {
      "geo": {
        "continent_name": "Asia",
        "region_iso_code": "CN-ZJ",
        "city_name": "Hangzhou",
        "country_iso_code": "CN",
        "region_name": "Zhejiang",
        "location": {
          "lon": 120.1619,
          "lat": 30.294
        }
      },
      "as": {
        "number": 37963,
        "organization": {
          "name": "Hangzhou Alibaba Advertising Co.,Ltd."
        }
      },
      "address": "127.0.0.1",
      "ip": "127.0.0.1"
    },
    "fileset": {
      "name": "access"
    },
    "url": {
      "original": "/.reporting-*/_search"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2020-06-22T17:38:37.000Z",
    "ecs": {
      "version": "1.4.0"
    },
    "service": {
      "type": "nginx"
    },
    "host": {
      "name": "H__D"
    },
    "http": {
      "request": {
        "referrer": "-",
        "method": "POST"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 159
        }
      },
      "version": "1.1"
    },
    "event": {
      "timezone": "+08:00",
      "created": "2020-06-22T17:38:39.155Z",
      "module": "nginx",
      "dataset": "nginx.access"
    },
    "user": {
      "name": "kibana"
    },
    "user_agent": {
      "original": "-",
      "name": "Other",
      "device": {
        "name": "Other"
      }
    }
  },
  "fields": {
    "event.created": [
      "2020-06-22T17:38:39.155Z"
    ],
    "suricata.eve.timestamp": [
      "2020-06-22T17:38:37.000Z"
    ],
    "@timestamp": [
      "2020-06-22T17:38:37.000Z"
    ]
  },
  "sort": [
    1592847517000
  ]
}

Filebeat 收集JSON格式数据

　　Nginx日志JSON格式如下：

log_format json '{"@timestamp":"$time_iso8601",'
             '"host":"$server_addr",'
             '"clientip":"$remote_addr",'
             '"size":$body_bytes_sent,'
             '"responsetime":$request_time,'
             '"upstreamtime":"$upstream_response_time",'
             '"upstreamhost":"$upstream_addr",'
             '"http_host":"$host",'
             '"url":"$uri",'
             '"referer":"$http_referer",'
             '"agent":"$http_user_agent",'
             '"status":"$status"}';

access_log  /data/logs/nginx-1.16.1/access.log    json;

　　1、配置test-nginx-json.yml文件

# 输入
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/logs/nginx-1.16.1/access.log
  # 默认情况下，解码后的JSON放置在输出文档中的“json”键下。
  # 如果启用此设置，则将密钥复制到输出文档的顶层。默认值为false。
  # 开启后，注意字段类型，要与原顶层字段类型一直
  json.keys_under_root: false

  # 如果keys_under_root和启用了此设置，则在发生冲突时，
  # 来自解码的JSON对象的值将覆盖Filebeat通常添加的字段（类型，源，偏移量等）。
  json.overwrite_keys: true

  # 如果启用了此设置，则在JSON解组错误或message_key在配置中定义
  # 但无法使用时，Filebeat将添加“ error.message”和“ error.type：json”键。
  json.add_error_key: true

  # 可选的配置设置，指定要在其上应用行过滤和多行设置的JSON密钥。
  json.message_key: clientip


# 指定索引的分区数
setup.template.settings:
  index.number_of_shards: 3

# 输出到指定ES的配置
output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  username: "elastic"
  password: "123456"

　　2、运行filebeat：

　　　　命令：./filebeat -e -c test-nginx-json.yml -d "publish"

　　3、查看ES数据如下：

{
  "_index": "filebeat-nginx3-2020.06.24-000001",
  "_type": "_doc",
  "_id": "5yvG5nIBk_UZVEKbwtBo",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-06-24T14:41:24.096Z",
    "log": {
      "offset": 101335182,
      "file": {
        "path": "/data/logs/nginx-1.16.1/access.log"
      }
    },
    "json": {
      "responsetime": 0.004,
      "upstreamhost": "127.0.0.1:19202",
      "status": "200",
      "referer": "-",
      "host": "127.0.0.1",
      "clientip": "127.0.0.1",
      "upstreamtime": "0.003",
      "@timestamp": "2020-06-24T22:41:24+08:00",
      "url": "/.reporting-*/_search",
      "agent": "-",
      "size": 159,
      "http_host": "127.0.0.1"
    },
    "input": {
      "type": "log"
    },
    "host": {
      "name": "H__D"
    },
    "agent": {
      "version": "7.6.1",
      "type": "filebeat",
      "ephemeral_id": "8bad05c6-a191-4550-a9c0-91e1f721748a",
      "hostname": "H__D",
      "id": "9f14c4db-2f85-4740-8183-36f475ffdfed"
    },
    "ecs": {
      "version": "1.4.0"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2020-06-24T14:41:24.096Z"
    ],
    "@timestamp": [
      "2020-06-24T14:41:24.096Z"
    ]
  },
  "sort": [
    1593009684096
  ]
}