filebeat收集nginx的json格式日志

一、在nginx主机上安装filebeat组件

[root@zabbix_server nginx]# cd /usr/local/src/
[root@zabbix_server src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpm
[root@zabbix_server src]# yum localinstall filebeat-6.2.4-x86_64.rpm 

二、配置filebeat配置文件

[root@zabbix_server src]# vim /etc/filebeat/filebeat.yml 
filebeat.prospectors:
  - type: log
    enabled: true
    json.keys_under_root: true    #json格式收集
    json.overwrite_keys: true       #json格式收集
    paths:
      - /var/log/nginx/access.log  #需要收集的日志文件路径
    fields:
      log_topics: nginx-172.28.18.75 #设置日志标题 
  
  output.logstash:
    hosts: ["172.28.18.69:10001"]  #输出到logstash服务地址和端口

三、配置nginx.conf文件，设置json格式日志

[root@zabbix_server src]# vim /etc/nginx/nginx.conf 
    log_format access_json_log  '{"@timestamp":"$time_local",'
                                  '"http_host":"$http_host",'
                                  '"clinetip":"$remote_addr",'
                                  '"request":"$request",'
                                  '"status":"$status",'
                                  '"size":"$body_bytes_sent",'
                                  '"upstream_addr":"$upstream_addr",'
                                  '"upstream_status":"$upstream_status",'
                                  '"upstream_response_time":"$upstream_response_time",'
                                  '"request_time":"$request_time",'
                                  '"http_referer":"$http_referer",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"http_x_forwarded_for":"$http_x_forwarded_for"}';
                                      
    access_log  /var/log/nginx/access.log  access_json_log;

四、检查配置文件语法，并重载

[root@zabbix_server src]# nginx -tc /etc/nginx/nginx.conf 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@zabbix_server src]# nginx -s reload

五、查看access.log日志输出

[root@zabbix_server src]# tail -f /var/log/nginx/access.log
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=311c76f6740bc2f7&action=widget.problems.view HTTP/1.1","status":"200","size":"33684","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"1.103","request_time":"1.103","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0","http_x_forwarded_for":"-"}
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=9203532d28b920b3&action=widget.problems.view HTTP/1.1","status":"200","size":"33856","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"0.759","request_time":"0.759","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","http_x_forwarded_for":"-"}

已经变成json格式的日志输出了。

六、在logstash服务器上的/etc/logstash/conf.d/目录下，新建一个nginx.conf文件

用于监听10001端口，负责收集filebeat传递过来的日志数据，并向elasticseatch服务发送日志数据

[root@server-1 conf.d]# vim /etc/logstash/conf.d/nginx.conf
input {
  beats {
   port=>10001   #监听filebeat发送日志的端口
  }
}

output {
  if[fields][log_topics]=="nginx-172.28.18.75"{   #判断是否是filebeat配置设置的fields字段，是则发送到elastcisearch
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
     index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}"   #设置索引名字
    } 
  }
}

七、重启logstash服务，并查看端口

[root@server-1 conf.d]# systemctl restart logstash
[root@server-1 conf.d]# netstat -tunlp|grep 10001
tcp6       0      0 :::10001                :::*                    LISTEN      26599/java   

启动成功

八、重启nginx主机上的filebeat服务

[root@zabbix_server src]# systemctl restart filebeat

九、查看elasticsearch服务器上是否有新建的索引

[root@zabbix_server etc]# curl 172.28.18.69:9200/_cat/indices?v
health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                       nQhtAX5YSSKzT1p0G-1XtA   1   0          4            0     25.2kb         25.2kb
yellow open   nginx-172.28.5.214-2019.07.15 tH8oVfZhQy-TdnIPHyFgvQ   5   1    2106272            0    668.8mb        668.8mb
yellow open   system-syslog-2019.07         REp7fM_gSaquo9PX2_sREQ   5   1   10772928            0      2.6gb          2.6gb

可以看到日志已经成功收集到elasticsearch服务器上了

十、配置kibana，展示日志数据

系统管理---索引模式--创建索引模式

发现菜单里已经可以看到数据了