使用filebeat收集日志

目录

• 一、初始化环境
  • 1.1 环境准备
  • 1.2 安装es
  • 1.3 安装Kibana
• 二、收集nginx日志
• 三、收集tomcat日志
• 四、收集ES日志
• 五、收集docker容器日志
• 六、使用filebeat自带模块进行监控
• 七、Kibana的x-pack监控

一、初始化环境

1.1 环境准备

系统版本	主机名	IP地址	服务
Centos 7.5	node	192.168.1.1	es、kibana
Centos 7.5	test	192.168.1.2	filebeat

1.2 安装es

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
$ yum -y install elasticsearch-6.6.0.rpm
$ egrep -v '#|^$' /etc/elasticsearch/elasticsearch.yml 
node.name: node
path.data: /elk/data
path.logs: /elk/log
network.host: 192.168.1.1
http.port: 9200
$ mkdir -p /elk/{data,log}
$ chown elasticsearch.elasticsearch /elk -R
$ systemctl start elasticsearch
$ ss -lnt | grep 9200
LISTEN     0      128     ::ffff:192.168.1.1:9200                    :::*    

1.3 安装Kibana

$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.0-x86_64.rpm
$ yum -y install kibana-6.6.0-x86_64.rpm
$ egrep -v '#|^$' /etc/kibana/kibana.yml 
server.port: 5601
server.host: "192.168.1.1"
server.name: "node"
elasticsearch.hosts: ["http://192.168.1.1:9200"]
kibana.index: ".kibana"
$ systemctl start kibana
$ ss -lnt | grep 5601
LISTEN     0      128    192.168.1.1:5601                     *:*         

二、收集nginx日志

由于nginx的日志格式不是json的，收集起来也无法立即定位到关键信息，所以就直接转为json格式并进行拆分！

$ vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
$ yum -y install nginx httpd-tools
$ vim /etc/nginx/nginx.conf
#添加以下内容将其日志格式转换为json格式
    log_format json '{ "@time_local": "$time_local", '
                        '"remote_addr": "$remote_addr", '
                        '"referer": "$http_referer", '
                        '"request": "$request", '
                        '"status": $status, '
                        '"bytes": $body_bytes_sent, '
                        '"agent": "$http_user_agent", '
                        '"x_forwarded": "$http_x_forwarded_for", '
                        '"up_addr": "$upstream_addr",'
                        '"up_host": "$upstream_http_host",'
                        '"up_resp_time": "$upstream_response_time",'
                        '"request_time": "$request_time"'
' }';  

    access_log  /var/log/nginx/access.log  json;
$ nginx -t
$ nginx
$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.0-x86_64.rpm
$ yum -y install filebeat-6.6.0-x86_64.rpm
$ rm -rf /etc/filebeat/filebeat.yml 
$ vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["192.168.1.1:9200"]
  indices:
    - index: "nginx-acess-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
$ systemctl start filebeat
$ ab -n 1000 -c 100 http://192.168.1.2/
$ ab -n 1000 -c 100 http://192.168.1.2/test

kibana自行添加索引：

这样就可以将日志拆分成好几个字段，便于查找关键信息！

三、收集tomcat日志

tomcat日志默认情况下虽然是json格式，但是并没有进行拆分，所以，需要进行以下配置进行拆分！

$ yum -y install tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp tomcat-javadoc
$ vim /etc/tomcat/server.xml
#139行原本的删除，添加以下内容：
pattern="{"client":"%h",  "client user":"%l",   "authenticated":"%u",   "access time":"%t",     "method":"%r",   "status":"%s",  "send bytes":"%b",  "Query?string":"%q",  "partner":"%{Referer}i",  "Agent version":"%{User-Agent}i"}"/>
$ systemctl start tomcat
$ vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/tomcat/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]

output.elasticsearch:
  hosts: ["192.168.1.1:9200"]
  indices:
    - index: "tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "tomcat"

setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
setup.template.enabled: false
setup.template.overwrite: true
$ systemctl restart filebeat

自行访问tomcat，使其产生日志！

自行添加tomcat索引！

四、收集ES日志

因为ES的日志有点不同，需要用到多行匹配模式！直接在node主机上安装filebeat进行操作！

$ yum -y install filebeat-6.6.0-x86_64.rpm
$ vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /elk/log/elasticsearch.log
  tags: ["es"]
  multiline.pattern: '^['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["192.168.1.1:9200"]
  indices:
    - index: "es-java-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "es"

setup.template.name: "es"
setup.template.pattern: "es-java-*"
setup.template.enabled: false
setup.template.overwrite: true
$ systemctl start filebeat

想办法让ES产生一些错误日志！

自行创建索引。

这就是ES错误的特点，所以需要使用以上多行合并技术！

五、收集docker容器日志

如果需要实现安装docker、docker-compose！

$ yum install -y yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
$ yum install -y docker-ce-18.09.0-3.el7 docker-ce-cli-18.09.0-3.el7 containerd.io-1.2.0-3.el7
$ systemctl daemon-reload && systemctl start docker
$ curl -L https://github.com/docker/compose/releases/download/1.25.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose
$ mkdir compose && cd compose
[root@test compose]# vim docker-compose.yaml 
version: '3'
services:
  nginx:
    image: nginx
    labels:
      service: nginx
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"
  db:
    image: nginx
    labels:
      service: db
    logging:
      options:
        labels: "service"
    ports:
      - "3306:80"
#使用一个nginx镜像模拟两种服务
[root@test compose]# docker-compose up
$ vim /etc/filebeat/filebeat.yml 

filebeat.inputs:
- type: log
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true


output.elasticsearch:
  hosts: ["192.168.1.1:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
        stream: "stdout"

    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
        stream: "stderr"

    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"
        stream: "stdout"

    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"
        stream: "stderr"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
$ systemctl restart filebeat

访问容器中的服务，使其产生日志！

自行添加索引！

六、使用filebeat自带模块进行监控

filebeat自带了很多模块，这里以nginx为例！

$ vim /etc/filebeat/filebeat.yml 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s


output.elasticsearch:
  hosts: ["192.168.1.1:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "access"

    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
$ filebeat modules enable nginx
Enabled nginx
$ vim /etc/filebeat/modules.d/nginx.yml 

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]
$ yum -y install nginx
$ nginx

ES服务器需要安装以下两个插件才支持此功能！

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
$ systemctl restart elasticsearch

安装完成之后：

$ filebeat setup
$ systemctl restart filebeat
$ ab -c 100 -n 100 http://192.168.1.2/

添加错误日志索引：

七、Kibana的x-pack监控