filebeat 收集日志

一、filebeat收集单日志到本地文件

1.配置

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.file:
  path: "/tmp/"
  filename: "nginx.log"r

2.启动

[root@web01 ~]# systemctl start filebeat.service

二、filebeat收集单日志到ES

1.配置

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]

2.启动

[root@web01 ~]# systemctl restart filebeat.service

三、filebeat收集单日志json格式到ES

1.配置nginx的json格式日志

[root@web01 ~]# cat /etc/nginx/nginx.conf
http {
	... ...
	log_format json '{ "time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr",'
                          '"up_host": "$upstream_http_host",'
                          '"upstream_time": "$upstream_response_time",'
                          '"request_time": "$request_time" }';

    access_log  /var/log/nginx/access.log  json;
 ... ...

2.配置收集日志

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]

3.启动

[root@web01 ~]# systemctl restart nginx

四、自定义ES索引名称

1.配置

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  index: "nginx_json_log_%{+yyyy-MM-dd}"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

#注意：配置索引模板需要顶头写，模板名称与指定索引名字无关

2.启动

[root@web01 ~]# systemctl restart nginx

五、filebeat收集单日志到redis

1.配置

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.redis:
  hosts: ["10.0.0.81:6379"]
  key: "nginx_log"
  db: 0

2.启动

3.redis查看数据

127.0.0.1:6379> keys *
1) "nginx_log"
127.0.0.1:6379> LLEN nginx_log
(integer) 33

六、filebeat收集单日志到logstash

1.配置

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  hosts: ["10.0.0.81:7890"]

2.启动

[root@web01 ~]# systemctl restart filebeat.service

3.配置logstash

[root@redis01 ~]# vim /etc/logstash/conf.d/filebeat_logstash_es.conf
input {
  beats {
    port => "7890"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.71:9200"]
    index => "filebeat_logstash_%{+YYYY-MM-dd}"
  }
}

[root@redis01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat_logstash_es.conf &

七、filebeat收集多日志到ES

1.方法一：

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
  json.keys_und:er_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  index: "nginx_json_%{+yyyy-MM-dd}"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

2.方法二：

[root@web01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  index: "nginx_json_%{+yyyy-MM-dd}"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

八、filebeat收集多日志到多个ES索引

1.方法一：

[root@web01 ~]# cat !$
cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  indices:
    - index: "nginx_access_%{+yyyy-MM-dd}"
      when.contains:
        source: "/var/log/nginx/access.log"
    - index: "nginx_error_%{+yyyy-MM-dd}"
      when.contains:
        source: "/var/log/nginx/error.log"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

2.方法二

[root@web01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  indices:
    - index: "nginx_access_%{+yyyy-MM-dd}"
      when.contains:
        tags: "access"
    - index: "nginx_error_%{+yyyy-MM-dd}"
      when.contains:
        tags: "error"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

九、filebeat收集java的报错日志

1.配置收集tomcat日志

[root@web01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/tomcat_access_json.*.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  index: "tomcat_access_%{+yyyy-MM-dd}"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"

2.配置收集java报错日志

# 修改java配置文件改回默认日志格式
[root@web01 ~]# vim /usr/local/tomcat/conf/server.xml

        <!--Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_json" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/-->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

[root@web01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/localhost_access_log.*.txt
  multiline.pattern: '^['
  multiline.negate: true
  multiline.match: after
  json.keys_under_root: true
  json.overwrite_keys: true
  json.message_key: log

output.elasticsearch:
  hosts: ["http://10.0.0.71:9200"]
  index: "tomcat_access_%{+yyyy-MM-dd}"
setup.template.name: "filebeat-*"
setup.template.pattern: "filebeat-*"