Elastic Stack之Redis集群使用
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
本篇博客数据流走向：FileBeat ===》Redis ===》logstash ===> elasticsearch。
一.安装Redis服务器
1>.安装repo源
[root@node105 ~]# [root@node105 ~]# yum -y install epel-release Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.cn99.com * extras: mirror.lzu.edu.cn * updates: mirror.lzu.edu.cn Resolving Dependencies --> Running transaction check ---> Package epel-release.noarch 0:7-11 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================== Installing: epel-release noarch 7-11 extras 15 k Transaction Summary =================================================================================================================================================================================================== Install 1 Package Total download size: 15 k Installed size: 24 k Downloading packages: epel-release-7-11.noarch.rpm | 15 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : epel-release-7-11.noarch 1/1 Verifying : epel-release-7-11.noarch 1/1 Installed: epel-release.noarch 0:7-11 Complete! [root@node105 ~]#
2>.安装redis服务器
[root@node105 ~]# [root@node105 ~]# yum -y install redis Loaded plugins: fastestmirror epel/x86_64/metalink | 7.9 kB 00:00:00 epel | 4.7 kB 00:00:00 (1/3): epel/x86_64/group_gz | 88 kB 00:00:00 (2/3): epel/x86_64/updateinfo | 959 kB 00:00:01 (3/3): epel/x86_64/primary_db | 6.6 MB 00:00:01 Loading mirror speeds from cached hostfile * base: mirrors.cn99.com * epel: mirrors.yun-idc.com * extras: mirror.lzu.edu.cn * updates: mirror.lzu.edu.cn Resolving Dependencies --> Running transaction check ---> Package redis.x86_64 0:3.2.12-2.el7 will be installed --> Processing Dependency: libjemalloc.so.1()(64bit) for package: redis-3.2.12-2.el7.x86_64 --> Running transaction check ---> Package jemalloc.x86_64 0:3.6.0-1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================== Installing: redis x86_64 3.2.12-2.el7 epel 544 k Installing for dependencies: jemalloc x86_64 3.6.0-1.el7 epel 105 k Transaction Summary =================================================================================================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 648 k Installed size: 1.7 M Downloading packages: warning: /var/cache/yum/x86_64/7/epel/packages/jemalloc-3.6.0-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY ] 0.0 B/s | 0 B --:--:-- ETA Public key for jemalloc-3.6.0-1.el7.x86_64.rpm is not installed (1/2): jemalloc-3.6.0-1.el7.x86_64.rpm | 105 kB 00:00:00 (2/2): redis-3.2.12-2.el7.x86_64.rpm | 544 kB 00:00:00 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 564 kB/s | 648 kB 00:00:01 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Importing GPG key 0x352C64E5: Userid : "Fedora EPEL (7) <epel@fedoraproject.org>" Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5 Package : epel-release-7-11.noarch (@extras) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : jemalloc-3.6.0-1.el7.x86_64 1/2 Installing : redis-3.2.12-2.el7.x86_64 2/2 Verifying : redis-3.2.12-2.el7.x86_64 1/2 Verifying : jemalloc-3.6.0-1.el7.x86_64 2/2 Installed: redis.x86_64 0:3.2.12-2.el7 Dependency Installed: jemalloc.x86_64 0:3.6.0-1.el7 Complete! [root@node105 ~]#
3>.修改Redis服务器的默认配置
[root@node105 ~]# [root@node105 ~]# grep bind /etc/redis.conf | grep -v ^# #指定绑定的ip地址为服务器的所有网卡！ bind 0.0.0.0 [root@node105 ~]# [root@node105 ~]# grep requirepass /etc/redis.conf | grep -v ^# #开启认证 requirepass yinzhengjie.org.cn [root@node105 ~]# [root@node105 ~]#
4>.启动redis服务器
[root@node105 ~]# [root@node105 ~]# systemctl enable redis Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /usr/lib/systemd/system/redis.service. [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# systemctl start redis [root@node105 ~]# [root@node105 ~]# systemctl status redis ● redis.service - Redis persistent key-value database Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/redis.service.d └─limit.conf Active: active (running) since Mon 2019-03-11 23:17:20 CST; 4s ago Main PID: 4102 (redis-server) CGroup: /system.slice/redis.service └─4102 /usr/bin/redis-server 0.0.0.0:6379 Mar 11 23:17:20 node105.yinzhengjie.org.cn systemd[1]: Starting Redis persistent key-value database... Mar 11 23:17:20 node105.yinzhengjie.org.cn systemd[1]: Started Redis persistent key-value database. [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# ss -tnl | grep 6379 LISTEN 0 128 *:6379 *:* [root@node105 ~]# [root@node105 ~]#
二.使用FileBeat结合redis收集日志文件案例
1>.安装FileBeat软件
详情请参考： Elastic Stack之FileBeat使用实战。
2>.修改FileBeat的配置文件并启动
[root@node105 ~]# [root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #" filebeat.prospectors: - input_type: log paths: - /var/log/httpd/access_log output.redis: enabled: true hosts: ["node105.yinzhengjie.org.cn:6379"] key: filebeat password: yinzhengjie.org.cn db: 0 datatype: list [root@node105 ~]# [root@node105 ~]#
3>.查看redis服务器
[root@node105 ~]# [root@node105 ~]# redis-cli -a yinzhengjie.org.cn 127.0.0.1:6379> 127.0.0.1:6379> KEYS * 1) "filebeat" 127.0.0.1:6379> 127.0.0.1:6379> LINDEX filebeat 0 "{"@timestamp":"2019-03-11T15:34:37.114Z","beat":{"hostname":"node105.yinzhengjie.org.cn","name":"node105.yinzhengjie.org.cn","version":"5.6.12"},"input_type":"log","message":"80.129.1.1 - - [11/Mar/2019:23:05:48 +0800] \"GET /test42.html HTTP/1.1\" 200 8 \"-\" \"curl/7.29.0\"","offset":134141,"source":"/var/log/httpd/access_log","type":"log"}" 127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat (integer) 1540 127.0.0.1:6379> [root@node105 ~]# redis-cli -a yinzhengjie.org.cn
4>.配置logstash的redis输入插件（参考链接：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html）
[root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf input { redis { host => "node105.yinzhengjie.org.cn" port => 6379 password => "yinzhengjie.org.cn" db => 0 key => "filebeat" data_type => "list" } } filter { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } remove_field => ["message","beat"] } date { match => ["timestamp","dd/MMM/YYYY:H:m:s Z"] remove_field => "timestamp" } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/maxmind/GeoLite2-City.mmdb" } } output { elasticsearch { hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"] index => "logstash-%{+YYYY.MM.dd}" document_type => "httpd_access_logs" } } [root@node105 ~]#
[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@node105 ~]#
5>.启动logstash并查看redis服务器
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done Page 34 Page 27 Page 7 Page 41 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> Page 48 Page 26 Page 40 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 8 Page 20 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test59.html was not found on this server.</p> </body></html> Page 43 Page 15 Page 46 Page 24 Page 19 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 32 Page 30 Page 20 Page 33 Page 8 Page 26 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> Page 25 Page 37 Page 20 Page 23 Page 19 Page 37 Page 47 Page 34 Page 10 Page 9 Page 10 Page 22 Page 23 Page 37 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 49 Page 19 Page 21 Page 6 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 15 Page 4 Page 27 Page 19 Page 29 Page 8 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 33 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> Page 49 Page 23 Page 27 Page 30 Page 9 Page 1 Page 44 Page 31 Page 12 Page 46 Page 26 Page 18 Page 6 Page 17 Page 27 Page 5 Page 49 Page 14 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> Page 36 Page 27 Page 32 Page 16 Page 25 Page 15 Page 4 Page 48 Page 25 Page 13 Page 32 Page 6 Page 1 Page 35 Page 25 Page 6 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 34 Page 49 Page 31 Page 15 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 12 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test60.html was not found on this server.</p> </body></html> Page 15 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 34 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 26 Page 8 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test56.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 36 Page 41 Page 49 Page 20 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 27 Page 5 Page 49 Page 23 Page 45 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test60.html was not found on this server.</p> </body></html> Page 6 Page 26 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> Page 5 Page 43 Page 46 Page 16 Page 39 Page 47 Page 22 Page 25 Page 15 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 21 Page 21 Page 11 Page 26 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test59.html was not found on this server.</p> </body></html> Page 27 Page 4 Page 49 Page 5 Page 5 Page 7 Page 12 Page 20 Page 12 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> Page 8 Page 18 Page 30 Page 27 Page 46 Page 34 Page 17 Page 5 Page 12 Page 26 Page 37 Page 13 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 44 Page 23 Page 11 Page 27 Page 16 Page 19 Page 18 Page 6 Page 34 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test59.html was not found on this server.</p> </body></html> Page 19 Page 25 Page 45 Page 36 Page 23 Page 1 Page 41 Page 47 Page 24 Page 45 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 43 Page 48 Page 28 Page 13 Page 30 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 19 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> Page 45 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test56.html was not found on this server.</p> </body></html> Page 43 Page 13 Page 6 Page 33 Page 17 Page 1 Page 17 Page 8 Page 42 Page 12 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 22 Page 15 Page 43 Page 48 Page 46 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 9 Page 29 Page 18 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 34 Page 37 Page 13 Page 13 Page 18 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test59.html was not found on this server.</p> </body></html> Page 15 Page 12 Page 33 Page 39 Page 5 Page 40 Page 16 Page 10 Page 18 Page 14 Page 31 Page 42 Page 4 Page 18 Page 11 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 14 Page 22 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 38 Page 46 Page 23 Page 46 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 44 Page 36 Page 22 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> Page 32 Page 15 Page 50 Page 23 Page 4 Page 11 Page 46 Page 10 Page 45 Page 15 Page 31 Page 5 Page 42 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 40 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test56.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test60.html was not found on this server.</p> </body></html> Page 40 Page 17 Page 33 Page 11 Page 16 Page 44 Page 15 Page 1 Page 7 Page 13 Page 42 Page 18 Page 30 Page 22 Page 36 Page 37 Page 30 Page 13 Page 12 Page 6 Page 1 Page 39 Page 45 Page 9 Page 16 Page 47 Page 20 Page 12 Page 4 Page 24 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 18 Page 4 Page 27 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> Page 6 Page 47 Page 1 Page 5 Page 4 Page 20 Page 14 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test56.html was not found on this server.</p> </body></html> Page 6 Page 26 Page 42 Page 34 Page 38 Page 4 Page 7 Page 15 Page 19 Page 38 Page 15 Page 8 Page 21 Page 47 Page 17 Page 28 Page 47 ^C [root@node103 ~]#
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn 127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat (integer) 5438 127.0.0.1:6379>
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn 127.0.0.1:6379> 127.0.0.1:6379> 127.0.0.1:6379> LLEN filebeat (integer) 0 127.0.0.1:6379>
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9035 100 9035 0 0 428k 0 --:--:-- --:--:-- --:--:-- 441k { "took": 11, "timed_out": false, "_shards": { "total": 15, "successful": 15, "skipped": 0, "failed": 0 }, "hits": { "total": 1445, "max_score": 1.8790942, "hits": [ { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAEi", "_score": 1.8790942, "_source": { "request": "/test55.html", "geoip": { "ip": "100.242.1.1", "latitude": 37.751, "country_name": "United States", "country_code2": "US", "continent_code": "NA", "country_code3": "US", "location": { "lon": -97.822, "lat": 37.751 }, "longitude": -97.822 }, "offset": 25922, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:02:33.000Z", "response": "404", "bytes": "209", "clientip": "100.242.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAEq", "_score": 1.8790942, "_source": { "request": "/test52.html", "geoip": { "timezone": "Europe/Paris", "ip": "82.234.1.1", "latitude": 48.6942, "continent_code": "EU", "city_name": "Brunoy", "country_name": "France", "country_code2": "FR", "country_code3": "FR", "region_name": "Essonne", "location": { "lon": 2.4922, "lat": 48.6942 }, "postal_code": "91800", "region_code": "91", "longitude": 2.4922 }, "offset": 26692, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:02:41.000Z", "response": "404", "bytes": "209", "clientip": "82.234.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAEt", "_score": 1.8790942, "_source": { "request": "/test52.html", "geoip": { "timezone": "Africa/Accra", "ip": "45.209.1.1", "latitude": 8, "country_name": "Ghana", "country_code2": "GH", "continent_code": "AF", "country_code3": "GH", "location": { "lon": -2, "lat": 8 }, "longitude": -2 }, "offset": 26979, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:02:44.000Z", "response": "404", "bytes": "209", "clientip": "45.209.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAE9", "_score": 1.8790942, "_source": { "request": "/test58.html", "geoip": { "timezone": "Asia/Tokyo", "ip": "111.89.1.1", "latitude": 34.9833, "continent_code": "AS", "city_name": "Shobara", "country_name": "Japan", "country_code2": "JP", "country_code3": "JP", "region_name": "Hiroshima", "location": { "lon": 132.9833, "lat": 34.9833 }, "postal_code": "727-0321", "region_code": "34", "longitude": 132.9833 }, "offset": 28514, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:01.000Z", "response": "404", "bytes": "209", "clientip": "111.89.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAFH", "_score": 1.8790942, "_source": { "request": "/test51.html", "geoip": { "timezone": "Europe/Madrid", "ip": "83.47.1.1", "latitude": 36.54, "continent_code": "EU", "city_name": "Fuengirola", "country_name": "Spain", "country_code2": "ES", "country_code3": "ES", "region_name": "Malaga", "location": { "lon": -4.6247, "lat": 36.54 }, "postal_code": "29640", "region_code": "MA", "longitude": -4.6247 }, "offset": 29476, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:11.000Z", "response": "404", "bytes": "209", "clientip": "83.47.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAFY", "_score": 1.8790942, "_source": { "request": "/test53.html", "geoip": { "timezone": "Europe/Rome", "ip": "94.164.1.1", "latitude": 42.5245, "continent_code": "EU", "city_name": "Piansano", "country_name": "Italy", "country_code2": "IT", "country_code3": "IT", "region_name": "Provincia di Viterbo", "location": { "lon": 11.8298, "lat": 42.5245 }, "postal_code": "01010", "region_code": "VT", "longitude": 11.8298 }, "offset": 31107, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:28.000Z", "response": "404", "bytes": "209", "clientip": "94.164.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAFd", "_score": 1.8790942, "_source": { "request": "/test55.html", "geoip": { "timezone": "Africa/Tunis", "ip": "154.106.1.1", "latitude": 34, "country_name": "Tunisia", "country_code2": "TN", "continent_code": "AF", "country_code3": "TN", "location": { "lon": 9, "lat": 34 }, "longitude": 9 }, "offset": 31590, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:33.000Z", "response": "404", "bytes": "209", "clientip": "154.106.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN80lXxXllWpXYAFq", "_score": 1.8790942, "_source": { "request": "/test56.html", "geoip": { "ip": "33.65.1.1", "latitude": 37.751, "country_name": "United States", "country_code2": "US", "continent_code": "NA", "country_code3": "US", "location": { "lon": -97.822, "lat": 37.751 }, "longitude": -97.822 }, "offset": 32832, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:46.000Z", "response": "404", "bytes": "209", "clientip": "33.65.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN85S3WCT5NaOiwHV", "_score": 1.8790942, "_source": { "request": "/test57.html", "geoip": { "timezone": "America/Bogota", "ip": "179.19.1.1", "latitude": 4.5981, "country_name": "Colombia", "country_code2": "CO", "continent_code": "SA", "country_code3": "CO", "location": { "lon": -74.0758, "lat": 4.5981 }, "longitude": -74.0758 }, "offset": 33600, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:03:54.000Z", "response": "404", "bytes": "209", "clientip": "179.19.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltN85T3WCT5NaOiwHc", "_score": 1.8790942, "_source": { "request": "/test51.html", "geoip": { "city_name": "Central", "timezone": "Asia/Hong_Kong", "ip": "13.94.1.1", "latitude": 22.2909, "country_name": "Hong Kong", "country_code2": "HK", "continent_code": "AS", "country_code3": "HK", "region_name": "Central and Western District", "location": { "lon": 114.15, "lat": 22.2909 }, "region_code": "HCW", "longitude": 114.15 }, "offset": 34272, "auth": "-", "ident": "-", "input_type": "log", "verb": "GET", "source": "/var/log/httpd/access_log", "type": "log", "tags": [ "beats_input_codec_plain_applied" ], "referrer": ""-"", "@timestamp": "2019-03-11T14:04:01.000Z", "response": "404", "bytes": "209", "clientip": "13.94.1.1", "@version": "1", "beat": { "name": "node105.yinzhengjie.org.cn", "hostname": "node105.yinzhengjie.org.cn", "version": "5.6.12" }, "host": "node105.yinzhengjie.org.cn", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } } ] } } [root@node101 ~]#
三.使用一个FileBeat加载多种不同的日志案例
1>.编写FileBeat的配置文件
[root@node105 ~]# [root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #" filebeat.prospectors: - input_type: log paths: - /var/log/httpd/access_log fields: log_type: access - paths: - /var/log/httpd/error_log fields: log_type: errors output.redis: enabled: true hosts: ["node105.yinzhengjie.org.cn:6379"] key: filebeat password: yinzhengjie.org.cn db: 0 datatype: list logging.level: debug [root@node105 ~]# [root@node105 ~]#
2>.编写logstash的配置文件
[root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf input { redis { host => "node105.yinzhengjie.org.cn" port => 6379 password => "yinzhengjie.org.cn" db => 0 key => "filebeat" data_type => "list" } } filter { if [fields][log_type] == "access" { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } remove_field => ["message","beat"] } date { match => ["timestamp","dd/MMM/YYYY:H:m:s Z"] remove_field => "timestamp" } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/maxmind/GeoLite2-City.mmdb" } } } output { if [fields][log_type] == "access" { elasticsearch { hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"] index => "logstash-%{+YYYY.MM.dd}" document_type => "httpd_access_logs" } } else { elasticsearch { hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"] index => "logstash-%{+YYYY.MM.dd}" document_type => "httpd_error_logs" } } } [root@node105 ~]# [root@node105 ~]#
[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@node105 ~]#
3>.启动logstash服务
[root@node105 ~]# [root@node105 ~]# mkdir -pv /yinzhengjie/logstash/conf.d mkdir: created directory ‘/yinzhengjie/logstash’ mkdir: created directory ‘/yinzhengjie/logstash/conf.d’ [root@node105 ~]# [root@node105 ~]# mv /etc/logstash/conf.d/* /yinzhengjie/logstash/conf.d/ [root@node105 ~]# [root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/ [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# ll /etc/logstash/conf.d/ total 4 -rw-r--r--. 1 root root 1112 Mar 12 21:45 redis-condition-elasticsearch.conf [root@node105 ~]#
[root@node105 ~]# [root@node105 ~]# [root@node105 ~]# systemctl start logstash [root@node105 ~]# [root@node105 ~]# systemctl status logstash ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-12 21:46:06 CST; 5s ago Main PID: 3200 (java) CGroup: /system.slice/logstash.service └─3200 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8... Mar 12 21:46:06 node105.yinzhengjie.org.cn systemd[1]: Started logstash. Mar 12 21:46:06 node105.yinzhengjie.org.cn systemd[1]: Starting logstash... [root@node105 ~]# [root@node105 ~]#
[root@node105 ~]# [root@node105 ~]# ps -ef | grep logstash | grep -v grep logstash 3200 1 15 21:46 ? 00:00:51 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# netstat -untalp | grep 3200 tcp6 0 0 127.0.0.1:9600 :::* LISTEN 3200/java tcp6 0 0 172.30.1.105:48739 172.30.1.101:9200 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:46093 172.30.1.102:9200 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:50903 172.30.1.103:9200 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:42682 172.30.1.105:6379 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:50900 172.30.1.103:9200 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:46090 172.30.1.102:9200 ESTABLISHED 3200/java tcp6 0 0 172.30.1.105:48736 172.30.1.101:9200 ESTABLISHED 3200/java [root@node105 ~]# [root@node105 ~]#