Elastic Stack之Logstash进阶
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.使用GeoLite2和logstash 过滤插件的geoip案例
1>.GeoLite2概述
GeoLite2数据库是免费的IP地理定位数据库，与MaxMind的GeoIP2数据库相当，但不太准确。GeoLite2国家和城市数据库在每个月的第一个星期二更新。GeoLite2 ASN数据库每周二更新一次。官方网址：https://www.maxmind.com/en/home。
2>.下载GeoLite2的免费库（下载地址：https://dev.maxmind.com/geoip/geoip2/geolite2/）
[root@node105 ~]# ll total 102868 -rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz --2019-03-11 21:01:11-- https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz Resolving geolite.maxmind.com (geolite.maxmind.com)... 104.17.201.89, 104.17.200.89, 2606:4700::6811:c959, ... Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.17.201.89|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 28513410 (27M) [application/gzip] Saving to: ‘GeoLite2-City.tar.gz’ 100%[===========================================================================================================================================================>] 28,513,410 197KB/s in 1m 59s 2019-03-11 21:03:12 (234 KB/s) - ‘GeoLite2-City.tar.gz’ saved [28513410/28513410] [root@node105 ~]# [root@node105 ~]# ll total 130716 -rw-r--r--. 1 root root 28513410 Mar 4 23:29 GeoLite2-City.tar.gz -rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm [root@node105 ~]# [root@node105 ~]#
3>.解压GeoLite并创建软连接
[root@node105 ~]# [root@node105 ~]# mkdir /etc/logstash/maxmind [root@node105 ~]# [root@node105 ~]# ll total 130716 -rw-r--r--. 1 root root 28513410 Mar 4 23:29 GeoLite2-City.tar.gz -rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/ [root@node105 ~]# [root@node105 ~]# ll /etc/logstash/maxmind/GeoLite2-City_20190305/ total 58236 -rw-r--r--. 1 2000 2000 55 Mar 4 23:29 COPYRIGHT.txt -rw-r--r--. 1 2000 2000 59618725 Mar 4 23:29 GeoLite2-City.mmdb -rw-r--r--. 1 2000 2000 433 Mar 4 23:29 LICENSE.txt -rw-r--r--. 1 2000 2000 116 Mar 4 23:29 README.txt [root@node105 ~]# [root@node105 ~]#
[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/ ‘/etc/logstash/maxmind/GeoLite2-City.mmdb’ -> ‘/etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb’ [root@node105 ~]# [root@node105 ~]# ll /etc/logstash/maxmind/ total 0 drwxr-xr-x. 2 2000 2000 86 Mar 4 23:29 GeoLite2-City_20190305 lrwxrwxrwx. 1 root root 63 Mar 11 21:13 GeoLite2-City.mmdb -> /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb [root@node105 ~]# [root@node105 ~]#
4>.编写logstash配置文件并测试语法（）
[root@node105 ~]# [root@node105 ~]# cp /etc/logstash/conf.d/file-date-stdout.conf /etc/logstash/conf.d/file-date-geoip-stdout.conf [root@node105 ~]# [root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf input { file { path => ["/var/log/httpd/access_log"] start_position => "beginning" } } filter { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } remove_field => "message" } date { match => ["timestamp","dd/MMM/YYYY:H:m:s Z"] remove_field => "timestamp" } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/maxmind/GeoLite2-City.mmdb" } } output { stdout { codec => rubydebug } } [root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@node105 ~]#
5>.启动logstash的geoip相关配置文件（参考链接：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-geoip.html）
[root@node103 ~]# [root@node103 ~]# [root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done Page 49 Page 50 Page 43 Page 8 Page 22 Page 44 Page 10 Page 4 Page 24 Page 12 Page 45 Page 12 Page 40 Page 8 Page 30 Page 47 Page 14 Page 35 Page 41 Page 40 Page 8 Page 33 Page 13 Page 10 Page 47 Page 31 Page 35 Page 12 Page 37 Page 38 Page 11 Page 14 Page 25 Page 7 Page 44 Page 5 Page 43 Page 27 Page 2 Page 24 Page 2 Page 25 Page 36 Page 4 Page 5 Page 21 Page 11 Page 8 Page 45 Page 10 Page 21 Page 2 Page 16 Page 23 Page 24 Page 49 Page 8 Page 7 Page 29 Page 1 Page 41 Page 37 Page 4 Page 49 Page 35 Page 29 Page 32 Page 50 Page 3 Page 6 Page 6 Page 39 Page 50 Page 2 Page 10 Page 24 Page 16 Page 47 Page 48 Page 5 Page 48 Page 3 Page 40 Page 7 Page 18 Page 49 Page 24 Page 48 Page 1 Page 50 Page 49 Page 18 Page 3 Page 10 Page 12 Page 12 Page 48 Page 50 Page 27 Page 18 Page 21 Page 15 Page 2 Page 49 Page 25 Page 41 Page 10 Page 35 Page 10 Page 30 Page 48 Page 25 Page 31 Page 21 Page 36 Page 7 Page 26 Page 35 Page 27 Page 45 Page 28 Page 18 Page 47 Page 47 Page 33 Page 20 Page 25 Page 19 Page 17 Page 19 ^C [root@node103 ~]#
[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console { "request" => "/test35.html", "agent" => ""curl/7.29.0"", "geoip" => { "timezone" => "Europe/London", "ip" => "85.211.1.1", "latitude" => 52.4768, "continent_code" => "EU", "city_name" => "Birmingham", "country_name" => "United Kingdom", "country_code2" => "GB", "country_code3" => "GB", "region_name" => "Birmingham", "location" => { "lon" => -1.9341, "lat" => 52.4768 }, "postal_code" => "B16", "region_code" => "BIR", "longitude" => -1.9341 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:03.000Z, "response" => "200", "bytes" => "8", "clientip" => "85.211.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } { "request" => "/test12.html", "agent" => ""curl/7.29.0"", "geoip" => { "timezone" => "America/New_York", "ip" => "108.5.1.1", "latitude" => 40.7667, "continent_code" => "NA", "city_name" => "Union City", "country_name" => "United States", "country_code2" => "US", "dma_code" => 501, "country_code3" => "US", "region_name" => "New Jersey", "location" => { "lon" => -74.0311, "lat" => 40.7667 }, "postal_code" => "07087", "region_code" => "NJ", "longitude" => -74.0311 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:04.000Z, "response" => "200", "bytes" => "8", "clientip" => "108.5.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } { "request" => "/test37.html", "agent" => ""curl/7.29.0"", "geoip" => { "timezone" => "America/Chicago", "ip" => "24.118.1.1", "latitude" => 45.0139, "continent_code" => "NA", "city_name" => "Saint Paul", "country_name" => "United States", "country_code2" => "US", "dma_code" => 613, "country_code3" => "US", "region_name" => "Minnesota", "location" => { "lon" => -93.1545, "lat" => 45.0139 }, "postal_code" => "55113", "region_code" => "MN", "longitude" => -93.1545 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:05.000Z, "response" => "200", "bytes" => "8", "clientip" => "24.118.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } { "request" => "/test38.html", "agent" => ""curl/7.29.0"", "geoip" => { "ip" => "55.27.1.1", "latitude" => 37.751, "country_name" => "United States", "country_code2" => "US", "continent_code" => "NA", "country_code3" => "US", "location" => { "lon" => -97.822, "lat" => 37.751 }, "longitude" => -97.822 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:06.000Z, "response" => "200", "bytes" => "8", "clientip" => "55.27.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } { "request" => "/test11.html", "agent" => ""curl/7.29.0"", "geoip" => { "timezone" => "America/Los_Angeles", "ip" => "3.173.1.1", "latitude" => 47.6348, "continent_code" => "NA", "city_name" => "Seattle", "country_name" => "United States", "country_code2" => "US", "dma_code" => 819, "country_code3" => "US", "region_name" => "Washington", "location" => { "lon" => -122.3451, "lat" => 47.6348 }, "postal_code" => "98109", "region_code" => "WA", "longitude" => -122.3451 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:07.000Z, "response" => "200", "bytes" => "8", "clientip" => "3.173.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } { "request" => "/test14.html", "agent" => ""curl/7.29.0"", "geoip" => { "city_name" => "Guayaquil", "timezone" => "America/Guayaquil", "ip" => "191.99.1.1", "latitude" => -2.1664, "country_name" => "Ecuador", "country_code2" => "EC", "continent_code" => "SA", "country_code3" => "EC", "region_name" => "Provincia del Guayas", "location" => { "lon" => -79.9011, "lat" => -2.1664 }, "region_code" => "G", "longitude" => -79.9011 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:27:08.000Z, "response" => "200", "bytes" => "8", "clientip" => "191.99.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1" } ^C[root@node105 ~]#
二.logstash 过滤插件的Mutate案例
1>.mutate概述
mutate过滤器允许您在字段上执行常规突变。您可以重命名，删除，替换和修改事件中的字段。详情请参考：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html。
2>.编写mutate案例
[root@node105 ~]# [root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf [root@node105 ~]# [root@node105 ~]# vi /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf [root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf input { file { path => ["/var/log/httpd/access_log"] start_position => "beginning" } } filter { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } remove_field => "message" } date { match => ["timestamp","dd/MMM/YYYY:H:m:s Z"] remove_field => "timestamp" } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/maxmind/GeoLite2-City.mmdb" } mutate { rename => { "agent" => "user_agent" } } } output { stdout { codec => rubydebug } } [root@node105 ~]# [root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf ^C [root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@node105 ~]#
3>.启动案例
[root@node103 ~]# [root@node103 ~]# [root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done Page 32 Page 32 Page 38 Page 15 Page 17 Page 19 Page 2 Page 6 Page 34 Page 18 Page 35 Page 49 Page 4 Page 42 Page 49 Page 28 Page 4 Page 42 Page 41 Page 45 Page 38 Page 28 Page 10 Page 24 Page 30 Page 41 Page 5 Page 39 Page 45 Page 26 Page 13 Page 39 Page 44 Page 43 Page 9 Page 45 Page 12 Page 3 Page 48 Page 3 Page 24 Page 17 Page 12 Page 8 Page 30 ^C [root@node103 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console { "request" => "/test32.html", "geoip" => { "timezone" => "America/New_York", "ip" => "73.137.1.1", "latitude" => 33.9135, "continent_code" => "NA", "city_name" => "Powder Springs", "country_name" => "United States", "country_code2" => "US", "dma_code" => 524, "country_code3" => "US", "region_name" => "Georgia", "location" => { "lon" => -84.6859, "lat" => 33.9135 }, "postal_code" => "30127", "region_code" => "GA", "longitude" => -84.6859 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:39:34.000Z, "response" => "200", "bytes" => "8", "clientip" => "73.137.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1", "user_agent" => ""curl/7.29.0"" } { "request" => "/test32.html", "geoip" => { "city_name" => "Daegu", "timezone" => "Asia/Seoul", "ip" => "119.201.1.1", "latitude" => 35.8723, "country_name" => "South Korea", "country_code2" => "KR", "continent_code" => "AS", "country_code3" => "KR", "region_name" => "Daegu", "location" => { "lon" => 128.5924, "lat" => 35.8723 }, "region_code" => "27", "longitude" => 128.5924 }, "auth" => "-", "ident" => "-", "verb" => "GET", "path" => "/var/log/httpd/access_log", "referrer" => ""-"", "@timestamp" => 2019-03-11T13:39:35.000Z, "response" => "200", "bytes" => "8", "clientip" => "119.201.1.1", "@version" => "1", "host" => "0.0.0.0", "httpversion" => "1.1", "user_agent" => ""curl/7.29.0"" } ^C[root@node105 ~]#
三.logstash 输出插件之elasticsearch输出插件
1>.elasticsearch输出插件概述
此插件是在Elasticsearch中存储日志的推荐方法。如果您打算使用Kibana Web界面，则需要使用此输出。此输出仅说HTTP协议。从Logstash 2.0开始，HTTP是与Elasticsearch交互的首选协议。出于多种原因，我们强烈建议在节点协议上使用HTTP。HTTP只是稍微慢一点，但更容易管理和使用。使用HTTP协议时，可以升级Elasticsearch版本，而无需在锁定步骤中升级Logstash。官方文档：https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-elasticsearch.html。
2>.配置elasticsearch集群输出
[root@node105 ~]# [root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf input { file { path => ["/var/log/httpd/access_log"] start_position => "beginning" } } filter { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}" } remove_field => "message" } date { match => ["timestamp","dd/MMM/YYYY:H:m:s Z"] remove_field => "timestamp" } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/maxmind/GeoLite2-City.mmdb" } mutate { rename => { "agent" => "user_agent" } } } output { elasticsearch { hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"] index => "logstash-%{+YYYY.MM.dd}" document_type => "httpd_access_logs" } } [root@node105 ~]#
[root@node105 ~]# [root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@node105 ~]#
3>.运行logstash 配置文件并查看es集群是否有新的索引
[root@node103 ~]# [root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test59.html was not found on this server.</p> </body></html> Page 32 Page 24 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 27 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test60.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test60.html was not found on this server.</p> </body></html> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> Page 45 Page 35 Page 16 Page 26 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 26 Page 25 Page 24 Page 20 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 6 Page 7 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 41 Page 30 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test52.html was not found on this server.</p> </body></html> Page 34 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 37 Page 40 Page 42 Page 39 Page 23 Page 4 Page 11 Page 24 Page 40 Page 26 Page 2 Page 47 Page 25 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test58.html was not found on this server.</p> </body></html> Page 47 Page 26 Page 5 Page 39 Page 31 Page 23 Page 27 Page 36 Page 26 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 43 Page 41 Page 43 Page 46 Page 29 Page 40 Page 49 Page 18 Page 24 Page 46 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test54.html was not found on this server.</p> </body></html> Page 21 Page 41 Page 38 Page 40 Page 47 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test53.html was not found on this server.</p> </body></html> Page 40 Page 22 Page 38 Page 34 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test55.html was not found on this server.</p> </body></html> Page 8 Page 5 Page 38 Page 25 Page 5 Page 6 Page 22 Page 26 Page 47 Page 45 Page 44 Page 31 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test56.html was not found on this server.</p> </body></html> Page 47 Page 41 Page 21 Page 50 Page 43 Page 21 Page 40 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> Page 22 Page 20 Page 50 Page 37 Page 21 Page 32 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test51.html was not found on this server.</p> </body></html> Page 34 Page 3 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /test57.html was not found on this server.</p> </body></html> Page 29 ^C [root@node103 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node101 ~]# [root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 134 100 134 0 0 14773 0 --:--:-- --:--:-- --:--:-- 16750 { "took": 2, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 0, "max_score": null, "hits": [] } } [root@node101 ~]#
[root@node101 ~]# [root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 877 100 877 0 0 73567 0 --:--:-- --:--:-- --:--:-- 73083 { "took": 5, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 2.0794415, "hits": [ { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltCr5Hsru-A5a8RIhU", "_score": 2.0794415, "_source": { "request": "/test17.html", "geoip": { "timezone": "America/Mexico_City", "ip": "187.152.1.1", "latitude": 20.6347, "continent_code": "NA", "city_name": "Guadalajara", "country_name": "Mexico", "country_code2": "MX", "country_code3": "MX", "region_name": "Jalisco", "location": { "lon": -103.4344, "lat": 20.6347 }, "postal_code": "44970", "region_code": "JAL", "longitude": -103.4344 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T13:40:15.000Z", "response": "200", "bytes": "8", "clientip": "187.152.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } } ] } } [root@node101 ~]#
[root@node101 ~]# [root@node101 ~]# [root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6817 100 6817 0 0 250k 0 --:--:-- --:--:-- --:--:-- 256k { "took": 19, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 20, "max_score": 2.3795462, "hits": [ { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltEH9tsru-A5a8RIhq", "_score": 2.3795462, "_source": { "request": "/test51.html", "geoip": { "timezone": "Europe/Madrid", "ip": "83.47.1.1", "latitude": 36.54, "continent_code": "EU", "city_name": "Fuengirola", "country_name": "Spain", "country_code2": "ES", "country_code3": "ES", "region_name": "Malaga", "location": { "lon": -4.6247, "lat": 36.54 }, "postal_code": "29640", "region_code": "MA", "longitude": -4.6247 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:03:11.000Z", "response": "404", "bytes": "209", "clientip": "83.47.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltEUMs3WCT5NaOiwE7", "_score": 2.3795462, "_source": { "request": "/test51.html", "geoip": { "city_name": "Central", "timezone": "Asia/Hong_Kong", "ip": "13.94.1.1", "latitude": 22.2909, "country_name": "Hong Kong", "country_code2": "HK", "continent_code": "AS", "country_code3": "HK", "region_name": "Central and Western District", "location": { "lon": 114.15, "lat": 22.2909 }, "region_code": "HCW", "longitude": 114.15 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:04:01.000Z", "response": "404", "bytes": "209", "clientip": "13.94.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltECF4sru-A5a8RIhi", "_score": 2.0794415, "_source": { "request": "/test51.html", "geoip": { "timezone": "Europe/Oslo", "ip": "78.91.1.1", "latitude": 63.4167, "continent_code": "EU", "city_name": "Trondheim", "country_name": "Norway", "country_code2": "NO", "country_code3": "NO", "region_name": "Trøndelag", "location": { "lon": 10.4167, "lat": 63.4167 }, "postal_code": "7003", "region_code": "23", "longitude": 10.4167 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:02:46.000Z", "response": "404", "bytes": "209", "clientip": "78.91.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltD9sF3WCT5NaOiwEd", "_score": 2.0794415, "_source": { "request": "/test57.html", "geoip": { "ip": "175.91.1.1", "latitude": 34.7725, "country_name": "China", "country_code2": "CN", "continent_code": "AS", "country_code3": "CN", "location": { "lon": 113.7266, "lat": 34.7725 }, "longitude": 113.7266 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:02:28.000Z", "response": "404", "bytes": "209", "clientip": "175.91.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltD-6fXxXllWpXYACG", "_score": 2.0794415, "_source": { "request": "/test55.html", "geoip": { "ip": "100.242.1.1", "latitude": 37.751, "country_name": "United States", "country_code2": "US", "continent_code": "NA", "country_code3": "US", "location": { "lon": -97.822, "lat": 37.751 }, "longitude": -97.822 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:02:33.000Z", "response": "404", "bytes": "209", "clientip": "100.242.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltD7u03WCT5NaOiwEZ", "_score": 2.0794415, "_source": { "request": "/test59.html", "geoip": { "timezone": "Asia/Tokyo", "ip": "126.210.1.1", "latitude": 35.69, "country_name": "Japan", "country_code2": "JP", "continent_code": "AS", "country_code3": "JP", "location": { "lon": 139.69, "lat": 35.69 }, "longitude": 139.69 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:02:20.000Z", "response": "404", "bytes": "209", "clientip": "126.210.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltEKqCsru-A5a8RIhw", "_score": 2.0512707, "_source": { "request": "/test54.html", "geoip": { "timezone": "Asia/Tokyo", "ip": "60.137.1.1", "latitude": 34.9667, "continent_code": "AS", "city_name": "Nagoya", "country_name": "Japan", "country_code2": "JP", "country_code3": "JP", "region_name": "Aichi", "location": { "lon": 136.9667, "lat": 34.9667 }, "postal_code": "470-2101", "region_code": "23", "longitude": 136.9667 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:03:22.000Z", "response": "404", "bytes": "209", "clientip": "60.137.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltD9Mu3WCT5NaOiwEc", "_score": 2.0512707, "_source": { "request": "/test58.html", "geoip": { "ip": "12.254.1.1", "latitude": 37.751, "country_name": "United States", "country_code2": "US", "continent_code": "NA", "country_code3": "US", "location": { "lon": -97.822, "lat": 37.751 }, "longitude": -97.822 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:02:26.000Z", "response": "404", "bytes": "209", "clientip": "12.254.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltEVLT3WCT5NaOiwE9", "_score": 2.0512707, "_source": { "request": "/test57.html", "geoip": { "timezone": "Asia/Shanghai", "ip": "113.8.1.1", "latitude": 45.75, "country_name": "China", "country_code2": "CN", "continent_code": "AS", "country_code3": "CN", "region_name": "Heilongjiang", "location": { "lon": 126.65, "lat": 45.75 }, "region_code": "HL", "longitude": 126.65 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:04:04.000Z", "response": "404", "bytes": "209", "clientip": "113.8.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } }, { "_index": "logstash-2019.03.11", "_type": "httpd_access_logs", "_id": "AWltESfDsru-A5a8RIh5", "_score": 2.0512707, "_source": { "request": "/test57.html", "geoip": { "timezone": "America/Bogota", "ip": "179.19.1.1", "latitude": 4.5981, "country_name": "Colombia", "country_code2": "CO", "continent_code": "SA", "country_code3": "CO", "location": { "lon": -74.0758, "lat": 4.5981 }, "longitude": -74.0758 }, "auth": "-", "ident": "-", "verb": "GET", "path": "/var/log/httpd/access_log", "referrer": ""-"", "@timestamp": "2019-03-11T14:03:54.000Z", "response": "404", "bytes": "209", "clientip": "179.19.1.1", "@version": "1", "host": "0.0.0.0", "httpversion": "1.1", "user_agent": ""curl/7.29.0"" } } ] } } [root@node101 ~]#