Ubuntu环境部署Logstash实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.准备环境
1>.部署环境说明
Logstash可以单独找一台机器部署，它需要安装JDK环境，我这里为了省事，就直接和一台Elasticsearch节点复用同一个节点(es103.yinzhengjie.com)。 博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12953504.html
2>.下载kibana软件包
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12934518.html
二.部署Logstash实操
1>.将下载的Logstash软件包上传到es103.yinzhengjie.com节点并安装
[root@es103.yinzhengjie.com ~]# dpkg -i logstash-6.8.9.deb Selecting previously unselected package logstash. (Reading database ... 103047 files and directories currently installed.) Preparing to unpack logstash-6.8.9.deb ... Unpacking logstash (1:6.8.9-1) ... Setting up logstash (1:6.8.9-1) ... Using provided startup.options file: /etc/logstash/startup.options /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated Successfully created system startup script for Logstash [root@es103.yinzhengjie.com ~]#
2>.查看logstath的启动脚本
[root@es103.yinzhengjie.com ~]# find / -name logstash.service /etc/systemd/system/logstash.service [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service [Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target [root@es103.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service [Unit] Description=logstash [Service] Type=simple User=root Group=root # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target [root@es102.yinzhengjie.com ~]#
三.以标准输入为数据源测试Logstash可用性
1>.输出类型为标准输出案例
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}' WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-04 03:01:56.765 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-04 03:01:56.776 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-04 03:02:01.365 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-04 03:02:01.473 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x19207ef3 run>"} The stdin plugin is now waiting for input: [INFO ] 2020-06-04 03:02:01.540 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-04 03:02:01.778 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} 尹正杰到此一游! #这是我输出的数据 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@version" => "1", #事件版本号，一个事件就是一个ruby对象 "@timestamp" => 2020-06-04T03:02:40.589Z, #当前事件的发生时间 "message" => "尹正杰到此一游!", #消息的具体内容 "host" => "es103.yinzhengjie.com" #标记时间发生在哪个主机 }
2>.输出类型为文件案例
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file {path => "/tmp/log.txt"}}' WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-04 03:32:00.760 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-04 03:32:00.771 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-04 03:32:05.126 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-04 03:32:05.258 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b0b27c1 run>"} The stdin plugin is now waiting for input: [INFO ] 2020-06-04 03:32:05.319 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-04 03:32:05.522 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} 尹正杰到此一游! [INFO ] 2020-06-04 03:33:48.228 [[main]>worker1] file - Opening file {:path=>"/tmp/log.txt"} https://www.cnblogs.com/yinzhengjie/
[root@es103.yinzhengjie.com ~]# tail -10f /tmp/log.txt {"@timestamp":"2020-06-04T03:33:47.887Z","host":"es103.yinzhengjie.com","message":"尹正杰到此一游!","@version":"1"} {"@timestamp":"2020-06-04T03:34:28.523Z","host":"es103.yinzhengjie.com","message":"https://www.cnblogs.com/yinzhengjie/","@version":"1"}
3>.输出类型为Elasticsearch案例
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts=> ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:920 0"] index=> "yinzhengjie-log-%{+YYYY.MM.dd}"}}' WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-04 03:43:44.554 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-04 03:43:44.567 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-04 03:43:48.816 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-04 03:43:49.205 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102 .yinzhengjie.com:9200/]}}[WARN ] 2020-06-04 03:43:49.422 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [INFO ] 2020-06-04 03:43:49.561 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6} [WARN ] 2020-06-04 03:43:49.563 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version =>6}[WARN ] 2020-06-04 03:43:49.575 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [INFO ] 2020-06-04 03:43:49.626 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://es102.yinzhengjie.com:9200"]}[INFO ] 2020-06-04 03:43:49.651 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [INFO ] 2020-06-04 03:43:49.678 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refr esh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[INFO ] 2020-06-04 03:43:49.720 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x696425bf run>"} [INFO ] 2020-06-04 03:43:49.774 [Ruby-0-Thread-5: :1] elasticsearch - Installing elasticsearch template to _template/logstash The stdin plugin is now waiting for input: [INFO ] 2020-06-04 03:43:49.807 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-04 03:43:50.234 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} 尹正杰到此一游~ https://www.cnblogs.com/yinzhengjie/
数据是否写入成功我们在Kibana无法直接查看到，我们可以通过Elasticsearch的head插件看到。但这并不说Kibana无法查看刚刚通过Logstash写入的数据，而是需要创建索引。
四.Kibana创建索引
1>.打开Kibana的WebUI，如下图所示，依次点击"管理" ---> "索引模式"
2>.如下图所示，点击"创建索引模式"
3>.如下图所示，输入匹配模式后，点击"下一步"
4>.如下图所示，设置筛选时间的名称为"timestamp"即可
5>.索引创建成功
6>.点击Discover，查看刚刚创建的索引内容
五.博主推荐阅读