ELK安装部署

一、ELK简介

ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。
Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等结构提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。
Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括 syslog、消息传递（例如 RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。
Kibana是一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据，不仅允许用户创建他们自己的数据的定制仪表板视图，还允许他们以特殊的方式查询和过滤数据。
Beats：filebeat:日志文件，metricbeat:类似cpu/内存度量数据，packetbeat:网络数据，winlogbeat:windows数据，heartbeat：健康检查
官网地址：https://www.elastic.co/cn/

二、ELK安装环境

服务器IP地址	操作系统版本	节点名称	备注
192.168.8.120	RHEL6.6_X86_64	kibana节点	elasticsearch
192.168.8.121	RHEL6.6_X86_64	logstash节点	elasticsearch
192.168.8.122	RHEL6.6_X86_64		elasticsearch

在主机hosts文件中配置主机IP解析

192.168.8.120 master-node
192.168.8.121 data-node1
192.168.8.122 data-node2

1.软件下载

elasticsearch下载：https://www.elastic.co/cn/downloads/elasticsearch

　　　　　　https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.0-linux-x86_64.tar.gz

logstash下载：https://www.elastic.co/cn/downloads/logstash

kibana下载：https://www.elastic.co/cn/downloads/kibana

　　　　　　https://artifacts.elastic.co/downloads/kibana/kibana-7.5.0-linux-x86_64.tar.gz

JDK下载：https://download.oracle.com/otn/java/jdk/8u231-b11/

2.基础架构图

3.普通用户创建

[root@Linux7 soft]# useradd elkuser
[root@Linux7 soft]# passwd elkuser
Changing password for user elkuser.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

4.jdk安装

[root@Linux7 soft]# rpm -ivh jdk-8u231-linux-x64.rpm

5.操作系统环境配置

[root@Linux7 ~]# vi /etc/security/limits.conf

* soft nproc 204800
* hard nproc 204800
* soft nofile 655360
* hard nofile 655360
* soft memlock unlimited
* hard memlock unlimited

[root@Linux7 ~]# vi /etc/sysctl.conf

vm.max_map_count=1024000

###RHEL6.0:[root@Linux6 data]# vi /etc/security/limits.d/90-nproc.conf  增加配置内容：*          soft    nproc     65535

###RHEL7.0：[root@Linux7 ~]# vi /etc/security/limits.d/20-nproc.conf    增加配置内容：*          soft    nproc     65535

[root@Linux7 ~]# sysctl  -p

6.elasticsearch安装

[root@Linux7 soft]# tar -xzvf elasticsearch-7.5.0-linux-x86_64.tar.gz

[root@Linux7 soft]# mv elasticsearch-7.5.0 /usr/local/elasticsearch7.5

[root@Linux7 soft]# chown -R elkuser.elkuser /usr/local/elasticsearch7.5

elasticsearch 参数配置：

[elkuser@Linux7 elasticsearch7.5]$ vi config/elasticsearch.yml

network.host: 192.168.8.120
cluster.initial_master_nodes: ["node-1"]

elasticsearch 启动：

[root@Linux7 elasticsearch7.5]# su - elkuser
[elkuser@Linux7 ~]$ cd /usr/local/elasticsearch7.5/

[elkuser@Linux7 elasticsearch7.5]$ bin/elasticsearch

elasticsearch 启动状态查看：

[2019-12-13T11:03:45,087][INFO ][o.e.h.AbstractHttpServerTransport] [Linux7] publish_address {10.231.35.131:9200}, bound_addresses {10.231.35.131:9200}
[2019-12-13T11:03:45,088][INFO ][o.e.n.Node               ] [Linux7] started
[2019-12-13T11:03:45,336][INFO ][o.e.l.LicenseService     ] [Linux7] license [149d87ef-d8ca-498a-aef8-5f79f95df570] mode [basic] - valid
[2019-12-13T11:03:45,337][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [Linux7] Active license is now [BASIC]; Security is disabled
[2019-12-13T11:03:45,353][INFO ][o.e.g.GatewayService     ] [Linux7] recovered [0] indices into cluster_state

elasticsearch 访问：

浏览器地址栏输入：192.168.8.120:9200 

7.kibana安装

[root@Linux7 soft]# tar -xzvf kibana-7.5.0-linux-x86_64.tar.gz
[root@Linux7 soft]# mv kibana-7.5.0-linux-x86_64 /usr/local/kibana7.5
[root@Linux7 soft]# chown -R elkuser:elkuser /usr/local/kibana7.5/

kibana配置修改：

[elkuser@Linux7 kibana7.5]$ vi config/kibana.yml

server.host: "192.168.8.120"
elasticsearch.hosts: ["http://192.168.8.120:9200"]

kibana启动：

[root@Linux7 soft]# su - elkuser
[elkuser@Linux7 kibana7.5]$ bin/kibana

kibana访问：

http://192.168.8.120:5601

三、总结

1.安装elastic stack的需要jdk1.8版本环境

2.运行elastic stack的用户需要非root

如果是不root模式运行，报错信息如下

[root@Linux7 elasticsearch7.5]# bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-12-13T10:34:41,302][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [Linux7] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.5.0.jar:7.5.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.5.0.jar:7.5.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.0.jar:7.5.0]
        ... 6 more

3.elastic stack分为生产（配置network.host为非localhost）和非生产模式，其中生产模式要对运行环境做检测，比如用户的会话限制、cluster.initail_master_nodes等参数。

修改config/elasticsearch.yml配置文件network.host为非localhost模式后，启动报错信息如下

ERROR: [2] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-12-13T10:46:44,400][INFO ][o.e.n.Node               ] [Linux7] stopping ...
[2019-12-13T10:46:44,427][INFO ][o.e.n.Node               ] [Linux7] stopped
[2019-12-13T10:46:44,428][INFO ][o.e.n.Node               ] [Linux7] closing ...
[2019-12-13T10:46:44,471][INFO ][o.e.n.Node               ] [Linux7] closed
[2019-12-13T10:46:44,476][INFO ][o.e.x.m.p.NativeController] [Linux7] Native controller process has stopped - no new native processes can be started