部署elasticsearch（三节点）集群+filebeat+kibana

1. 用途
   ▷ 通过各个beat实时收集日志、传输至elasticsearch集群
   ▷ 通过kibana展示日志

2. 实验架构
   名称：IP地址：CPU：内存
   kibana&cerebro：192.168.75.20：1核：2G
   es-1：192.168.75.21：2核：4G
   es-2：192.168.75.22：2核：4G
   es-3：192.168.75.23：2核：4G

各beat均安装在ES集群上，也就是三个es主机节点

3. 软件安装

• 版本说明：
  均为7.3.0版本
• 注意事项
  各组件版本必须一致，elasticsearch必须3台及其以上且总数量为单数
• 软件保存路径：/usr/local/src
• 采用rpm文件方式安装
• 官方下载地址：https://www.elastic.co/cn/downloads/past-releases

3.1 elasticsearch

3.1.1 安装

3台es均执行相同的安装步骤

echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

cd /usr/local/src
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.0-x86_64.rpm
rpm -vi elasticsearch-7.3.0-x86_64.rpm

默认配置文件路径：/etc/elasticsearch/
默认程序安装路径：/usr/share/elasticsearch/
默认存储文件路径：/var/lib/elasticsearch/
默认日志文件路径：/var/log/elasticsearch/

3.1.2 配置

# es-1主机

# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'

# 集群名字
cluster.name: my-application

# 节点名称
node.name: 192.168.75.21

# 数据路径
path.data: /var/lib/elasticsearch

# 日志路径
path.logs: /var/log/elasticsearch

# 本界面访问IP
network.host: 192.168.75.21

# 本届点访问端口
http.port: 9200

# 节点运输端口
transport.port: 9300

# 集群中其他主机列表
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]

# 首次启动全新的Elasticsearch集群时，在第一次选举中便对其票数进行计数的master节点的集合
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]

# 启用跨域资源共享
http.cors.enabled: true
http.cors.allow-origin: "*"

# es-2主机
# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'
cluster.name: my-application
node.name: 192.168.75.22
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.75.22
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# es-3主机
# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'
cluster.name: my-application
node.name: 192.168.75.23
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.75.23
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
http.cors.enabled: true
http.cors.allow-origin: "*"

3.1.3 启动/重启/停止

systemctl start elasticsearch
systemctl stop elasticsearch
systemctl restart elasticsearch
systemctl status elasticsearch

3.1.4 查看集群健康、节点状态

# 注：第一个IP为集群中任一节点ip,第二个ip为界面名称

# elasticsearch启动后查看是否启动成功
curl -XGET "http://192.168.75.21:9200/_cluster/health?pretty=true"

# 停止elasticsearch应用
curl -XPOST "http://192.168.75.21:9200/_shutdown"

# 查看集群健康
curl 192.168.75.21:9200/_cluster/health?pretty

# 检查集群状态
curl 192.168.75.21:9200/_cluster/stats?pretty

# 节点状态
curl 192.168.75.21:9200/_nodes/process?pretty

curl 192.168.75.21:9200/_nodes/192.168.75.21/process?pretty

# 当你不知道有那些属性可以查看时,会返回可以查看的属性
curl '192.168.75.21:9200/_cat/'

3.2 kibana

3.2.1 安装

cd /usr/local/src
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.3.0-x86_64.rpm
rpm -vi kibana-7.3.0-x86_64.rpm

默认配置文件路径：/etc/kibana/
默认程序安装路径：/usr/share/kibana/

3.2.3 配置

# cat /etc/kibana/kibana.yml| grep -v '^#'

# 访问端口号
server.port: 5601
# 访问地址
server.host: "192.168.75.20"
# 名称
server.name: "192.168.75.20"
# es集群地址
elasticsearch.hosts: ["http://192.168.75.21:9200", "http://192.168.75.22:9200","http://192.168.75.23:9200"]
# 中文界面
i18n.locale: "zh-CN"

3.2.3 启动/重启/停止

systemctl start kibana
systemctl stop kibana
systemctl restart kibana
systemctl status kibana

3.2.4 浏览器访问
地址： http://192.168.75.20:5601

3.3 cerebro

官方地址：https://github.com/lmenezes/cerebro
下载地址：https://github.com/lmenezes/cerebro/releases

前提条件：需要有java环境

wget https://github.com/lmenezes/cerebro/releases/download/v0.8.5/cerebro-0.8.5.tgz
tar xzf cerebro-0.8.5.tgz -C /usr/local


# 配置ES服务器
# 非必须：如果经常使用的话，可以先在conf/application.conf中配置好ElasticSearch服务器地址
# 第一个是不加密的，第二个是加密的
hosts = [
  {
    host = "http://192.168.75.21:9200" # 设置集群中的一个地址
    name = "Localhost cluster"
    headers-whitelist = [ "x-proxy-user", "x-proxy-roles", "X-Forwarded-For" ]
  }
  # Example of host with authentication
  # {
  #  host = "http://172.17.107.187:9203"
  #  name = "my-application"
  #  auth = {
  #    username = "elastic"
  #    password = "escluter123456"
  #  }
  # }
]


# 启动,默认使用9000端口
cerebro-0.8.1/bin/cerebro
[info] play.api.Play - Application started (Prod)
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

# 指定端口启动
bin/cerebro -Dhttp.port=8080

# 指定地址启动
bin/cerebro -Dhttp.address=192.168.75.20

# 使用浏览器访问http://192.168.75.20:9000

3.4 Filebeat

在三台es主机节点上进行安装,收集elasticsearch的日志，默认索引为filebeat-7.3.0-*
Filebeat 模块 elasticsearch 解析 Elasticsearch 创建的日志

# 下载并安装 Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.0-x86_64.rpm
rpm -vi filebeat-7.3.0-x86_64.rpm

# 修改 /etc/filebeat/filebeat.yml 以设置连接信息：

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 启用和配置 elasticsearch 模块,在 /etc/filebeat/modules.d/elasticsearch.yml 文件中修改设置
filebeat modules enable elasticsearch

# 启动 Filebeat
filebeat setup
service filebeat start

3.5 Metricbeat

在三台es主机节点上进行安装,收集es主机的日志，默认索引为metricbeat-7.3.0-*

# 下载并安装 metricbeat
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.3.0-x86_64.rpm
rpm -vi metricbeat-7.3.0-x86_64.rpm

# 修改 /etc/metricbeat/metricbeat.yml 以设置连接信息：

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 启用和配置 system 模块，在 /etc/metricbeat/modules.d/system.yml 文件中修改设置
metricbeat modules enable system

# 启动 Metricbeat
metricbeat setup
service metricbeat start

3.6 Heartbeat

在三台es主机节点上进行安装,通过主动探测来监测服务的可用性，默认索引为heartbeat-7.3.0-*

# 下载并安装 heartbeat
curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-7.3.0-x86_64.rpm
rpm -vi heartbeat-7.3.0-x86_64.rpm

# 修改 /etc/heartbeat/heartbeat.yml 以设置连接信息

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 添加监测,注意每台es节点主机ip不同，其中 <http://192.168.75.21:9200> 是受监测 URL
heartbeat.monitors:
- type: http
  urls: ["http://192.168.75.21:9200"] 
  schedule: "@every 10s"


# 启动 Heartbeat
heartbeat setup
service heartbeat-elastic start