Kafka+elk安装描述
1.安装elasticsearch
(1)tar -zxvf elasticsearch-7.6.0-linux-x86_64.tar.gz
(2)进入es根目录
(3)修改配置文件
node.name: node-1
network.host: 172.17.100.58
http.port: 9200
discovery.seed_hosts: ["172.17.100.58"]
cluster.initial_master_nodes: ["node-1"]
(4)添加用户并授权 es无法使用root用户启动
adduser es
passwd es
chown -R es:es /usr/local/elk/elasticsearch-7.6.0
(5)修改服务器配置
-----虚拟内存临时设置
sysctl -w vm.max_map_count=262144
-----文件描述符修改
vim /etc/security/limits.conf
添加
es hard nofile 65536
es soft nofile 65536u
-----使用es用户查看是否生效
ulimit -n
(6)切换用户并启动
su es
cd /usr/local/elk/elasticsearch-7.6.0
./bin/elasticsearch -d
2.安装kibana
(1)解压
(2)修改配置文件 elasticsearch.hosts: ["http://172.17.100.58:9200"]
(3)为es用户授权
chown -R es:es /usr/local/elk/kibana-7.6.0-linux-x86_64
(4)切换用户并启动
su es
./bin/kibana &
3.安装logstash
(1)解压
(2)添加配置文件test.conf到config目录下
-----文件内容
input {
#file {
#path => ["/mydata/logs/ejx/biz.log.2021-06-25.0.log"]
#start_position =>"beginning"
#}
kafka{
bootstrap_servers=> "172.17.100.57:9092"
topics => ["test-logs-mylogs"]
codec => "json"
}
}
output {
elasticsearch {
hosts => ["http://172.17.100.58:9200"]
index => "mylog-%{+YYYY.MM.dd}"
}
}
(3)使用root用户后台启动
./bin/logstash -f ./config/test.conf &
4.安装kafka（使用自带的zk）
(1)解压
(2)修改配置文件
server.properties
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://172.17.100.57:9092
zookeeper.connect=172.17.100.57:2181
zookeeper.properties
clientPort=2181
(3) 启动
先启动zk
./zookeeper-server-start.sh -daemon ../config/zookeeper.properties
./kafka-server-start.sh -daemon ../config/server.properties