ELK集群

kafka集群
-------------------
1. 下载
wget http://mirror.rise.ph/apache/kafka/0.11.0.0/kafka_2.12-0.11.0.0.tgz
2. 安装
tar xf kafka_2.12-0.11.0.0.tgz
mv kafka_2.12-0.11.0.0 /data/elk/kafka
3. 配置文件
3.1 节点1
#cat config/zookeeper.propertie
dataDir=/data/zookeeper
maxClientCnxns=0
tickTime=2000
initLimit=20
syncLimit=10
server.1=59.188.25.xxxx:2888:3888
server.2=59.188.25.xxxx:2888:3888
#mkdir /data/zookeeper
#echo "1" >/data/zookeeper/myid
#cat config/server.properties
broker.id=1
host.name=59.188.25.xxxx
log.dirs=/data/logs/kafka-logs
zookeeper.connect=59.188.25.xxxx:2181,59.188.25.xxxx:2181
num.partitions=16
log.retention.hours=24

3.1 节点2
#cat config/zookeeper.propertie
dataDir=/data/zookeeper
maxClientCnxns=0
tickTime=2000
initLimit=20
syncLimit=10
server.1=59.188.25.xxxx:2888:3888
server.2=59.188.25.xxxx:2888:3888
#mkdir /data/zookeeper
#echo "1" >/data/zookeeper/myid
#cat config/server.properties
broker.id=1
host.name=59.188.25.xxxx
log.dirs=/data/logs/kafka-logs
zookeeper.connect=59.188.25.xxxx:2181,59.188.25.226:xxxx
num.partitions=16
log.retention.hours=24

logstash多实例
-------------------
开启多个logstash实例，根据每个topic的大小，适当安排每个实例处理的topic

elasticsearch集群
---------------------
配置相同的集群名称，配置discovery.zen.ping.unicast.hosts: