kafka集群 zookeeper集群

kafka + zookeeper 集群

一、服务规划

系统：CentOS Linux release 7.8.2003 
版本：redis-5.0.10
地址清单：
176.16.2.11
176.16.2.12
176.16.2.13

zookeeper 信息

配置信息：
port:2181
server.1=176.16.2.11:2888:3888
server.2=176.16.2.12:2888:3888
server.3=176.16.2.13:2888:3888

配置信息 （注意 zk 集群每个节点的myid标识分别为 1，2，3）

zookeeper 集群标识：
# vim zookeeper/data/myid 
1

zookeeper 配置信息：
# vim zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/services/zookeeper/data
clientPort=2181
server.1=176.16.2.11:2888:3888
server.2=176.16.2.12:2888:3888
server.3=176.16.2.13:2888:3888

启动：
cd zookeeper/bin/ && ./zkServer.sh restart

kafka 信息

配置目录：kafka/config
数据目录：/tmp/zookeeper
端口：9092

配置文件：（server.properties 中 broker.id 标识分别为1，2，3）

# vim config/producer.properties
bootstrap.servers=176.16.2.11:9092,176.16.2.12:9092,176.16.2.13:9092

# vim config/consumer.properties
bootstrap.servers=176.16.2.11:9092,176.16.2.12:9092,176.16.2.13:9092
group.id=test-consumer-group

# cat config/zookeeper.properties
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false

kakfa节点1：server.properties

# cat config/server.properties
broker.id=1
listeners=PLAINTEXT://176.16.2.11:9092
advertised.listeners=PLAINTEXT://176.16.2.11:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/logs/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=3
offsets.topic.replication.factor=3
# 默认是1  集群副本及更改为3
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=176.16.2.11:2181,176.16.2.12:2181,176.16.2.13:2181
zookeeper.connection.timeout.ms=18000

kafka节点2：server.properties

# cat config/server.properties
broker.id=2
listeners=PLAINTEXT://176.16.2.12:9092
advertised.listeners=PLAINTEXT://176.16.2.12:9092
log.dirs=/opt/logs/kafka-logs
zookeeper.connect=176.16.2.11:2181,176.16.2.12:2181,176.16.2.13:2181
zookeeper.connection.timeout.ms=18000

kafka节点3：server.properties

# cat config/server.properties
broker.id=3
listeners=PLAINTEXT://176.16.2.13:9092
advertised.listeners=PLAINTEXT://176.16.2.13:9092
log.dirs=/opt/logs/kafka-logs
zookeeper.connect=176.16.2.11:2181,176.16.2.12:2181,176.16.2.13:2181
zookeeper.connection.timeout.ms=18000

启动：

nohup kafka/bin/kafka-server-start.sh kafka/config/server.properties &