docker kafka 单节点及集群部署

docker kafka 单节点

宿主机IP: 192.168.43.177

1.下载镜像
这里使用了wurstmeister/kafka和wurstmeister/zookeeper这两个版本的镜像
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
在命令中运行docker images验证两个镜像已经安装完毕
2.启动
启动zookeeper容器
docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper

启动kafka容器
docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.43.177:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.43.177:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka

3.进入容器操作：
cd /opt/kafka_2.12-2.3.0/bin
创建主题
sh kafka-topics.sh --create --zookeeper 192.168.43.177:2181 --replication-factor 1 --partitions 2 --topic test
删除主题【需要在server.properties中设置delete.topic.enable=true】
sh kafka-topics.sh --delete --zookeeper 192.168.43.177:2181 --topic test
查看主题：
sh kafka-topics.sh --list --zookeeper 192.168.43.177:2181
sh kafka-topics.sh --describe --zookeeper 192.168.43.177:2181 --topic test
生产消息：
sh kafka-console-producer.sh --broker-list 192.168.43.177:9092 --topic test
消费消息：
sh kafka-console-consumer.sh --bootstrap-server 192.168.43.177:9092 --from-beginning --topic test
sh kafka-consol e-consumer.sh --bootstrap-server 192.168.43.177:9092 --topic test

--------------------------------------------------------------------------------

docker kafka 集群部署

宿主机IP: 192.168.43.177

1.下载镜像
这里使用了wurstmeister/kafka和zookeeper这两个版本的镜像
docker pull zookeeper
docker pull wurstmeister/kafka
在命令中运行docker images验证两个镜像已经安装完毕
2.编写docker-kafka.yml文件，内容如下：

version: '2'
services:
zoo1:
image: zookeeper
hostname: zoo1
container_name: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

zoo2:
image: zookeeper
hostname: zoo2
container_name: zoo2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181

zoo3:
image: zookeeper
hostname: zoo3
container_name: zoo3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181


kafka1:
image: wurstmeister/kafka
hostname: kafka1
container_name: kafka1
ports:
- "9091:9091"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 1
KAFKA_HOST_NAME: kafka1
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka1:9091
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9091
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3

kafka2:
image: wurstmeister/kafka
hostname: kafka2
container_name: kafka2
ports:
- "9092:9092"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 2
KAFKA_HOST_NAME: kafka2
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka2:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3

kafka3:
image: wurstmeister/kafka
hostname: kafka3
container_name: kafka3
ports:
- "9093:9093"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 3
KAFKA_HOST_NAME: kafka3
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka3:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9093
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3

3.创建kafka集群，宿主机docker-kafka.yml文件目录下执行
docker-compose -f docker-kafka.yml up -d

4.修改宿主机或其他客户端主机hosts配置，增加以下配置
192.168.43.177 kafka1
192.168.43.177 kafka2
192.168.43.177 kafka3

相关命令：
停止容器：
docker stop zoo1 zoo2 zoo3 kafka1 kafka2 kafka3
删除容器：
docker rm zoo1 zoo2 zoo3 kafka1 kafka2 kafka3
创建主题：
sh kafka-topics.sh --create --zookeeper 192.168.43.177:2181,192.168.43.177:2182,192.168.43.177:2183 --replication-factor 1 --partitions 2 --topic test
查看主题：
sh kafka-topics.sh --list --zookeeper 192.168.43.177:2181,192.168.43.177:2182,192.168.43.177:2183
sh kafka-topics.sh --describe --zookeeper 192.168.43.177:2181,192.168.43.177:2182,192.168.43.177:2183 --topic test
生产消息：
sh kafka-console-producer.sh --broker-list 192.168.43.177:9091,192.168.43.177:9092,192.168.43.177:9093 --topic test
消费消息：
sh kafka-console-consumer.sh --bootstrap-server 192.168.43.177:9091,192.168.43.177:9092,192.168.43.177:9093 --from-beginning --topic test

问题解决：
1、在windows中使用docker-compse运行compose-docker.yml遇到问题。Mount denied: The source path "\\var\\run\\docker.sock:/var/run/docker.sock" is not a valid Windows path
解决方法：设置环境变量 COMPOSE_CONVERT_WINDOWS_PATHS=1

2、java.io.IOException: Can't resolve address: kafka1:9091
解决方法：修改宿主机或其他客户端主机hosts配置，增加以下配置

3、[2019-12-04 17:37:00,848] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-04 17:37:00,953] ERROR Error when sending message to topic test with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
解决方法：修改容器内kafka相关端口，kafka1示例：
ports:
- "9091:9091"
KAFKA_LISTENERS: PLAINTEXT://kafka1:9091
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9091