zoukankan      html  css  js  c++  java
  • 基于docker搭建zookeeper集群、kafka集群---二(多台真机之间的集群)

    三台真机,容器采用host

    192.168.0.128  192.168.0.141 192.168.0.142

    zookeeper集群搭建

    在每台机上执行脚本,

    #!/bin/bash
    #Get zookeeper image
    zkimage=`docker images | grep zookeeper | awk {'print $1'}`
    if [ -n "$zkimage" ]
    then
        echo 'The zookeeper image is already existed.'
    else
        echo 'Pull the latest zookeeper image.'
        docker pull zookeeper
    fi
    
    #Create network for zookeeper containers
    zknet=`docker network ls | grep yapi_net | awk {'print $2'}`
    if [ -n "$zknet" ]
    then
        echo 'The zknetwork is already existed.'
    else
        echo 'Create zknetwork.'
        docker network create --subnet 172.30.0.0/16 yapi_net
    fi
    
    #Start zookeeper cluster
    echo 'Start 3 zookeeper servers.'
    rm -rf   /opt/zookeeper_1/data  /opt/zookeeper_1/datalog  /var/log/zookeeper_1/log
    rm -rf   /opt/zookeeper_2/data  /opt/zookeeper_2/datalog  /var/log/zookeeper_2/log
    rm -rf   /opt/zookeeper_3/data  /opt/zookeeper_3/datalog  /var/log/zookeeper_3/log
    
    mkdir -p  /opt/zookeeper_1/data  /opt/zookeeper_1/datalog  /var/log/zookeeper_1/log
    mkdir -p  /opt/zookeeper_2/data  /opt/zookeeper_2/datalog  /var/log/zookeeper_2/log
    mkdir -p  /opt/zookeeper_3/data  /opt/zookeeper_3/datalog  /var/log/zookeeper_3/log

    ZOO_SERVERS="server.1=192.168.0.128:2888:3888 server.2=192.168.0.142:2888:3888 server.3=192.168.0.141:2888:3888"
    #新版 ZOO_SERVERS="clientPort=2181  server.1=192.168.0.146:2888:3888 server.2=192.168.0.145:2888:3888"
    docker run --network host -d --restart always -v /opt/zookeeper_1/data:/data -v /opt/zookeeper_1/datalog:/datalog -v /var/log/zookeeper_1/log:/logs -e ZOO_SERVERS="$ZOO_SERVERS" -e ZOO_MY_ID=1 --name zookeeper_1 -p 2182:2181 -p 2888:2888 -p 3888:3888 docker.io/zookeeper

    注意每台机的id号不同

    官方的镜像有可能配置的环境变量不同,需要修改脚本,参考Entrypoint文件,和配置文件,配置文件如下:

    clientPort=2181
    dataDir=/data
    dataLogDir=/datalog
    tickTime=2000
    initLimit=5
    syncLimit=2
    autopurge.snapRetainCount=3
    autopurge.purgeInterval=0
    maxClientCnxns=60
    server.1=192.168.0.128:2888:3888
    server.2=192.168.0.142:2888:3888
    server.3=192.168.0.141:2888:3888

    单机(非集群)的配置文件,telnet  ip  2181 = 》stat ==》Mode: standalone

    telnet  新版的不能验证stat,访问http://192.168.0.145:8080/commands/stat,可验证,参考docker hub的zookeeper镜像

    http://192.168.1.129:8080/commands/mntr  #查看集权信息

    clientPort=2181
    dataDir=/data
    dataLogDir=/datalog
    tickTime=2000
    initLimit=5
    syncLimit=2
    autopurge.snapRetainCount=3
    autopurge.purgeInterval=0
    maxClientCnxns=60

    kafka集群

    在每台机上执行脚本,

    #!/bin/bash
    #Get zookeeper image
    kfkimage=`docker images | grep 'docker.io/wurstmeister/kafka' | awk {'print $1'}`
    if [ -n "$kfkimage" ]
    then
        echo 'The docker.io/wurstmeister/kafka is already existed.'
    else
        echo 'Pull the image.'
        docker pull docker.io/wurstmeister/kafka
    fi
    
    #Create network for zookeeper containers
    kfknet=`docker network ls | grep yapi_net | awk {'print $2'}`
    if [ -n "$kfknet" ]
    then
        echo 'The kfknetwork is already existed.'
    else
        echo 'Create kfknetwork.'
        docker network create --subnet 172.30.0.0/16 yapi_net
    fi
    
    #Start 3  zookeeper cluster
    echo 'Start 3 kafka servers.'
    
    rm -rf  /opt/kafka_1/logdata
    rm -rf  /opt/kafka_2/logdata
    rm -rf  /opt/kafka_3/logdata
    
    mkdir -p  /opt/kafka_1/logdata
    mkdir -p  /opt/kafka_2/logdata
    mkdir -p  /opt/kafka_3/logdata
    
    #kafka ip
    kfk_1_ip='172.30.0.41'
    kfk_2_ip='172.30.0.42'
    kfk_3_ip='172.30.0.43'
    #zk_jiqun_ip='172.30.0.31:2181'
    zk_jiqun_ip='192.168.0.128:2181,192.168.0.142:2181,192.168.0.141:2181'
    docker run --restart always -d --name kafka_1 --network host -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='141' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://192.168.0.141:9092' -v /opt/kafka_1/logdata:/kafka -p 9092:9092  docker.io/wurstmeister/kafka

    注意id号  和PLAINTEXT://192.168.0.141:9092,(创建容器后,再进入容器修改,restart后会重置,所以必须在创建容器时,就要指定好环境变量。因为启动脚本会重新修改配置文件,所以要想修改环境变量,只需要在启动脚本里加一个即可)

    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    补充:单机docker run -d --restart always -v /opt/zookeeper/data:/data -v /opt/zookeeper/datalog:/datalog -v /var/log/zookeeper/log:/logs   --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 8081:8080 docker.io/zookeeper

    docker run --restart always -d --name kafka -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='163' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.18.72.163:9092' -v /opt/kafka/logdata:/kafka -p 9092:9092  docker.io/wurstmeister/kafka

    ---------------------------------------------------------------------------------------------------------

    补充2:云上部署,不同主机节点,zookeeper采用host网络:因为采用容器网络,zookeeper不同主机节点之间通信,找不到容器网络,

                 不同主机节点,kafka采用host网络,单参数KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.18.72.163:9092'  ,发消息时,会通过配置文件PLAINTEXT://172.18.72.163:9092,找kafka集群的节点,线上配置私网即可,因为,服务之间可以通过私网访问,如果本地需要用来访问kafka时,就需要要配置公网ip:因为远程,如果是私网,就会连接超时,

    test是测kafka

    ping是测zookeeper

     

     kafka连接zookeeper超时

    [2019-01-02 14:41:03,450] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
    kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
        at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:75)
        at kafka.Kafka.main(Kafka.scala)

    在各个节点上hosts文件,添加对应ip即可

  • 相关阅读:
    HDU 1800 Flying to the Mars 字典树,STL中的map ,哈希树
    字典树 HDU 1075 What Are You Talking About
    字典树 HDU 1251 统计难题
    最小生成树prim算法 POJ2031
    POJ 1287 Networking 最小生成树
    次小生成树 POJ 2728
    最短路N题Tram SPFA
    poj2236 并查集
    POJ 1611并查集
    Number Sequence
  • 原文地址:https://www.cnblogs.com/fanever/p/10979036.html
Copyright © 2011-2022 走看看