zoukankan      html  css  js  c++  java
  • kafka 常用命令

    Kafka


    解压,进入kafka目录下的bin目录


    启动zk

    nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &

    nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &


    启动kafka

    nohup ./kafka-server-start.sh ../config/server.properties &

    nohup ./bin/kafka-server-start.sh ./config/server.properties &


    创建topic

    ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

    ./kafka-topics.sh --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test2 --create


    列出topic

    ./kafka-topics.sh --list --zookeeper localhost:2181

    ./kafka-topic --zookeeper localhost:2181 --list


    启动生产者并发送消息

    ./kafka-console-producer.sh --broker-list localhost:9092 --topic zyl

    输入消息

    nihao

    china


    另外开个终端,启动消费者接受消息

    ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic zyl --from-beginning

    可以接受到

    nihao

    china



    另外开个终端,可以看到有4个进程(分别为zk、kafka、控制台消费者、控制台生产者)

    [root@aa bin]# jps | grep -v Jps

    9802 QuorumPeerMain

    10392 Kafka

    10956 ConsoleConsumer

    10887 ConsoleProducer



    使用集群3个节点(多个节点也同理,加上之前的,我一共弄了4个节点)

    拷贝并修改配置文件

    cp server.properties server1.properties

    cp server.properties server2.properties

    cp server.properties server3.properties


    nohup ./kafka-server-start.sh ../config/server1.properties &

    nohup ./kafka-server-start.sh ../config/server2.properties &

    nohup ./kafka-server-start.sh ../config/server3.properties &


    [root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic(注意,指定的是3)

    Created topic "my-replicated-topic".

    [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

    Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

    Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0



    [root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic

    Error while executing topic command Topic "my-replicated-topic" already exists.

    kafka.common.TopicExistsException: Topic "my-replicated-topic" already exists.

    at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:171)

    at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:156)

    at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)

    at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)

    at kafka.admin.TopicCommand.main(TopicCommand.scala)


    [root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic2

    Created topic "my-replicated-topic2".

    [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

    Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

    Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0



    [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic zyl

    Topic:zyl PartitionCount:1 ReplicationFactor:1 Configs:

    Topic: zyl Partition: 0 Leader: 0 Replicas: 0 Isr: 0




    启动生产者并发送消息

    ./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic


    另外开个终端,启动消费者接受消息

    ./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic



    另外开个终端,可以看到有9个进程(分别为1个zk、4个kafka、2个控制台消费者、2个控制台生产者)

    [root@aa ~]# jps | grep -v Jps

    9802 QuorumPeerMain

    10392 Kafka

    11547 Kafka

    12007 ConsoleProducer

    10956 ConsoleConsumer

    10887 ConsoleProducer

    11469 Kafka

    12054 ConsoleConsumer

    11710 Kafka



    查看单独的进程:

    ps -ef | grep server.properties | grep -v grep

    ps -ef | grep "server1.properties" | grep -v grep



    测试leader

    杀掉进程,然后--describe查看

    [root@aaa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

    Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

    Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

    可以看到leader是2.

     

     [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

    Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

    Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0

    可以看到leader是2.


    [root@aa bin]# ps -ef | grep server1.properties | grep -v grep




    杀掉某个kafka进程

    kill -9 11469




     ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

    Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

    Topic: my-replicated-topic2 Partition: 0 Leader: 2 Replicas: 1,2,3,0 Isr: 2,3,0

    [root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

    Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

    Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

    可以看到my-replicated-topic2的leader变成了2。


    问题:

    1.zookeeper-shell.sh如何使用

    2.能不能通过kafka-server-stop.sh停止某个kafka呢











    Command must include exactly one action: --list, --describe, --create or --alter

    Option                                  Description

    ------                                  -----------

    --alter                                 Alter the configuration for the topic.

    --config <name=value>                   A topic configuration override for the

                                              topic being created or altered.

    --create                                Create a new topic.

    --deleteConfig <name>                   A topic configuration override to be

                                              removed for an existing topic

    --describe                              List details for the given topics.

    --help                                  Print usage information.

    --list                                  List all available topics.

    --partitions <Integer: # of partitions> The number of partitions for the topic

                                              being created or altered (WARNING:

                                              If partitions are increased for a

                                              topic that has a key, the partition

                                              logic or ordering of the messages

                                              will be affected

    --replica-assignment                    A list of manual partition-to-broker

      <broker_id_for_part1_replica1 :         assignments for the topic being

      broker_id_for_part1_replica2 ,          created or altered.

      broker_id_for_part2_replica1 :

      broker_id_for_part2_replica2 , ...>

    --replication-factor <Integer:          The replication factor for each

      replication factor>                     partition in the topic being created.

    --topic <topic>                         The topic to be create, alter or

                                              describe. Can also accept a regular

                                              expression except for --create option

    --topics-with-overrides                 if set when describing topics, only

                                              show topics that have overridden

                                              configs

    --unavailable-partitions                if set when describing topics, only

                                              show partitions whose leader is not

                                              available

    --under-replicated-partitions           if set when describing topics, only

                                              show under replicated partitions

    --zookeeper <urls>                      REQUIRED: The connection string for

                                              the zookeeper connection in the form

                                              host:port. Multiple URLS can be

                                              given to allow fail-over.

  • 相关阅读:
    [国家集训队]墨墨的等式(同余最短路)
    [洛谷P2575]高手过招
    [CSP校内集训]rank
    杀人游戏(tarjan思维好题)
    骑士游戏(spfa好题)
    机房模拟测试4:计数类dp+水题+树上计数
    机房测试模拟2:模拟+数学+数位dp
    机房测试11:最小生成树(最小生成树+二分)
    机房测试模拟1(day2):矩阵+树上贪心+bfs+状压
    机房测试16:字符串专题(AC自动机+dp+kmp)
  • 原文地址:https://www.cnblogs.com/jing1617/p/7449223.html
Copyright © 2011-2022 走看看