zoukankan      html  css  js  c++  java
  • Kafka Broker | Command-line Options and Procedure(转发)

    原文:

    https://data-flair.training/blogs/kafka-broker/

    In this Apache Kafka tutorial, we are going to learn Kafka Broker.

    Kafka Broker manages the storage of messages in the topic(s).

    If Apache Kafka has more than one broker, that is what we call a Kafka cluster.

    Moreover, in this Kafka Broker Tutorial, we will learn how to start Kafka Broker and Kafka command-line Option.


    So, let’s start Apache Kafka Broker.

    What is Kafka Broker?

    A Kafka broker is also known as Kafka server and a Kafka node. These all names are its synonyms.

    In simple words, a broker is a mediator between two. However, Kafka broker is more precisely described as a Message Broker which is responsible for mediating the conversation between different computer systems, guaranteeing delivery of the message to the correct parties.

    Read Apache Kafka Security | Need and Components of Kafka

    Hence, the Kafka cluster typically consists of multiple brokers.

    Kafka Cluster uses Zookeeper for maintaining the cluster state.

    A single Broker can handle thousands of reads and writes per second. Whereas, if there is no performance impact, each broker can handle TB of messages.

    In addition, to be very sure that ZooKeeper performs broker leader election.


    Kafka Broker

    Basically, a broker in Kafka is modeled as KafkaServer, which hosts topics.

    Here, given topics are always partitioned across brokers, in a cluster a single broker hosts topic partitions of one or more topics actually, even when a topic is only partitioned to just a single partition.

    In addition, producers send a message to a broker, after receiving broker stores them on disk keyed by unique offset.

    Moreover, by topic, partition and offset a broker allows consumers to fetch messages.

    However, brokers can create a cluster just by sharing information with each other directly or indirectly using Zookeeper.

    Also, we can say Kafka cluster has exactly one broker which acts as the Controller. By using kafka-server-start.sh script, we can start a single Kafka broker.

    How to Start Kafka Broker?

    • Start Zookeeper.
    1. ./bin/zookeeper-server-start.sh config/zookeeper.properties

    We can start a Kafka server, only when Zookeeper is up and running(that will connect to Zookeeper).

    1. ./bin/kafka-server-start.sh config/server.properties
    2. kafka-server-start.sh script

    Hence, kafka-server-start.sh starts a broker.

    1. $ ./bin/kafka-server-start.sh
    2. USAGE: ./bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*

    Note- make sure that Zookeeper is up and running before you run kafka-server-start.sh.

    Moreover, use zookeeper-server-start shell script.

    Have a look at Apache Kafka + Spark Streaming Integration

    kafka-server-start.sh uses config/log4j.properties for logging configuration that we can override using KAFKA_LOG4J_OPTS environment variable.

    1. KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:config/log4j.properties"

    kafka-server-start.sh accepts KAFKA_HEAP_OPTS and EXTRA_ARGS environment variables.

    Kafka Command-line Options

    -name

    Defaults to kafkaServer when in daemon mode.

    -loggc

    Enabled when in daemon mode.

    -daemon

    Enables daemon mode.

    –override property=value

    A value that should override the value set for property in server.properties file.

    Let’s revise Apache Kafka Streams | Stream Processing Topology

    1. $ ./bin/kafka-server-start.sh config/server.properties --override broker.id=100
    2. ...
    3. INFO [KafkaServer id=100] started (kafka.server.KafkaServer)

    Conclusion

    Hence, we have learned the complete Kafka Broker, along with commands to start it. However, if any doubt occurs, feel free to ask in the comment section.

    See also – 

    Kafka- serialization and deserialization

    For reference

  • 相关阅读:
    错误 : 资产文件“项目objproject.assets.json”没有“.NETCoreApp,Version=v2.0”的目标。确保已运行还原,且“netcoreapp2.0”已包含在项目的 TargetFrameworks 中。
    .NET core RSA帮助类
    vs2010单步调试崩溃
    常用排序算法比较与分析
    js原生的轮播图
    vue-cli脚手架使用-- 初学者
    vue-cli配置基础
    jq轮播图插件
    bootstrap 获得轮播中的索引或当前活动的焦点对象
    jQuery学习笔记之Ajax用法详解
  • 原文地址:https://www.cnblogs.com/panpanwelcome/p/13534341.html
Copyright © 2011-2022 走看看