zoukankan      html  css  js  c++  java
  • Kafka and ZooKeeper

    Introduction

    Apache Kafka® is a distributed streaming platform. What exactly does that mean?

    We think of a streaming platform as having three key capabilities:

    1. It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
    2. It lets you store streams of records in a fault-tolerant way.
    3. It lets you process streams of records as they occur.

    What is Kafka good for?

    It gets used for two broad classes of application:

    1. Building real-time streaming data pipelines that reliably get data between systems or applications
    2. Building real-time streaming applications that transform or react to the streams of data

    To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up.

    First a few concepts:

    • Kafka is run as a cluster on one or more servers.
    • The Kafka cluster stores streams of records in categories called topics.
    • Each record consists of a key, a value, and a timestamp.

    Kafka has four core APIs:

    • The Producer API allows an application to publish a stream of records to one or more Kafka topics.
    • The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
    • The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
    • The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.

    In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages.

    ZooKeeper: Because Coordinating Distributed Systems is a Zoo

     

    ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.

    The following documents describe concepts and procedures to get you started using ZooKeeper. If you have more questions, please ask the mailing list or browse the archives.

    • ZooKeeper Overview

      Technical Overview Documents for Client Developers, Adminstrators, and Contributors

      • Overview - a bird's eye view of ZooKeeper, including design concepts and architecture
      • Getting Started - a tutorial-style guide for developers to install, run, and program to ZooKeeper
      • Release Notes - new developer and user facing features, improvements, and incompatibilities
    • Developers

      Documents for Developers using the ZooKeeper Client API

    • Administrators & Operators

      Documents for Administrators and Operations Engineers of ZooKeeper Deployments

      • Administrator's Guide - a guide for system administrators and anyone else who might deploy ZooKeeper
      • Quota Guide - a guide for system administrators on Quotas in ZooKeeper. 
      • JMX - how to enable JMX in ZooKeeper
      • Hierarchical quorums
      • Observers - non-voting ensemble members that easily improve ZooKeeper's scalability
    • Contributors

      Documents for Developers Contributing to the ZooKeeper Open Source Project

    • Miscellaneous ZooKeeper Documentation
    • BookKeeper Documentation

      BookKeeper is a highly-available system that implements high-performance write-ahead logging. It uses ZooKeeper for metadata, which is the main reason for being a ZooKeeper contrib.

    Ref:Kafka

    Ref:ZooKeeper

    Ref:Apache Kafka:下一代分布式消息系统

    Ref:Kafka设计解析

    Ref:ZooKeeper应用案例

    Ref:ZooKeeper架构设计及其应用要点

    Ref:ZooKeeper-3.3.4集群安装配置

    Ref:Zookeeper学习系列

    Ref:记一次JavaWeb网站技术架构总结

  • 相关阅读:
    修复TabControl在Binding情况下Canvas被复用的问题
    避免缓加载时因违反惯例导致的空引用!
    乱说一气
    WPF中的数据验证
    [zz]GPU architecture
    [zz]DirectX 11 and Shared Model 5.0
    网页栅格系统中的最佳宽度:960px
    复习html标签及其属性
    去除链接虚线边框css
    使用jquery解决IE6不兼容的伪类
  • 原文地址:https://www.cnblogs.com/ncore/p/6878874.html
Copyright © 2011-2022 走看看