Kafka and ZooKeeper

Introduction

Apache Kafka® is a distributed streaming platform. What exactly does that mean?

We think of a streaming platform as having three key capabilities:

1. It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
2. It lets you store streams of records in a fault-tolerant way.
3. It lets you process streams of records as they occur.

What is Kafka good for?

It gets used for two broad classes of application:

1. Building real-time streaming data pipelines that reliably get data between systems or applications
2. Building real-time streaming applications that transform or react to the streams of data

To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up.

First a few concepts:

• Kafka is run as a cluster on one or more servers.
• The Kafka cluster stores streams of records in categories called topics.
• Each record consists of a key, a value, and a timestamp.

Kafka has four core APIs:

• The Producer API allows an application to publish a stream of records to one or more Kafka topics.
• The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
• The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
• The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.

In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages.

ZooKeeper: Because Coordinating Distributed Systems is a Zoo

 

ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.

The following documents describe concepts and procedures to get you started using ZooKeeper. If you have more questions, please ask the mailing list or browse the archives.

• ZooKeeper Overview

  Technical Overview Documents for Client Developers, Adminstrators, and Contributors

  • Overview - a bird's eye view of ZooKeeper, including design concepts and architecture
  • Getting Started - a tutorial-style guide for developers to install, run, and program to ZooKeeper
  • Release Notes - new developer and user facing features, improvements, and incompatibilities
• Developers

  Documents for Developers using the ZooKeeper Client API

  • API Docs - the technical reference to ZooKeeper Client APIs
  • Programmer's Guide - a client application developer's guide to ZooKeeper
  • ZooKeeper Java Example - a simple Zookeeper client appplication, written in Java
  • Barrier and Queue Tutorial - sample implementations of barriers and queues
  • ZooKeeper Recipes - higher level solutions to common problems in distributed applications
• Administrators & Operators

  Documents for Administrators and Operations Engineers of ZooKeeper Deployments

  • Administrator's Guide - a guide for system administrators and anyone else who might deploy ZooKeeper
  • Quota Guide - a guide for system administrators on Quotas in ZooKeeper. 
  • JMX - how to enable JMX in ZooKeeper
  • Hierarchical quorums
  • Observers - non-voting ensemble members that easily improve ZooKeeper's scalability
• Contributors

  Documents for Developers Contributing to the ZooKeeper Open Source Project

  • ZooKeeper Internals - assorted topics on the inner workings of ZooKeeper
• Miscellaneous ZooKeeper Documentation
  • Wiki
  • FAQ
• BookKeeper Documentation

  BookKeeper is a highly-available system that implements high-performance write-ahead logging. It uses ZooKeeper for metadata, which is the main reason for being a ZooKeeper contrib.

  • henn, what's it again?
  • Ok, now how do I try it out
  • Awesome, but how do I integrate it with my app?
  • Can I stream bytes instead of entries?

Ref:Kafka

Ref:ZooKeeper

Ref:Apache Kafka：下一代分布式消息系统

Ref:Kafka设计解析

Ref:ZooKeeper应用案例

Ref:ZooKeeper架构设计及其应用要点

Ref:ZooKeeper-3.3.4集群安装配置

Ref:Zookeeper学习系列

Ref:记一次JavaWeb网站技术架构总结