1.kafka的基本原理
Kafka 架构原理解析
https://mp.weixin.qq.com/s?src=11×tamp=1632310110&ver=3330&signature=HXfuaoxorZiEZgAH3l0n0x6-Gcf3hTEQMOy0aOSGCCJm93UGIUUDP3KyrwodX3dWNQkQVYnmgFOFz314cw1-UWUMTHuvvfC01amVhar3E1GRWlVm9AjTWF2BRmGSDJ&new=1
2.kafka最新现状：
趋势：
kafka去zookeeper（zab）入quorum（raft）。
去Java8，支持更高Java版本。
Raft对比ZAB协议-阿里云开发者社区
https://developer.aliyun.com/article/62901
未来竞争对手：Apache Pulsar
Apache Kafka
https://kafka.apache.org/downloads
3.0.0
Released September 21, 2021
Release Notes
Source download: kafka-3.0.0-src.tgz (asc, sha512)
Binary downloads:
Scala 2.12 - kafka_2.12-3.0.0.tgz (asc, sha512)
Scala 2.13 - kafka_2.13-3.0.0.tgz (asc, sha512)
We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.13 is recommended).
Kafka 3.0.0 includes a number of significant new features. Here is a summary of some notable changes:
The deprecation of support for Java 8 and Scala 2.12
Kafka Raft support for snapshots of the metadata topic and other improvements in the self-managed quorum
Stronger delivery guarantees for the Kafka producer enabled by default
Deprecation of message formats v0 and v1
Optimizations in OffsetFetch and FindCoordinator requests
More flexible Mirror Maker 2 configuration and deprecation of Mirror Maker 1
Ability to restart a connector's tasks on a single call in Kafka Connect
Connector log contexts and connector client overrides are now enabled by default
Enhanced semantics for timestamp synchronization in Kafka Streams
Revamped public API for Stream's TaskId
Default serde becomes null in Kafka
For more information, please read the detailed Release Notes.
2.8.0
Released April 19, 2021
Release Notes
Source download: kafka-2.8.0-src.tgz (asc, sha512)
Binary downloads:
Scala 2.12 - kafka_2.12-2.8.0.tgz (asc, sha512)
Scala 2.13 - kafka_2.13-2.8.0.tgz (asc, sha512)
We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.13 is recommended).
Kafka 2.8.0 includes a number of significant new features. Here is a summary of some notable changes:
Early access of replace ZooKeeper with a self-managed quorum
Add Describe Cluster API
Support mutual TLS authentication on SASL_SSL listeners
JSON request/response debug logs
Limit broker connection creation rate
Topic identifiers
Expose task configurations in Connect REST API
Update Streams FSM to clarify ERROR state meaning
Extend StreamJoined to allow more store configs
More convenient TopologyTestDriver construtors
Introduce Kafka-Streams-specific uncaught exception handler
API to start and shut down Streams threads
Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size
Improve timeouts and retries in Kafka Streams
For more information, please read the detailed Release Notes.
2.5.0
Released April 15, 2020
Release Notes
Source download: kafka-2.5.0-src.tgz (asc, sha512)
Binary downloads:
Scala 2.12 - kafka_2.12-2.5.0.tgz (asc, sha512)
Scala 2.13 - kafka_2.13-2.5.0.tgz (asc, sha512)
We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.12 is recommended).
Kafka 2.5.0 includes a number of significant new features. Here is a summary of some notable changes:
TLS 1.3 support (1.2 is now the default)
Co-groups for Kafka Streams
Incremental rebalance for Kafka Consumer
New metrics for better operational insight
Upgrade Zookeeper to 3.5.7
Deprecate support for Scala 2.11
For more information, please read the detailed Release Notes.
2.4.1
Released March 12, 2020
Release Notes
Source download: kafka-2.4.1-src.tgz (asc, sha512)
Binary downloads:
Scala 2.11 - kafka_2.11-2.4.1.tgz (asc, sha512)
Scala 2.12 - kafka_2.12-2.4.1.tgz (asc, sha512)
Scala 2.13 - kafka_2.13-2.4.1.tgz (asc, sha512)
We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.12 is recommended).
For more information, please read the detailed Release Notes.
2.4.0
Released December 16, 2019
Release Notes
Source download: kafka-2.4.0-src.tgz (asc, sha512)
Binary downloads:
Scala 2.11 - kafka_2.11-2.4.0.tgz (asc, sha512)
Scala 2.12 - kafka_2.12-2.4.0.tgz (asc, sha512)
Scala 2.13 - kafka_2.13-2.4.0.tgz (asc, sha512)
We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.12 is recommended).
Kafka 2.4.0 includes a number of significant new features. Here is a summary of some notable changes:
Allow consumers to fetch from closest replica.
Support for incremental cooperative rebalancing to the consumer rebalance protocol.
MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication engine.
New Java authorizer Interface.
Support for non-key joining in KTable.
Administrative API for replica reassignment.
For more information, please read the detailed Release Notes.
Kafka connect版本升级至2.3.2 - HDFS集群 - 大数据平台 - Confluence
https://cwiki.guazi.com/pages/viewpage.action?pageId=190796249
kafka生态管理员手册 - 大数据平台 - Confluence
https://cwiki.guazi.com/pages/viewpage.action?pageId=45680136
Kafka、Zookeeper - 线上商城技术团队 - Confluence
https://cwiki.guazi.com/pages/viewpage.action?pageId=257673354
kafka · 语雀
https://www.yuque.com/docs/share/d28e0330-0244-407b-8de8-fba411124865
zookeeper · 语雀
https://www.yuque.com/docs/share/88898aec-cf51-40d6-9476-799b94251651
技术分享 - 工具平台 - Confluence
https://cwiki.guazi.com/pages/viewpage.action?pageId=79304597&preview=%2F79304597%2F246599606%2FApache+Flink+各版本新特性跟踪.pptx
kafka consumer - 线上商城技术团队 - Confluence
https://cwiki.guazi.com/display/xsscjs/kafka+consumer?preview=%2F259239735%2F259239734%2Fkafka+consumer.pptx
Hive版本升级方案 - 大数据平台 - Confluence
https://cwiki.guazi.com/pages/viewpage.action?pageId=233421406