skills_kafka

目录

• skills_kafka
  • 1 offset记录位置
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

skills_kafka

1 offset记录位置

1. 首先来说说消费者如果是根据java-api来消费，也就是kafka.javaapi.consumer.ConsumerConnector，通过配置参数zookeeper.connect来消费。这种情况下，消费者的offset会更新到zookeeper的consumers/{group}/offsets/{topic}/{partition}目录下。

[zk: 100.5.14.161:2181(CONNECTED) 20] get /consumers/console-consumer-21030/offsets/topic_test_1/0
37
cZxid = 0x50000053a
ctime = Thu Dec 20 17:26:58 CST 2018
mZxid = 0x50000053a
mtime = Thu Dec 20 17:26:58 CST 2018
pZxid = 0x50000053a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 2
numChildren = 0

2. 如果是根据kafka默认的api来消费，即org.apache.kafka.clients.consumer.KafkaConsumer，我们会配置参数bootstrap.servers来消费。而其消费者的offset会更新到一个kafka自带的topic：__consumer_offsets下面，查看当前group的消费进度，则要依靠kafka自带的工具kafka-consumer-offset-checker

[2018-12-25 11:48:10,163] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Group           Topic                          Pid Offset          logSize         Lag             Owner
test_group_1    topic_test_1                   0   489             491             2               none
test_group_1    topic_test_1                   1   489             490             1               none
test_group_1    topic_test_1                   2   490             491             1               none

小结：kafka的offset即不记录在producer中，也不记录在consumer中。
参考链接：https://blog.csdn.net/lijingjingchn/article/details/85245705

2

3

4

5

6

7

8