1、安装jdk,kafka的使用要用到jdk
node2:/root#which java
/usr/java/jdk1.8.0_20/bin/java
node2:/root#java -version
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
You have new mail in /var/spool/mail/root
2、安装zookeeper
cd /usr/local/kafka
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
tar -zxvf zookeeper-3.3.6.tar.gz
export zookeeper_home=/usr/local/kafka/zookeeper-3.3.6
测试zookeeper 是否按照成功:
node2:/usr/local/kafka/zookeeper-3.3.6/bin#./zkServer.sh start
JMX enabled by default
Using config: /usr/local/kafka/zookeeper-3.3.6/bin/../conf/zoo.cfg
grep: /usr/local/kafka/zookeeper-3.3.6/bin/../conf/zoo.cfg: No such file or directory
Starting zookeeper ... STARTED
node2:/usr/local/kafka/zookeeper-3.3.6/bin#
说明启动成功
3、安装kafka
cd /usr/local/kafka
wget https://archive.apache.org/dist/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz
tar -zxvf kafka_2.8.0-0.8.0.tar.gz
配置kafka,修改文件server.properties
cd /usr/local/kafka/kafka_2.8.0-0.8.0/config
host.name=192.168.137.3
log.dirs=/usr/local/kafka/log/kafka
zookeeper.connect=localhost:2181
默认为2181端口
配置kafka下zookeeper
mkdir /usr/local/kafka/zookeeper #创建zookeeper目录
mkdir /usr/local/kafka/log/zookeeper #创建zookeeper日志目录
cd /usr/local/kafka/kafka_2.8.0-0.8.0/config
vim zookeeper.properties
dataDir=/usr/local/kafka/zookeeper
dataLogDir=/usr/local/kafka/log/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=100
tickTime=2000
initLimit=10
启动kafka:
cd /usr/local/kafka/kafka_2.8.0-0.8.0/bin
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./zookeeper-server-start.sh /usr/local/kafka/kafka_2.8.0-0.8.0/config/zookeeper.properties &
测试kafka:
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-create-topic.sh -partition 1 -replica 1 -zookeeper localhost:2181 -topic test
[2017-09-28 20:52:35,942] INFO Accepted socket connection from /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:35,966] INFO Client attempting to establish new session at /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:35,969] INFO Established session 0x15ec87bb0b60001 with negotiated timeout 30000 for client /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:36,128] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60001 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics Error:KeeperErrorCode = NoNode for /brokers/topics (org.apache.zookeeper.server.PrepRequestProcessor)
creation succeeded!
[2017-09-28 20:52:36,191] INFO Processed session termination for sessionid: 0x15ec87bb0b60001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,201] INFO Closed socket connection for client /127.0.0.1:35639 which had sessionid 0x15ec87bb0b60001 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:36,281] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60000 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/test/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/test/partitions/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,283] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60000 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/test/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/test/partitions (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,372] INFO [Replica Manager on Broker 0]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_192.168.137.3-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:192.168.137.3,port:9092 (kafka.server.ReplicaManager)
[2017-09-28 20:52:36,396] INFO [ReplicaFetcherManager on broker 0] Removing fetcher for partition [test,0] (kafka.server.ReplicaFetcherManager)
[2017-09-28 20:52:36,442] INFO [Kafka Log on Broker 0], Completed load of log test-0 with log end offset 0 (kafka.log.Log)
[2017-09-28 20:52:36,446] INFO [Log Manager on Broker 0] Created log for partition [test,0] in /usr/local/kafka/log/kafka. (kafka.log.LogManager)
[2017-09-28 20:52:36,452] WARN No highwatermark file is found. Returning 0 as the highwatermark for partition [test,0] (kafka.server.HighwaterMarkCheckpoint)
[2017-09-28 20:52:36,469] INFO [Replica Manager on Broker 0]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_192.168.137.3-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:192.168.137.3,port:9092 (kafka.server.ReplicaManager)
检查主题是否创建成功
./kafka-list-topic.sh --zookeeper localhost:2181
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-list-topic.sh -zookeeper localhost:2181
[2017-09-28 20:53:09,275] INFO Accepted socket connection from /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:53:09,286] INFO Client attempting to establish new session at /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:53:09,289] INFO Established session 0x15ec87bb0b60002 with negotiated timeout 30000 for client /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
[2017-09-28 20:53:09,813] INFO Processed session termination for sessionid: 0x15ec87bb0b60002 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:53:09,815] INFO Closed socket connection for client /127.0.0.1:35640 which had sessionid 0x15ec87bb0b60002 (org.apache.zookeeper.server.NIOServerCnxn)
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#
（2）启动producer
./kafka-console-producer.sh -broker-list 192.168.137.3:9092 -topic test
（192.168.18.229是本地服务器IP地址，读者根据情况填写
查看kafka消息:
##查看topic分布情况kafka-list-topic.sh
bin/kafka-list-topic.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 （列出所有topic的分区情况）
bin/kafka-list-topic.sh --zookeeper localhost:2181 --topic test （查看test的分区情况）
kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group ** --topic *** --zookeeper *:2181,*:2181,*:2181/kafka
kafka-list-topic.sh --zookeeper 192.168.137.3:2181
查看所有topic列表：
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-list-topic.sh --zookeeper localhost:2181
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#
查看指定topic信息
./kafka-list-topic.sh --zookeeper localhost:2181 --describe --topic test
(1) 启动kafka:
node2:/root/sbin#cat startkafka.sh
cd /usr/local/kafka/kafka_2.8.0-0.8.0/bin
./zookeeper-server-start.sh /usr/local/kafka/kafka_2.8.0-0.8.0/config/zookeeper.properties &
sleep 3
（2）启动producer
./kafka-console-producer.sh -broker-list 192.168.137.3:9092 -topic test
（192.168.18.229是本地服务器IP地址，读者根据情况填写
（3）启动consumer
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
ddddddddddddd
ffffffffffffff