linux上kafka搭建小结

1、安装java

yum -y list java

可以查看java的版本，然后选1.8的64位那个安装

yum install java_1.8.0_openjdk.x86_64

或者从网站下载安装

tar -zxvf jre-8u271-linux-x64.tar.gz
mkdir /usr/lib/jvm
mv jre-8u271-linux-x64 /usr/lib/jvm
chown root:root -R /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64

java安装之后就开始设置环境变量

vi /etc/profile

#set java environment
JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64
PATH=$PATH:$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME CLASSPATH PATH

使之生效命令

source /etc/profile

或者

.  /etc/profile

2、 zookeeper

装kafka的时候才知道，必须要先安装zookeeper

从官网下载了3.5.5的包

http://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/

tar zxvf apache-zookeeper-3.5.5-bin.tar.gz
cd apache-zookeeper-3.5.5-bin
cd conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg

修改dataDir=后面的路径，就改成程序路径下，然后mkdir data  | cd data | mkdir zookeeper | mkdir log

修改环境变量/etc/profile

ZOOKEEPER_HOME=/root/hzt/apache-zookeeper-3.5.5-bin
PATH=$PATH:$ZOOKEEPER_HOME/bin

并生效

source /etc/profile

启动程序

sh zkServer.sh start

停止命令把start改成stop就行。

测试下可以用windows的cmd

telnet 替换实际的机器ip 2181

或者linux下看状态

sh zkServer.sh status

查看状态发现了一个状况

ZooKeeper JMX enabled by default
Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.

查看logs文件夹下的.out文件

2020-12-14 16:59:13,825 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0
.0:2181
2020-12-14 16:59:13,825 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2020-12-14 16:59:13,830 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set 
to 3
2020-12-14 16:59:13,832 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to
 0
2020-12-14 16:59:13,832 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2020-12-14 16:59:13,832 [myid:] - WARN  [main:QuorumPeerMain@125] - Either no config or no quorum define
d

zoo.cfg中datadir也配置过了，修改了下也没效果。然后把系统中开启的tomcat关掉后再启动就行了。不知道是不是跟8080端口被占用有关系

[opt@test bin]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone

3、安装kafka

从官网下载非源码版本的程序，带有src安装了会有问题

tar xzvf kafka_2.12-2.3.0.tgz

cd kafka_2.12-2.3.0

启动程序

bin/kafka-server-start.sh config/server.properties &

检测2181与9092端口

netstat -tunlp|egrep '(2181|9092)'

在用普通用户启动时候会报

Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
[ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)

换成root用户启动就好了，怀疑普通用户没有默认配置中跟路径权限。后期把配置修改下再试试。
功能测试，新开一个窗口作为生产者

运行producer

bin/kafka-console-producer.sh --broker-list 192.168.1.1:9092 --topic test
再开一个窗口作为消费者
sh kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.1:9092 --topic test --from-beginning
然后生产者界面输入的内容会在消费者界面出现

4、客户端开发
从github上download相应的src，使用librdkafka
unzip librdkafka-master.zip
cd librdkafka-master
sh ./configure
make
sudo make install

https://github.com/edenhill/librdkafka
完了之后可以进examples文件夹中运行demo
生产者：
./producer kafka机器的ip:9092 topic
然后就可以丢数据进去了
消费者：
./consumer kafka机器的ip:9092 groupid(Consumer group id) topics
可以订阅一个或者多个主题，组id的用途还没搞懂

如果遇到连接被kafka拒绝的情况，则需要修改下kafka的conf/server.properties中的listeners这项

本机运行的话可以改成
listeners=PLAINTEXT://localhost:9092

参考文档：
1、http://www.xwood.net/_site_domain_/_root/5870/5930/5932/t_c261426.html
2、https://www.cnblogs.com/wjup/p/11041274.html
3、https://blog.csdn.net/qq_41094332/article/details/104366315
4、https://www.cnblogs.com/justuntil/p/8033792.html
5、https://www.cnblogs.com/wangyangliuping/p/5546506.html