confluent部署:

confluent介绍
https://www.cnblogs.com/dadadechengzi/p/9506964.html

kafka connect:
https://www.cnblogs.com/dadadechengzi/p/9506216.html

官网文档:
https://docs.confluent.io/current/

kudu sink:

https://docs.confluent.io/current/connect/kafka-connect-kudu/sink-connector/index.html#

部署:

https://blog.csdn.net/zzq900503/article/details/83419245   GG

https://www.jianshu.com/p/87ae3a822f2f

官网安装:

https://docs.confluent.io/current/quickstart/cos-quickstart.html#cos-quickstart

日志：

tail -500 /data1/confluent-5.2.2/logs/connect/kafka-connect.log |grep error

阿里：

/mnt/disk1/confluent-5.2.2/logs/connect/kafka-connect.log

日志是作业的日志,也是服务的日志

错误:

1/ [2020-02-18 11:34:16,624] WARN [Producer clientId=producer-3] Error connecting to node ali-bi-kafka-1:9092 (id: 101 rack: null) (org.apache.kafka.clients.NetworkClient)

连接kafka错误,是因为在/etc/hosts中没有注册内网域名:

cat /etc/hosts
10.52.5.208 ali-bi-kafka-1
10.52.5.209 ali-bi-kafka-2
10.52.5.210 ali-bi-kafka-3

confluent status
confluent start
confluent stop

平台自带监控平台: 适用期30天

Control Center

https://docs.confluent.io/current/control-center/installation/install-apache-kafka.html

Confluent的组件

Confluent Platform 包括更多的工具和服务，使构建和管理数据流平台更加容易。

Confluent Control Center（闭源）。管理和监控Kafka最全面的GUI驱动系统。

Confluent Kafka Connectors（开源）。连接SQL数据库/Hadoop/Hive

Confluent Kafka Clients（开源）。对于其他编程语言，包括C/C++,Python

Confluent Kafka REST Proxy（开源）。允许一些系统通过HTTP和kafka之间发送和接收消息。

Confluent Schema Registry（开源）。帮助确定每一个应用使用正确的schema当写数据或者读数据到kafka中。

confluent:
安装JDK
https://www.cnblogs.com/hongfeng2019/p/11331688.html
下载部署
https://docs.confluent.io/current/installation/installing_cp/zip-tar.html

curl -O http://packages.confluent.io/archive/5.3/confluent-5.3.1-2.12.tar.gz
tar xzf confluent-5.3.1-2.12.tar.gz -C /data1/

添加环境变量:
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:/data1/confluent-5.2.2/bin

source /etc/profile

可能还配zK,不用kafka就不需要

启动
confluent status

测试:
curl localhost:8083/connectors

[root@ali-gateway-confluent-1 system]# curl -X GET localhost:8083/connectors/
["opay-account-request-record-binlog-oss-sink","opay-account-request-record-mysql-binlog-source","opay-account-user-record-mysql-binlog-source","oride-db-mysql-binlog-source","opay-account-user-record-binlog-oss-sink","oride-db-binlog-oss-sink","oride-mysql-binlog-source-old-version","opay-bigorder-user-order-mysql-binlog

 #如果场景是指用connector,做一个systemctl启动服务

cd /usr/lib/systemd/system/

ll | grep confluent

[root@bigdata-confluent-1 system]# cat confluent-kafka-connect.service
[Unit]
Description=Apache Kafka Connect - distributed
Documentation=http://docs.confluent.io/
After=network.target

[Service]
Type=simple
Environment="KAFKA_HEAP_OPTS=-Xms1G -Xmx6G"   #调整连接kafka的heap大小
ExecStart=/data1/confluent-5.2.2/bin/connect-distributed /data1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties
TimeoutStopSec=30
Restart=always

[Install]
WantedBy=multi-user.target

---------------
systemctl daemon-reload
systemctl enable confluent-kafka-connect.service
systemctl star confluent-kafka-connect

/data1/confluent-5.2.2/etc/schema-registry
vim connect-avro-distributed.properties
bootstrap.servers=152.32.140.94:9092,152.32.140.94:19092,152.32.140.143:9092,152.32.140.143:19092

/data1/confluent-5.2.2/bin/connect-distributed /data1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties

公司：

ali-gateway-confluent-binglog-1-10.52.5.218上：

scp -r confluent-5.2.2/ root@10.52.5.227:/mnt/disk1/

#配置文件:

#cluster模式,都指向同一kafka就是一个集群,元数据都在kafka里,即消费的是kafka
#配置文件:
vim /mnt/disk1/confluent-5.2.2/etc/schema-registry/connect-avro-distributed.properties
bootstrap.servers=10.52.5.208:9092,10.52.5.209:9092,10.52.5.210:9092

#confluent中kudu的默认jar包

[root@ali-gateway-confluent-1 confluent-5.4.0]# ll /mnt/disk1/confluent-hup/share/confluent-hub-components/confluentinc-kafka-connect-kudu/lib
total 22220
-rw-r--r-- 1 root root 3482 Feb 4 10:22 animal-sniffer-annotations-1.14.jar
-rw-r--r-- 1 root root 20437 Feb 4 10:22 audience-annotations-0.5.0.jar
-rw-r--r-- 1 root root 31547 Feb 4 10:22 checker-compat-qual-2.0.0.jar
-rw-r--r-- 1 root root 51605 Feb 4 10:22 confluent-licensing-new-5.3.2-ce.jar
-rw-r--r-- 1 root root 883740 Feb 4 10:22 confluent-serializers-new-5.3.2-ce.jar
-rw-r--r-- 1 root root 12322 Feb 4 10:22 connect-licensing-extensions-0.4.4.jar
-rw-r--r-- 1 root root 62470 Feb 4 10:22 connect-utils-0.1.8.jar
-rw-r--r-- 1 root root 13704 Feb 4 10:22 error_prone_annotations-2.1.3.jar
-rw-r--r-- 1 root root 241622 Feb 4 10:22 gson-2.8.5.jar
-rw-r--r-- 1 root root 2719791 Feb 4 10:22 guava-24.0-jre.jar
-rw-r--r-- 1 root root 13348778 Feb 4 10:22 ImpalaJDBC4-2.6.4.1005.jar
-rw-r--r-- 1 root root 8782 Feb 4 10:22 j2objc-annotations-1.1.jar
-rw-r--r-- 1 root root 125632 Feb 4 10:22 jaxb-api-2.3.0.jar
-rw-r--r-- 1 root root 87325 Feb 4 10:22 jline-0.9.94.jar
-rw-r--r-- 1 root root 259680 Feb 4 10:22 jose4j-0.6.4.jar
-rw-r--r-- 1 root root 19936 Feb 4 10:22 jsr305-3.0.2.jar
-rw-r--r-- 1 root root 223667 Feb 4 10:22 kafka-connect-jdbc-5.3.1.jar
-rw-r--r-- 1 root root 258429 Feb 4 10:22 kafka-connect-kudu-1.0.1.jar
-rw-r--r-- 1 root root 489884 Feb 4 10:22 log4j-1.2.17.jar
-rw-r--r-- 1 root root 82123 Feb 4 10:22 metrics-core-2.2.0.jar
-rw-r--r-- 1 root root 1292696 Feb 4 10:22 netty-3.10.6.Final.jar
-rw-r--r-- 1 root root 1383882 Feb 4 10:22 protobuf-java-3.4.0.jar
-rw-r--r-- 1 root root 68545 Feb 4 10:22 protobuf-java-util-3.4.0.jar
-rw-r--r-- 1 root root 12189 Feb 4 10:22 slf4j-log4j12-1.7.26.jar
-rw-r--r-- 1 root root 15120 Feb 4 10:22 spotbugs-annotations-3.1.8.jar
-rw-r--r-- 1 root root 74798 Feb 4 10:22 zkclient-0.10.jar
-rw-r--r-- 1 root root 911603 Feb 4 10:22 zookeeper-3.4.14.jar