宜信数据采集平台DBus-All In One部署实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
官方文档可参考:https://github.com/BriData/DBus/blob/master/docs/quick-start.md
一.DBus的环境准备
DBus安装部署分为以下两种方式: All In One体验版: 该版本安装在单机上，自动部署安装dbus所依赖的相关组件，仅用于简单体验dbus基本功能，不可用于生产。 该版本支持mysql数据源接入体验。 集群部署: 用于生产环境或测试环境部署; 可部署在单机上或集群上; 该版本包含对Oracle，MySQL，logstash，filebeat，flume等数据源的支持。 温馨提示: 本篇博客在于帮助大家搭建一个学习环境，如果机器配置低的小伙伴可以先选一个干净的机器，不要和HDP或CDH集群混搭使用，也不要自己去安装zookeeper,kafka,storm,granfana,logstash等组件，但我们需要单独安装influxdb和mysql服务。 本篇的操作系统依旧选用CentOS 7.6发行版本。
1>.配置主机名解析
[root@dbus01.yinzhengjie.org.cn ~]# hostname -i 172.200.1.1 [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# grep dbus-n1 /etc/hosts 172.200.1.1 dbus-n1 dbus01.yinzhengjie.org.cn [root@dbus01.yinzhengjie.org.cn ~]#
2>.配置SSH免密登录
[root@dbus01.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/root/.ssh'. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:JMYBpQLQz/CwFBOP/WmkY2frcWnR8XkUOD94jUxoMQo root@dbus01.yinzhengjie.org.cn The key's randomart image is: +---[RSA 2048]----+ |+.+o.oo E o+. | | .== o . . .=... | | .oB+ = . o. *.o | | ..+= + . o.o* .| | + * S . o... | | . = . o . | | o + | | . + | | . | +----[SHA256]-----+ [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# ssh-copy-id root@dbus-n1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'dbus-n1 (172.200.1.1)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@dbus-n1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@dbus-n1'" and check to make sure that only the key(s) you wanted were added. [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# who root pts/0 2020-03-04 16:35 (172.200.0.1) [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# ssh root@dbus-n1 Last login: Wed Mar 4 16:56:56 2020 from 172.200.0.1 [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# who root pts/0 2020-03-04 16:35 (172.200.0.1) root pts/1 2020-03-04 17:28 (172.200.1.1) [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# exit logout Connection to dbus-n1 closed. [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
3>.宿主机的内存最少得16G以上(否则监测不通过，如下图所示)
4>.官方使用的app用户部署(如果你要采用这种方式旧的创建app用户)
[root@dbus02.yinzhengjie.org.cn ~]# adduser app [root@dbus02.yinzhengjie.org.cn ~]# [root@dbus02.yinzhengjie.org.cn ~]# echo "yinzhengjie" | passwd --stdin app Changing password for user app. passwd: all authentication tokens updated successfully. [root@dbus02.yinzhengjie.org.cn ~]#
5>.为app用户赋予权限
6>.配置免密
[root@dbus02.yinzhengjie.org.cn ~]# su - app [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ pwd /home/app [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/home/app/.ssh'. Your identification has been saved in /home/app/.ssh/id_rsa. Your public key has been saved in /home/app/.ssh/id_rsa.pub. The key fingerprint is: SHA256:WTLf8O8U0HaENUriX4rn0X92V7F4EWEIVRiVSdeT9QM app@dbus02.yinzhengjie.org.cn The key's randomart image is: +---[RSA 2048]----+ | ooEB@%| | . o+=**| | o o...o=+| | * +oo=.=| | S ..o*.+.| | o.o.o| | .o *| | o .+| | . | +----[SHA256]-----+ [app@dbus02.yinzhengjie.org.cn ~]$
[app@dbus02.yinzhengjie.org.cn ~]$ ssh-copy-id app@dbus-n1 /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/app/.ssh/id_rsa.pub" The authenticity of host 'dbus-n1 (172.200.1.2)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys app@dbus-n1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'app@dbus-n1'" and check to make sure that only the key(s) you wanted were added. [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ who root pts/0 2020-03-05 04:37 (172.200.0.1) root pts/1 2020-03-05 04:46 (172.200.0.1) [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ ssh dbus-n1 Last login: Thu Mar 5 04:47:35 2020 from 172.200.1.2 [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ who root pts/0 2020-03-05 04:37 (172.200.0.1) root pts/1 2020-03-05 04:46 (172.200.0.1) app pts/2 2020-03-05 04:47 (172.200.1.2) [app@dbus02.yinzhengjie.org.cn ~]$ [app@dbus02.yinzhengjie.org.cn ~]$ exit logout Connection to dbus-n1 closed. [app@dbus02.yinzhengjie.org.cn ~]$
二.安装InfluxDB
1>.下载并安装InfluxDB
[root@dbus01.yinzhengjie.org.cn ~]# ll total 0 [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# wget https://dl.influxdata.com/influxdb/releases/influxdb-1.1.0.x86_64.rpm --2020-03-04 08:06:04-- https://dl.influxdata.com/influxdb/releases/influxdb-1.1.0.x86_64.rpm Resolving dl.influxdata.com (dl.influxdata.com)... 99.84.194.79, 99.84.194.32, 99.84.194.77, ... Connecting to dl.influxdata.com (dl.influxdata.com)|99.84.194.79|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 15343957 (15M) [application/x-redhat-package-manager] Saving to: ‘influxdb-1.1.0.x86_64.rpm’ 100%[=========================================================================================>] 15,343,957 1.39MB/s in 11s 2020-03-04 08:06:16 (1.33 MB/s) - ‘influxdb-1.1.0.x86_64.rpm’ saved [15343957/15343957] [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# ll total 14988 -rw-r--r-- 1 root root 15343957 Nov 12 2016 influxdb-1.1.0.x86_64.rpm [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# yum -y localinstall influxdb-1.1.0.x86_64.rpm Loaded plugins: fastestmirror Examining influxdb-1.1.0.x86_64.rpm: influxdb-1.1.0-1.x86_64 Marking influxdb-1.1.0.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package influxdb.x86_64 0:1.1.0-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: influxdb x86_64 1.1.0-1 /influxdb-1.1.0.x86_64 47 M Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package Total size: 47 M Installed size: 47 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : influxdb-1.1.0-1.x86_64 1/1 Created symlink from /etc/systemd/system/influxd.service to /usr/lib/systemd/system/influxdb.service. Created symlink from /etc/systemd/system/multi-user.target.wants/influxdb.service to /usr/lib/systemd/system/influxdb.service. Verifying : influxdb-1.1.0-1.x86_64 1/1 Installed: influxdb.x86_64 0:1.1.0-1 Complete! [root@dbus01.yinzhengjie.org.cn ~]#
2>.启动InfluxDB
[root@dbus01.yinzhengjie.org.cn ~]# systemctl start influxdb.service [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# systemctl enable influxdb.service [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep influxdb influxdb.service enabled [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# systemctl status influxdb.service ● influxdb.service - InfluxDB is an open-source, distributed, time series database Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-03-04 08:11:56 CST; 4s ago Docs: https://docs.influxdata.com/influxdb/ Main PID: 5440 (influxd) CGroup: /system.slice/influxdb.service └─5440 /usr/bin/influxd -config /etc/influxdb/influxdb.conf Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [shard-precreation] 2020/03/04 08:11:56 Starting precreation service with check interval of 10m0s, advance period of 30m0s Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [snapshot] 2020/03/04 08:11:56 Starting snapshot service Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [continuous_querier] 2020/03/04 08:11:56 Starting continuous query service Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [httpd] 2020/03/04 08:11:56 Starting HTTP service Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [httpd] 2020/03/04 08:11:56 Authentication enabled: false Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [httpd] 2020/03/04 08:11:56 Listening on HTTP: [::]:8086 Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [retention] 2020/03/04 08:11:56 Starting retention policy enforcement service with check interval of 30m0s Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [run] 2020/03/04 08:11:56 Listening for signals Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: [monitor] 2020/03/04 08:11:56 Storing statistics in database '_internal' retention policy 'monitor', at interval 10s Mar 04 08:11:56 dbus01.yinzhengjie.org.cn influxd[5440]: 2020/03/04 08:11:56 Sending usage statistics to usage.influxdata.com [root@dbus01.yinzhengjie.org.cn ~]#
3>.初始化配置
#登录influx influx #执行初始化脚本 create database dbus_stat_db use dbus_stat_db CREATE USER "dbus" WITH PASSWORD 'dbus!@#123' ALTER RETENTION POLICY autogen ON dbus_stat_db DURATION 15d
> show databases; name: databases name ---- _internal dbus_stat_db > >
> show users user admin ---- ----- dbus false >
> show users user admin ---- ----- dbus false > > drop user dbus > > > show users; user admin ---- ----- >
> show databases name: databases name ---- _internal dbus_stat_db > drop database dbus_stat_db > > show databases name: databases name ---- _internal >
三.安装MySQL(cannal对MySQL有要求，推荐安装5.7)
1>.下载并安装MySQL
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/10371899.html
2>.修改MySQL的配置文件(下面标颜色的是必须得配置的)
[root@dbus01.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/my.cnf [mysqld] validate_password_policy=LOW log-bin=yinzhengjie-mysql-bin binlog-format=ROW server_id=1 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
3>.将MySQL设置为开机自启动
[root@dbus01.yinzhengjie.org.cn ~]# systemctl start mysqld.service [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# systemctl enable mysqld.service [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep mysqld.service mysqld.service enabled [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# systemctl status mysqld.service ● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-03-04 16:10:20 CST; 3min 9s ago Docs: man:mysqld(8) http://dev.mysql.com/doc/refman/en/using-systemd.html Process: 6391 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 6373 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 6394 (mysqld) CGroup: /system.slice/mysqld.service └─6394 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid Mar 04 16:10:19 dbus01.yinzhengjie.org.cn systemd[1]: Stopped MySQL Server. Mar 04 16:10:19 dbus01.yinzhengjie.org.cn systemd[1]: Starting MySQL Server... Mar 04 16:10:20 dbus01.yinzhengjie.org.cn systemd[1]: Started MySQL Server. [root@dbus01.yinzhengjie.org.cn ~]#
四.安装dbus-allinone
1>.下载dbus-allinone软件
下载地址: https://github.com/BriData/DBus/releases
2>.解压dbus-allinone.tar.gz软件包
[root@dbus01.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/softwares/dbus mkdir: created directory ‘/yinzhengjie/softwares/dbus’ [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# tar zxf dbus-allinone.tar.gz -C /yinzhengjie/softwares/dbus/ [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/ total 0 drwxr-xr-x 12 500 500 258 Aug 24 2018 dbus-allinone [root@dbus01.yinzhengjie.org.cn ~]#
3>.初始化数据库
[root@dbus01.yinzhengjie.org.cn ~]# mysql -uroot -p -e "show databases" Enter password: +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# wc -l /yinzhengjie/softwares/dbus/dbus-allinone/sql/init.sql 1106 /yinzhengjie/softwares/dbus/dbus-allinone/sql/init.sql [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# mysql -uroot -p < /yinzhengjie/softwares/dbus/dbus-allinone/sql/init.sql Enter password: [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# mysql -uroot -p -e "show databases" Enter password: +--------------------+ | Database | +--------------------+ | information_schema | | dbus | | dbusmgr | | mysql | | performance_schema | | sys | | testschema | +--------------------+ [root@dbus01.yinzhengjie.org.cn ~]#
[root@dbus01.yinzhengjie.org.cn ~]# mysql -uroot -p -e "select user,host from mysql.user" Enter password: +---------------+-----------+ | user | host | +---------------+-----------+ | canal | % | | dbus | % | | dbusmgr | % | | testschema | % | | mysql.session | localhost | | mysql.sys | localhost | | root | localhost | +---------------+-----------+ [root@dbus01.yinzhengjie.org.cn ~]#
4>.启动dbus服务
[root@dbus01.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/dbus-allinone/ total 20 drwxr-xr-x 4 500 500 50 Aug 24 2018 allinone-auto-check-0.5.0 drwxr-xr-x 15 500 500 323 Aug 15 2018 apache-storm-1.0.1 drwxrwxr-x 6 500 500 52 Sep 8 2017 canal drwxrwxr-x 5 500 500 61 Aug 23 2018 dbus-heartbeat-0.5.0 drwxrwxr-x 8 500 500 136 Aug 29 2017 grafana-4.2.0 drwxr-xr-x 8 500 500 119 Mar 2 2018 kafka_2.11-0.10.0.0 drwxrwxr-x 3 500 500 180 Aug 23 2018 keeper drwxr--r-- 14 500 500 306 Jan 25 2018 logstash-5.6.1 -rw-r--r-- 1 500 500 194 Mar 2 2018 readme.txt drwxrwxr-x 2 500 500 50 Aug 24 2018 sql -rwxr--r-- 1 500 500 9066 Aug 23 2018 start.sh drwxr-xr-x 11 500 500 4096 Mar 2 2018 zookeeper-3.4.6 [root@dbus01.yinzhengjie.org.cn ~]# [root@dbus01.yinzhengjie.org.cn ~]# cd /yinzhengjie/softwares/dbus/dbus-allinone/ [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone]# [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone]# ./start.sh #启动过程可能要花费2~3分钟，请耐心等待一下
5>.生成检查报告看是否启动正常
[root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0]# pwd /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0 [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0]# [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0]# ll total 8 -rwxr--r-- 1 500 500 1089 Aug 9 2018 auto-check.sh drwxr-xr-x 2 500 500 57 Aug 15 2018 conf drwxr-xr-x 2 500 500 4096 Aug 24 2018 lib [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0]# [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone/allinone-auto-check-0.5.0]# ./auto-check.sh
6>.
五.验证MySQL数据采集
1>.登录Grafana（用户名和密码默认均为admin）
2>.往MySQL插入数据验证
[root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone]# cat readme.txt dbus db user:dbus, passwd:j0<C6cqcqr:Dbus dbusmgr db user:dbusmgr, passwd:HxP31vevLw9PoiT/ canal user:canal, passwd:j0<C6cqcqr:Canal testdb db user:testschema, passwd:j0<C6cqcqr:TestSchema [root@dbus01.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-allinone]#
#登录测试用户 mysql -utestschema -p #testschema账户密码：j0<C6cqcqr:TestSchema #执行测试脚本 use testschema; INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00'); INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00'); INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00'); INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00'); INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');
3>.查看DBus是否实时获取到数据
4>.
六.DBus KeeperUI体验
1>.使用管理员登录(默认管理员用户名为:admin,默认密码为:12345678)
登录成功后，可以看到管理员界面，如下图所示。
2>.使用租户登录(默认租户的用户名为:user@dbus.com,默认密码为:12345678)
租户界面登录成功，如下图所示。
