日志写入数据库实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
写入数据库的目的是用于持久化保存重要数据，比如状态码，客户端IP，客户端浏览器版本等等，用于后期按月做数据统计等。
一.安装MySQL数据库
1>.查看Ubuntu默认支持安装MySQL的版本
[root@es102.yinzhengjie.com ~]# apt-cache madison mysql-server mysql-server | 5.7.30-0ubuntu0.18.04.1 | http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 Packages mysql-server | 5.7.30-0ubuntu0.18.04.1 | http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 Packages mysql-server | 5.7.21-1ubuntu1 | http://mirrors.aliyun.com/ubuntu bionic/main amd64 Packages mysql-5.7 | 5.7.21-1ubuntu1 | http://mirrors.aliyun.com/ubuntu bionic/main Sources mysql-5.7 | 5.7.30-0ubuntu0.18.04.1 | http://mirrors.aliyun.com/ubuntu bionic-security/main Sources mysql-5.7 | 5.7.30-0ubuntu0.18.04.1 | http://mirrors.aliyun.com/ubuntu bionic-updates/main Sources [root@es102.yinzhengjie.com ~]#
2>.安装MySQL
[root@es102.yinzhengjie.com ~]# apt -y install mysql-server Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libaio1 libcgi-fast-perl libcgi-pm-perl libencode-locale-perl libevent-core-2.1-6 libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libtimedate-perl liburi-perl mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server-5.7 mysql-server-core-5.7 Suggested packages: libdata-dump-perl libipc-sharedcache-perl libwww-perl mailx tinyca The following NEW packages will be installed: libaio1 libcgi-fast-perl libcgi-pm-perl libencode-locale-perl libevent-core-2.1-6 libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libtimedate-perl liburi-perl mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server mysql-server-5.7 mysql-server-core-5.7 0 upgraded, 21 newly installed, 0 to remove and 83 not upgraded. Need to get 19.7 MB of archives. After this operation, 156 MB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic/main amd64 mysql-common all 5.8+1.0.4 [7,308 B] Get:2 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 libaio1 amd64 0.3.110-5ubuntu0.1 [6,476 B] Get:3 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-client-core-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [6,649 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-client-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [1,945 kB] Get:5 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-server-core-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [7,437 kB] Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libevent-core-2.1-6 amd64 2.1.8-stable-4build1 [85.9 kB] Get:7 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-server-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [2,929 kB] Get:8 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libhtml-tagset-perl all 3.20-3 [12.1 kB] Get:9 http://mirrors.aliyun.com/ubuntu bionic/main amd64 liburi-perl all 1.73-1 [77.2 kB] Get:10 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libhtml-parser-perl amd64 3.72-3build1 [85.9 kB] Get:11 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libcgi-pm-perl all 4.38-1 [185 kB] Get:12 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfcgi-perl amd64 0.78-2build1 [32.8 kB] Get:13 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libcgi-fast-perl all 1:2.13-1 [9,940 B] Get:14 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libencode-locale-perl all 1.05-1 [12.3 kB] Get:15 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libhtml-template-perl all 2.97-1 [59.0 kB] Get:16 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libtimedate-perl all 2.3000-2 [37.5 kB] Get:17 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libhttp-date-perl all 6.02-1 [10.4 kB] Get:18 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libio-html-perl all 1.001-1 [14.9 kB] Get:19 http://mirrors.aliyun.com/ubuntu bionic/main amd64 liblwp-mediatypes-perl all 6.02-1 [21.7 kB] Get:20 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libhttp-message-perl all 6.14-1 [72.1 kB] Get:21 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-server all 5.7.30-0ubuntu0.18.04.1 [9,948 B] Fetched 19.7 MB in 7s (2,992 kB/s) Preconfiguring packages ... Selecting previously unselected package mysql-common. (Reading database ... 126818 files and directories currently installed.) Preparing to unpack .../0-mysql-common_5.8+1.0.4_all.deb ... Unpacking mysql-common (5.8+1.0.4) ......................................................................................................................................................] Selecting previously unselected package libaio1:amd64....................................................................................................................................] Preparing to unpack .../1-libaio1_0.3.110-5ubuntu0.1_amd64.deb ... Unpacking libaio1:amd64 (0.3.110-5ubuntu0.1) ............................................................................................................................................] Selecting previously unselected package mysql-client-core-5.7............................................................................................................................] Preparing to unpack .../2-mysql-client-core-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-client-core-5.7 (5.7.30-0ubuntu0.18.04.1) ...............................................................................................................................] Selecting previously unselected package mysql-client-5.7.................................................................................................................................] Preparing to unpack .../3-mysql-client-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-client-5.7 (5.7.30-0ubuntu0.18.04.1) ....................................................................................................................................] Selecting previously unselected package mysql-server-core-5.7............................................................................................................................] Preparing to unpack .../4-mysql-server-core-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-server-core-5.7 (5.7.30-0ubuntu0.18.04.1) ...............................................................................................................................] Selecting previously unselected package libevent-core-2.1-6:amd64........................................................................................................................] Preparing to unpack .../5-libevent-core-2.1-6_2.1.8-stable-4build1_amd64.deb ... Unpacking libevent-core-2.1-6:amd64 (2.1.8-stable-4build1) ..............................................................................................................................] Setting up mysql-common (5.8+1.0.4) ...#####.............................................................................................................................................] update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode.................................................................................] Selecting previously unselected package mysql-server-5.7.................................................................................................................................] (Reading database ... 126986 files and directories currently installed.) Preparing to unpack .../00-mysql-server-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-server-5.7 (5.7.30-0ubuntu0.18.04.1) ....................................................................................................................................] Selecting previously unselected package libhtml-tagset-perl..............................................................................................................................] Preparing to unpack .../01-libhtml-tagset-perl_3.20-3_all.deb ... Unpacking libhtml-tagset-perl (3.20-3) ...#############..................................................................................................................................] Selecting previously unselected package liburi-perl.######...............................................................................................................................] Preparing to unpack .../02-liburi-perl_1.73-1_all.deb ... Unpacking liburi-perl (1.73-1) ...##########################.............................................................................................................................] Selecting previously unselected package libhtml-parser-perl.###..........................................................................................................................] Preparing to unpack .../03-libhtml-parser-perl_3.72-3build1_amd64.deb ... Unpacking libhtml-parser-perl (3.72-3build1) ...#################........................................................................................................................] Selecting previously unselected package libcgi-pm-perl.#############.....................................................................................................................] Preparing to unpack .../04-libcgi-pm-perl_4.38-1_all.deb ... Unpacking libcgi-pm-perl (4.38-1) ...################################....................................................................................................................] Selecting previously unselected package libfcgi-perl.####################................................................................................................................] Preparing to unpack .../05-libfcgi-perl_0.78-2build1_amd64.deb ... Unpacking libfcgi-perl (0.78-2build1) ... Selecting previously unselected package libcgi-fast-perl.####################............................................................................................................] Preparing to unpack .../06-libcgi-fast-perl_1%3a2.13-1_all.deb ... Unpacking libcgi-fast-perl (1:2.13-1) ...######################################..........................................................................................................] Selecting previously unselected package libencode-locale-perl.####################.......................................................................................................] Preparing to unpack .../07-libencode-locale-perl_1.05-1_all.deb ... Unpacking libencode-locale-perl (1.05-1) ...########################################.....................................................................................................] Selecting previously unselected package libhtml-template-perl.#########################..................................................................................................] Preparing to unpack .../08-libhtml-template-perl_2.97-1_all.deb ... Unpacking libhtml-template-perl (2.97-1) ...############################################.................................................................................................] Selecting previously unselected package libtimedate-perl.###################################.............................................................................................] Preparing to unpack .../09-libtimedate-perl_2.3000-2_all.deb ... Unpacking libtimedate-perl (2.3000-2) ...####################################################............................................................................................] Selecting previously unselected package libhttp-date-perl.######################################.........................................................................................] Preparing to unpack .../10-libhttp-date-perl_6.02-1_all.deb ... Unpacking libhttp-date-perl (6.02-1) ...##########################################################.......................................................................................] Selecting previously unselected package libio-html-perl.#############################################....................................................................................] Preparing to unpack .../11-libio-html-perl_1.001-1_all.deb ... Unpacking libio-html-perl (1.001-1) ... Selecting previously unselected package liblwp-mediatypes-perl.###########################################...............................................................................] Preparing to unpack .../12-liblwp-mediatypes-perl_6.02-1_all.deb ... Unpacking liblwp-mediatypes-perl (6.02-1) ...##############################################################..............................................................................] Selecting previously unselected package libhttp-message-perl.#################################################...........................................................................] Preparing to unpack .../13-libhttp-message-perl_6.14-1_all.deb ... Unpacking libhttp-message-perl (6.14-1) ...#####################################################################.........................................................................] Selecting previously unselected package mysql-server.##############################################################......................................................................] Preparing to unpack .../14-mysql-server_5.7.30-0ubuntu0.18.04.1_all.deb ... Unpacking mysql-server (5.7.30-0ubuntu0.18.04.1) ...#################################################################....................................................................] Setting up libhtml-tagset-perl (3.20-3) ...#############################################################################.................................................................] Setting up libevent-core-2.1-6:amd64 (2.1.8-stable-4build1) ...############################################################..............................................................] Processing triggers for ureadahead (0.100.0-21) ...###########################################################################...........................................................] Setting up libencode-locale-perl (1.05-1) ... Setting up libtimedate-perl (2.3000-2) ...######################################################################################.........................................................] Setting up libio-html-perl (1.001-1) ...#############################################################################################....................................................] Setting up liblwp-mediatypes-perl (6.02-1) ...##########################################################################################.................................................] Processing triggers for libc-bin (2.27-3ubuntu1) ...#######################################################################################..............................................] Setting up libaio1:amd64 (0.3.110-5ubuntu0.1) ... Setting up liburi-perl (1.73-1) ...###########################################################################################################...........................................] Processing triggers for systemd (237-3ubuntu10.38) ...###########################################################################################........................................] Setting up libhtml-parser-perl (3.72-3build1) ... Setting up libcgi-pm-perl (4.38-1) ...##############################################################################################################.....................................] Processing triggers for man-db (2.8.3-2ubuntu0.1) ...##################################################################################################..................................] Setting up mysql-client-core-5.7 (5.7.30-0ubuntu0.18.04.1) ... Setting up libfcgi-perl (0.78-2build1) ...#################################################################################################################..............................] Setting up libhttp-date-perl (6.02-1) ...###################################################################################################################.............................] Setting up libhtml-template-perl (2.97-1) ...####################################################################################################################........................] Setting up mysql-server-core-5.7 (5.7.30-0ubuntu0.18.04.1) ...######################################################################################################.....................] Setting up libcgi-fast-perl (1:2.13-1) ...#############################################################################################################################..................] Setting up libhttp-message-perl (6.14-1) ...##############################################################################################################################...............] Setting up mysql-client-5.7 (5.7.30-0ubuntu0.18.04.1) ...####################################################################################################################............] Setting up mysql-server-5.7 (5.7.30-0ubuntu0.18.04.1) ...########################################################################################################################........] update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode################################################################################.......] Renaming removed key_buffer and myisam-recover options (if present) Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service. Setting up mysql-server (5.7.30-0ubuntu0.18.04.1) ...###############################################################################################################################.....] Processing triggers for libc-bin (2.27-3ubuntu1) ...###################################################################################################################################..] Processing triggers for systemd (237-3ubuntu10.38) ... Processing triggers for ureadahead (0.100.0-21) ... [root@es102.yinzhengjie.com ~]#
3>.配置MySQL
[root@es102.yinzhengjie.com ~]# vim /etc/mysql/mysql.conf.d/mysqld.cnf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# egrep -v "^*#|^$" /etc/mysql/mysql.conf.d/mysqld.cnf [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking bind-address = 172.200.5.102 key_buffer_size = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 myisam-recover-options = BACKUP query_cache_limit = 1M query_cache_size = 16M log_error = /var/log/mysql/error.log expire_logs_days = 10 max_binlog_size = 100M [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 80 127.0.0.1:3306 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# systemctl restart mysql.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 80 172.200.5.102:3306 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es102.yinzhengjie.com ~]#
4>.授权用户登录
[root@es102.yinzhengjie.com ~]# mysql Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 2 Server version: 5.7.30-0ubuntu0.18.04.1 (Ubuntu) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> mysql> CREATE DATABASE elk CHARACTER SET utf8 COLLATE utf8_bin; Query OK, 1 row affected (0.00 sec) mysql> mysql> GRANT ALL PRIVILEGES ON elk.* TO elk@'172.200.5.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> mysql> QUIT Bye [root@es102.yinzhengjie.com ~]#
5>.测试用户是否可以远程登录
[root@logstash105.yinzhengjie.com ~]# apt -y install mysql-client Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: linux-headers-4.15.0-55 linux-headers-4.15.0-55-generic linux-image-4.15.0-55-generic linux-modules-4.15.0-55-generic linux-modules-extra-4.15.0-55-generic Use 'apt autoremove' to remove them. The following additional packages will be installed: libaio1 mysql-client-5.7 mysql-client-core-5.7 mysql-common The following NEW packages will be installed: libaio1 mysql-client mysql-client-5.7 mysql-client-core-5.7 mysql-common 0 upgraded, 5 newly installed, 0 to remove and 82 not upgraded. Need to get 8,618 kB of archives. After this operation, 61.8 MB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 libaio1 amd64 0.3.110-5ubuntu0.1 [6,476 B] Get:2 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-client-core-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [6,649 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 mysql-common all 5.8+1.0.4 [7,308 B] Get:4 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-client-5.7 amd64 5.7.30-0ubuntu0.18.04.1 [1,945 kB] Get:5 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 mysql-client all 5.7.30-0ubuntu0.18.04.1 [9,816 B] Fetched 8,618 kB in 1s (7,375 kB/s) Selecting previously unselected package libaio1:amd64. (Reading database ... 154519 files and directories currently installed.) Preparing to unpack .../libaio1_0.3.110-5ubuntu0.1_amd64.deb ... Unpacking libaio1:amd64 (0.3.110-5ubuntu0.1) ............................................................................................................................................] Selecting previously unselected package mysql-client-core-5.7............................................................................................................................] Preparing to unpack .../mysql-client-core-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-client-core-5.7 (5.7.30-0ubuntu0.18.04.1) ...............................................................................................................................] Selecting previously unselected package mysql-common.###.................................................................................................................................] Preparing to unpack .../mysql-common_5.8+1.0.4_all.deb ... Unpacking mysql-common (5.8+1.0.4) ...########################...........................................................................................................................] Selecting previously unselected package mysql-client-5.7.##################..............................................................................................................] Preparing to unpack .../mysql-client-5.7_5.7.30-0ubuntu0.18.04.1_amd64.deb ... Unpacking mysql-client-5.7 (5.7.30-0ubuntu0.18.04.1) ...##########################.......................................................................................................] Selecting previously unselected package mysql-client.##########################################..........................................................................................] Preparing to unpack .../mysql-client_5.7.30-0ubuntu0.18.04.1_all.deb ... Unpacking mysql-client (5.7.30-0ubuntu0.18.04.1) ...#################################################....................................................................................] Setting up mysql-common (5.8+1.0.4) ...###########################################################################.......................................................................] update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode################.................................................................] Processing triggers for libc-bin (2.27-3ubuntu1) ...###########################################################################..........................................................] Setting up libaio1:amd64 (0.3.110-5ubuntu0.1) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ...#######################################################################################.............................................] Setting up mysql-client-core-5.7 (5.7.30-0ubuntu0.18.04.1) ... Setting up mysql-client-5.7 (5.7.30-0ubuntu0.18.04.1) ...###############################################################################################.................................] Setting up mysql-client (5.7.30-0ubuntu0.18.04.1) ...################################################################################################################....................] Processing triggers for libc-bin (2.27-3ubuntu1) ...##############################################################################################################################.......] [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# mysql -u elk -pyinzhengjie -h es102.yinzhengjie.com mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 3 Server version: 5.7.30-0ubuntu0.18.04.1 (Ubuntu) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | elk | +--------------------+ 2 rows in set (0.00 sec) mysql> mysql> QUIT Bye [root@logstash105.yinzhengjie.com ~]#
6>.创建accesslog表
二.配置logstash将日志写入数据库
1>.查看logstatsh已安装的插件
[root@logstash105.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash-plugin --help Usage: bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ... Parameters: SUBCOMMAND subcommand [ARG] ... subcommand arguments Subcommands: list List all installed Logstash plugins install Install a Logstash plugin remove Remove a Logstash plugin update Update a plugin pack Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead unpack Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead generate Create the foundation for a new plugin uninstall Uninstall a plugin. Deprecated: Please use remove instead prepare-offline-pack Create an archive of specified plugins to use for offline installation Options: -h, --help print help [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash-plugin list logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-http logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-filter-json logstash-filter-kv logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-xml logstash-input-azure_event_hubs logstash-input-beats logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jdbc logstash-input-kafka logstash-input-pipe logstash-input-rabbitmq logstash-input-redis logstash-input-s3 logstash-input-snmp logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-output-cloudwatch logstash-output-csv logstash-output-elastic_app_search logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-kafka logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pagerduty logstash-output-pipe logstash-output-rabbitmq logstash-output-redis logstash-output-s3 logstash-output-sns logstash-output-sqs logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core [root@logstash105.yinzhengjie.com ~]#
2>.安装jdbc插件
[root@logstash105.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash-plugin install logstash-output-jdbc Validating logstash-output-jdbc Installing logstash-output-jdbc Installation successful [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash-plugin list | grep logstash-output-jdbc logstash-output-jdbc [root@logstash105.yinzhengjie.com ~]#
3>.安装mysql-connector对应的jar包
[root@logstash105.yinzhengjie.com ~]# wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.20-1ubuntu18.04_all.deb --2020-06-22 20:38:17-- https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.20-1ubuntu18.04_all.deb Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11 Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java_8.0.20-1ubuntu18.04_all.deb [following] --2020-06-22 20:38:18-- https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java_8.0.20-1ubuntu18.04_all.deb Resolving cdn.mysql.com (cdn.mysql.com)... 223.119.236.209 Connecting to cdn.mysql.com (cdn.mysql.com)|223.119.236.209|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2354708 (2.2M) [application/x-debian-package] Saving to: ‘mysql-connector-java_8.0.20-1ubuntu18.04_all.deb’ mysql-connector-java_8.0.20-1ubuntu18.04_all.de 100%[====================================================================================================>] 2.25M 5.69MB/s in 0.4s 2020-06-22 20:38:19 (5.69 MB/s) - ‘mysql-connector-java_8.0.20-1ubuntu18.04_all.deb’ saved [2354708/2354708] [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# dpkg -c mysql-connector-java_8.0.20-1ubuntu18.04_all.deb drwxr-xr-x root/root 0 2020-03-09 10:24 ./ drwxr-xr-x root/root 0 2020-03-09 10:24 ./usr/ drwxr-xr-x root/root 0 2020-03-09 10:24 ./usr/share/ drwxr-xr-x root/root 0 2020-03-09 10:24 ./usr/share/doc/ drwxr-xr-x root/root 0 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/ -rw-r--r-- root/root 88307 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/CHANGES.gz -rw-r--r-- root/root 173 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/INFO_BIN -rw-r--r-- root/root 135 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/INFO_SRC -rw-r--r-- root/root 25888 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/LICENSE.gz -rw-r--r-- root/root 1266 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/README -rw-r--r-- root/root 211 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/changelog.Debian.gz -rw-r--r-- root/root 377 2020-03-09 10:24 ./usr/share/doc/mysql-connector-java/copyright drwxr-xr-x root/root 0 2020-03-09 10:24 ./usr/share/java/ -rw-r--r-- root/root 2385582 2020-03-09 10:24 ./usr/share/java/mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# dpkg -i mysql-connector-java_8.0.20-1ubuntu18.04_all.deb Selecting previously unselected package mysql-connector-java. (Reading database ... 154510 files and directories currently installed.) Preparing to unpack mysql-connector-java_8.0.20-1ubuntu18.04_all.deb ... Unpacking mysql-connector-java (8.0.20-1ubuntu18.04) ... Setting up mysql-connector-java (8.0.20-1ubuntu18.04) ... [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# ll /usr/share/java/mysql-connector-java-8.0.20.jar -rw-r--r-- 1 root root 2385582 Mar 9 10:24 /usr/share/java/mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# ll /usr/share/logstash/vendor total 16 drwxrwxr-x 4 logstash logstash 4096 Jun 22 21:44 ./ drwxrwxr-x 11 logstash logstash 4096 Jun 22 04:03 ../ drwxrwxr-x 3 logstash logstash 4096 Jun 22 03:53 bundle/ drwxrwxr-x 6 logstash logstash 4096 Jun 22 04:03 jruby/ [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# mkdir -pv /usr/share/logstash/vendor/jar/jdbc mkdir: created directory '/usr/share/logstash/vendor/jar' mkdir: created directory '/usr/share/logstash/vendor/jar/jdbc' [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# ll /usr/share/logstash/vendor total 20 drwxrwxr-x 5 logstash logstash 4096 Jun 22 21:44 ./ drwxrwxr-x 11 logstash logstash 4096 Jun 22 04:03 ../ drwxrwxr-x 3 logstash logstash 4096 Jun 22 03:53 bundle/ drwxr-xr-x 3 root root 4096 Jun 22 21:44 jar/ drwxrwxr-x 6 logstash logstash 4096 Jun 22 04:03 jruby/ [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# ll /usr/share/java/mysql-connector-java-8.0.20.jar -rw-r--r-- 1 root root 2385582 Mar 9 10:24 /usr/share/java/mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# cp /usr/share/java/mysql-connector-java-8.0.20.jar /usr/share/logstash/vendor/jar/jdbc/ [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# ll /usr/share/logstash/vendor/jar/jdbc/ total 2340 drwxr-xr-x 2 root root 4096 Jun 22 21:45 ./ drwxr-xr-x 3 root root 4096 Jun 22 21:44 ../ -rw-r--r-- 1 root root 2385582 Jun 22 21:45 mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# ll /usr/share/logstash/vendor/jar/jdbc/ total 2340 drwxr-xr-x 2 root root 4096 Jun 22 21:45 ./ drwxr-xr-x 3 root root 4096 Jun 22 21:44 ../ -rw-r--r-- 1 root root 2385582 Jun 22 21:45 mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# chown logstash:logstash -R /usr/share/logstash/vendor/jar/jdbc/ [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# ll /usr/share/logstash/vendor/jar/jdbc/ total 2340 drwxr-xr-x 2 logstash logstash 4096 Jun 22 21:45 ./ drwxr-xr-x 3 root root 4096 Jun 22 21:44 ../ -rw-r--r-- 1 logstash logstash 2385582 Jun 22 21:45 mysql-connector-java-8.0.20.jar [root@logstash105.yinzhengjie.com ~]#
4>.配置filebeat收集日志到logstash
[root@es103.yinzhengjie.com ~]# egrep -v "^*#|^$" /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/syslog fields: host: "172.200.5.103" type: "filebeat-syslog-172-200-5-103" app: "ubuntu-syslog" - type: log enable: true paths: - /var/log/nginx/access.log fields: host: "172.200.5.103" type: "filebeat-nginx-accesslog-172-200-5-103" app: "nginx" filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.logstash: hosts: ["logstash105.yinzhengjie.com:8888"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~ [root@es103.yinzhengjie.com ~]#
5>.配置logstash收集日志到redis和mysql
[root@logstash105.yinzhengjie.com ~]# vim /etc/logstash/conf.d/beats-to-redis.conf [root@logstash105.yinzhengjie.com ~]# [root@logstash105.yinzhengjie.com ~]# cat /etc/logstash/conf.d/beats-to-redis.conf input { beats { host => "logstash105.yinzhengjie.com" port => 8888 codec => "json" } } output { if [fields][app] == "ubuntu-syslog" { redis { host => "redis104.yinzhengjie.com" port => "6379" password => "yinzhengjie" db => "0" key => "filebeat-syslog-172-200-5-103" data_type => "list" codec => "json" } } if [fields][app] == "nginx" { redis { host => "redis104.yinzhengjie.com" port => "6379" password => "yinzhengjie" db => "0" key => "filebeat-nginx-172-200-5-103" data_type => "list" codec => "json" } jdbc { connection_string => "jdbc:mysql://es102.yinzhengjie.com/elk?user=elk&password=yinzhengjie&useUnicode=true&characterEncoding=UTF8" statement => ["INSERT INTO accesslog(clientip,status,uri,http_user_agent,http_host,responsetime) VALUES(?,?,?,?,?,?)","clientip","status","uri","http_user_agent","http_host","re sponsetime"] } } } [root@logstash105.yinzhengjie.com ~]#
[root@logstash105.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats-to-redis.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-22 21:54:11.824 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-22 21:54:11.836 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-22 21:54:17.312 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-22 21:54:17.470 [[main]-pipeline-manager] jdbc - JDBC - Starting up [INFO ] 2020-06-22 21:54:17.510 [[main]-pipeline-manager] HikariDataSource - HikariPool-1 - Starting... [INFO ] 2020-06-22 21:54:17.812 [[main]-pipeline-manager] HikariDataSource - HikariPool-1 - Start completed. [INFO ] 2020-06-22 21:54:18.198 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"logstash105.yinzhengjie.com:8888"} [INFO ] 2020-06-22 21:54:18.224 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x43e00117 run>"} [INFO ] 2020-06-22 21:54:18.303 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-22 21:54:18.365 [[main]<beats] Server - Starting server on port: 8888 [INFO ] 2020-06-22 21:54:18.566 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:283: warning: constant ::Fixnum is deprecated /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:283: warning: constant ::Fixnum is deprecated
