elasticsearch mysql logstash 同步 简单配置【环境centos7 elasticsearch 6.0 mysql 5.7 logstash 6.0】

插件：logstash-input-jdbc 安装插件logstash-input-jdbc

1.安装 gem
   yum install gem

2.替换国内的镜像
   gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

 

3.验证是否成功

   gem sources -l

[smartom@bogon ~]$ gem sources -l
*** CURRENT SOURCES ***

https://gems.ruby-china.org/

4.vim /usr/local/logstash/Gemfile

   修改 source 的值 为： "https://gems.ruby-china.org/"

 

5.vim /usr/local/logstash/Gemfile.jruby-1.9.lock

   修改 remote 的值 为：https://gems.ruby-china.org/

 

6. bundle config mirror.https://rubygems.org https://gems.ruby-china.org/

 

7.安装

   /usr/local/logstash/bin/logstash-plugin install logstash-input-jdbc

 

8.安装jdbc 巴拉巴拉。。。。。。。。

 

---

更改完配置文件可以使用下面测试

　　  测试：

　　  　　http://192.168.65.131:9200/akd_mall/akd_goods/3367?pretty=true  或者使用   kibana

8.配置文件 .conf 文件

input {
  jdbc {
    jdbc_driver_library => "/usr/local/jdk/jre/lib/ext/mysql-connector-java-5.1.44-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.65.131:3306/smar_mall?characterEncoding=UTF-8"
    jdbc_user => "root"
    jdbc_password => "123456"
    statement_filepath=> "smar_mall.sql"  #数据库引用路径
    tracking_column =>"create_time"     #查询更新字段
    tracking_column_type=>"timestamp"
    schedule => "* * * * *" #分时日 日月年
  }
}

filter {

}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts =>["127.0.0.1:9200"]
    index =>"=smar_mall"    #表示索引
    document_type=>"smar_goods"  #这里一定要和mapping里面的字段对应上
    document_id=>"%{id}"    #表示字段id
  }
}

9.sql文件

select id,name,description,keywords,create_time from smar_goods where 1 limit 1000

10.创建mapping

PUT smar_mall
{
    "mappings": {
      "smar_goods":{            #一定要对应上.conf里面的document_type
        "properties": {
            "name": {
              "type":"text",
              "analyzer": "ik_max_word"
            },
            "description": {
              "type":"text",
              "analyzer": "ik_max_word"
            },
            "keywords":{
              "type":"text",
              "analyzer": "ik_max_word"
            },
            "create_time":{
              "type": "date"
            }
        }
      }
    }
}