zoukankan      html  css  js  c++  java
  • laravel5+ElasticSearch+go-mysql-elasticsearch MySQL数据实时导入(mac)

    1. ElasticSearch安装

    直接使用brew install elasticsearch 安装最新版本的es,基本没有障碍。

    2.Laravel5 框架添加elasticsearch支持

    在composer.json文件中添加elasticsearch-php依赖:

     执行composer命令更新es代码加载到当前laravel框架中

      composer update

      laravel控制器中使用es

      use ElasticsearchClientBuilder;//引入
      $client = ClientBuilder::create()->build();//初始化
      $params = [
      'index' => 'my_index',
      'type' => 'my_type',
      'id' => 'my_id',
      'body' => ['testField' => 'abc']
      ];

      $response = $client->index($params);//添加索引
      print_r($response);

      $params = [
      'index' => 'my_index',
      'type' => 'my_type',
      'id' => 'my_id'
      ];

      $response = $client->get($params);//获取my_id对应的数据
      print_r($response);
    实例:

      

    <?php

    namespace AppHttpControllers;

    use IlluminateHttpRequest;

    use AppHttpRequests;
    use AppHttpControllersController;
    use ElasticsearchClientBuilder;



    class VideoController extends Controller
    {
    private $client;

    public function __construct()
    {
    $this->client = ClientBuilder::create()->build();
    }
    /**
    * Display a listing of the resource.
    *
    * @return IlluminateHttpResponse
    */
    public function index()
    {
    for ($i=0;$i<5000;$i++) {
    $params = [
    'index' => 'my_index',
    'type' => 'my_type',
    'id' => 'my_id' . $i,
    'body' => ['testField' => $i . '、Kitty,今天气很好!!!']
    ];
    $response = $this->client->index($params);
    }

    dd($response);
    }

    /**
    * Display the specified resource.
    *
    * @param string $str
    * @return IlluminateHttpResponse
    */
    public function show($str='abc')
    {
    // $params = [
    // 'index' => 'my_index',
    // 'type' => 'my_type',
    // 'id' => $id
    // ];
    $params = [
    'index' => 'my_index',
    'type' => 'my_type',
    //"scroll" => "3s",
    "from" => 0,
    "size" => 100,
    'body' => [
    'query' => [
    'match' => [
    'testField' => $str,
    ]
    ]
    ]
    ];

    $response = $this->client->search($params);
    dd($response);
    }

    }

    3. 实时导入插件go-mysql-elasticsearch

    安装go环境,设置环境变量

      brew install go
      vi ~/.bash_profile 添加以下PATH和GOPATH,并在命令行中执行这两条命令
      export PATH=$PATH:/usr/local/Cellar/go/1.7.4_2/bin/
      export GOPATH=/Users/User/go

    获取go-mysql-elasticsearch

      go get github.com/siddontang/go-mysql-elasticsearch

    编译go-mysql-elasticsearch

      cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
      make

    配置elasticsearch的etc目录下的river.toml

    # MySQL address, user and password
    # user must have replication privilege in MySQL.
    my_addr = "192.168.0.1:3306"#mysql地址
    my_user = "test"#用户名
    my_pass = "test123"#密码

    # Elasticsearch address
    es_addr = "127.0.0.1:9200"#es安装地址

    # Path to store data, like master.info, and dump MySQL data
    data_dir = "./var"

    # Inner Http status address
    stat_addr = "127.0.0.1:12800"

    # pseudo server id like a slave
    server_id = 1001

    # mysql or mariadb
    flavor = "mysql"#mysql数据库

    # mysqldump execution path
    # if not set or empty, ignore mysqldump.
    mysqldump = "mysqldump"

    # MySQL data source
    [[source]]
    schema = "dev_cms"#数据库名称

    # Only below tables will be synced into Elasticsearch.
    # "test_river_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023
    # I don't think it is necessary to sync all tables in a database.
    tables = ["test_content"]#数据表名称,可以多个

    # Below is for special rule mapping
    #[[rule]]
    #schema = "test"
    #table = "test_river"
    #index = "river"
    #type = "river"
    # only MySQL field in filter will be synced
    #filter=["title","tags"]
    # title is MySQL test_river field name, es_title is the customized name in Elasticsearch
    #[rule.field]
    # This will map column title to elastic search my_title
    #title="es_title"
    # This will map column tags to elastic search my_tags and use array type
    #tags="my_tags,list"
    # This will map column keywords to elastic search keywords and use array type
    #keywords=",list"

    # wildcard table rule, the wildcard table must be in source tables
    [[rule]]
    schema = "dev_cms"#数据库名称
    table = "test_content"#数据表名称
    index = "es_dev_cms"#生成es数据索引名称,对应schema
    type = "es_test_content"#生成es数据类型,对应table

    # title is MySQL test_river field name, es_title is the customized name in Elasticsearch
    [[rule.fields]]
    mysql = "title"
    elastic = "es_title"

    执行命令,导入数据

    cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
    ./bin/go-mysql-elasticsearch -config=./etc/river.toml &

    可以在谷歌浏览器安装sense插件查看es数据

    clipboard.png

    有需要交流的小伙伴可以点击这里加本人QQ:luke

  • 相关阅读:
    mysql设置定时任务
    Spark On Yarn:提交Spark应用程序到Yarn
    Spark On Yarn:提交Spark应用程序到Yarn
    在Yarn上运行spark-shell和spark-sql命令行
    在Yarn上运行spark-shell和spark-sql命令行
    SparkSQL On Yarn with Hive,操作和访问Hive表
    SparkSQL On Yarn with Hive,操作和访问Hive表
    使用hive访问elasticsearch的数据
    使用hive访问elasticsearch的数据
    redis数据类型之list
  • 原文地址:https://www.cnblogs.com/starluke/p/11806640.html
Copyright © 2011-2022 走看看