zoukankan      html  css  js  c++  java
  • maxwell简单部署使用

    详细资料可以参考maxwell官网  (mysql + maxwell + kafka + elasticsearch)
    说明:本文主要是关于配置maxwell监听mysql的数据修改并实时将修改内容同步到kafka中。

    - 配置mysql启用binlog

    配置/etc/my.cnf,然后重启mysql服务

    [mysqld]
    server-id=1
    log-bin=master
    binlog_format=row
    

    在mysql中给maxwell授予相应的权限并建立maxwell数据库

    MariaDB [(none)]>  GRANT ALL on maxwell.* to 'maxwell'@'%' identified by '123456';
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]>  GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    

    建立一个测试用表,并插入一条数据

    MariaDB [(none)]> use hivetest;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A
    
    Database changed
    MariaDB [hivetest]> insert into test set name = '11111', age = '12313';
    Query OK, 1 row affected (0.00 sec)
    

    - maxwell配置文件

    [root@datanode1 ~]# cat /opt/maxwell-1.10.6/config.properties  | grep -v ^#  | grep -v ^$
    log_level=info
    producer=kafka
    kafka.bootstrap.servers=192.168.190.11:9092,192.168.190.14:9092,192.168.190.17:9092
    host=localhost
    user=maxwell
    password=123456
    kafka_topic=test
    kafka.compression.type=snappy
    kafka.metadata.fetch.timeout.ms=5000
    kafka.retries=3
    kafka.acks=all
    kinesis_stream=maxwell
    

    启动maxwell测试下是否能正常获取binglog

    [root@datanode1 maxwell-1.10.6]# bin/maxwell --user='maxwell' --password='123456' --host='127.0.0.1' --producer=stdout
    14:12:01,700 WARN  MaxwellMetrics - Metrics will not be exposed: metricsReportingType not configured.
    14:12:01,933 INFO  SchemaStoreSchema - Creating maxwell database
    14:12:02,004 INFO  Maxwell - Maxwell v1.10.6 is booting (StdoutProducer), starting at Position[BinlogPosition[mysql-bin.000001:3997], lastHeartbeat=0]
    14:12:02,146 INFO  AbstractSchemaStore - Maxwell is capturing initial schema
    14:12:02,328 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin.000001:3997
    14:12:02,383 INFO  BinaryLogClient - Connected to 127.0.0.1:3306 at mysql-bin.000001/3997 (sid:6379, cid:7)
    {"database":"hivetest","table":"test","type":"insert","ts":1506319983,"xid":142,"commit":true,"data":{"id":168,"name":"11111","age":12313,"create_time":"2017-09-25 06:13:03"}}
    

    ok,测试成功。
    后台启动maxwell,并将数据发送到kafka(需要先在kafka中建好topic)

    [root@datanode1 ~]# nohup bin/maxwell --user='maxwell' --password='123456' --host='127.0.0


    作者:观天坏笑
    链接:https://www.jianshu.com/p/5621e801f142
    來源:简书
    简书著作权归作者所有,任何形式的转载都请联系作者获得授权并注明出处。
  • 相关阅读:
    memcached命令
    模块管理常规功能自己定义系统的设计与实现(14--模块图表分析的设计)
    [易飞]凭证设计扩展字段之内容-文字显示格式
    将替代ListView的RecyclerView 的使用(一)
    POJ 2049— Finding Nemo(三维BFS)10/200
    最好用的jquery列表拖动排列(由项目提取)
    编程算法
    java几个easy出错的小程序
    GoldenGate配置(三)之DDL复制配置
    aused by: org.apache.xmlbeans.SchemaTypeLoaderException: XML-BEANS compiled schema: Incompatible min
  • 原文地址:https://www.cnblogs.com/andydao/p/9712851.html
Copyright © 2011-2022 走看看