zoukankan      html  css  js  c++  java
  • Cannot sending data from mongodb into elasticsearch using logstash

    Question:

    Hi all, i have an issue when i try to get data from mongodb to elasticsearch using logstash-input-mongodb. There is no document that get into elastic but when i stdout using rubydebug, logstash show me the documents in mongodb. After that i check the indices in elastic, they create the index but there is no document. 0 Doc in index that i created.

    This my configuration file:

    input {
    mongodb {
    uri => 'mongodb://***:*********@localhost:27017/logs'
    placeholder_db_dir => 'E:ELKlogstash-5.5.0db_dir'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'tx_log'
    batch_size => 5000
    generateId => 'true'
    parse_method => "simple"
    }
    }
    output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "tx_logs"
    }
    stdout { codec => rubydebug }
    }

    and this is some output from logstash console after they print the data on the console:
    s
    D, [2017-09-18T14:20:16.807000 #26876] DEBUG -- : MONGODB | localhost:27017 | logs.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system.|$/}}}
    D, [2017-09-18T14:20:16.812000 #26876] DEBUG -- : MONGODB | localhost:27017 | gosdev_logs.listCollections | SUCCEEDED | 0.004s

    and it looping forever.
    Thanks

    Answer:
    so ladies and gentlemen, i already solved this problem. just add this filter:

    filter {
    mutate {
    remove_field => [ "_id" ]
    }
    }

    because, logstash generate mongo_id and _id with the same value

  • 相关阅读:
    利用杨辉三角和阶乘计算组合数
    验证字符串是否为回文数
    利用线性同余产生伪随机数+可变参数使用
    根据RandomStr.java:使用类型转换生成六位验证字符串。
    Java语言基础问题
    从命令行输入参数值,输出求和值。
    愚公移山_节选(伪代码)
    CodeForces
    CodeForces
    E
  • 原文地址:https://www.cnblogs.com/a-du/p/8298384.html
Copyright © 2011-2022 走看看