zoukankan      html  css  js  c++  java
  • Cannot sending data from mongodb into elasticsearch using logstash

    Question:

    Hi all, i have an issue when i try to get data from mongodb to elasticsearch using logstash-input-mongodb. There is no document that get into elastic but when i stdout using rubydebug, logstash show me the documents in mongodb. After that i check the indices in elastic, they create the index but there is no document. 0 Doc in index that i created.

    This my configuration file:

    input {
    mongodb {
    uri => 'mongodb://***:*********@localhost:27017/logs'
    placeholder_db_dir => 'E:ELKlogstash-5.5.0db_dir'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'tx_log'
    batch_size => 5000
    generateId => 'true'
    parse_method => "simple"
    }
    }
    output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "tx_logs"
    }
    stdout { codec => rubydebug }
    }

    and this is some output from logstash console after they print the data on the console:
    s
    D, [2017-09-18T14:20:16.807000 #26876] DEBUG -- : MONGODB | localhost:27017 | logs.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system.|$/}}}
    D, [2017-09-18T14:20:16.812000 #26876] DEBUG -- : MONGODB | localhost:27017 | gosdev_logs.listCollections | SUCCEEDED | 0.004s

    and it looping forever.
    Thanks

    Answer:
    so ladies and gentlemen, i already solved this problem. just add this filter:

    filter {
    mutate {
    remove_field => [ "_id" ]
    }
    }

    because, logstash generate mongo_id and _id with the same value

  • 相关阅读:
    细说google,baidu引擎收录习惯
    考研机试 11.二叉树遍历
    考研机试 18.特殊乘法
    考研机试 19.今年的第几天
    考研机试 20.完数VS盈数
    考研机试 12.玛雅人的密码
    考研机试 10.球的半径和体积
    考研机试 17.n的阶乘
    考研机试 15.abc
    考研机试 9.成绩排序
  • 原文地址:https://www.cnblogs.com/a-du/p/8298384.html
Copyright © 2011-2022 走看看