zoukankan      html  css  js  c++  java
  • Cannot sending data from mongodb into elasticsearch using logstash

    Question:

    Hi all, i have an issue when i try to get data from mongodb to elasticsearch using logstash-input-mongodb. There is no document that get into elastic but when i stdout using rubydebug, logstash show me the documents in mongodb. After that i check the indices in elastic, they create the index but there is no document. 0 Doc in index that i created.

    This my configuration file:

    input {
    mongodb {
    uri => 'mongodb://***:*********@localhost:27017/logs'
    placeholder_db_dir => 'E:ELKlogstash-5.5.0db_dir'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'tx_log'
    batch_size => 5000
    generateId => 'true'
    parse_method => "simple"
    }
    }
    output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "tx_logs"
    }
    stdout { codec => rubydebug }
    }

    and this is some output from logstash console after they print the data on the console:
    s
    D, [2017-09-18T14:20:16.807000 #26876] DEBUG -- : MONGODB | localhost:27017 | logs.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system.|$/}}}
    D, [2017-09-18T14:20:16.812000 #26876] DEBUG -- : MONGODB | localhost:27017 | gosdev_logs.listCollections | SUCCEEDED | 0.004s

    and it looping forever.
    Thanks

    Answer:
    so ladies and gentlemen, i already solved this problem. just add this filter:

    filter {
    mutate {
    remove_field => [ "_id" ]
    }
    }

    because, logstash generate mongo_id and _id with the same value

  • 相关阅读:
    3dsmax script export/import tools
    BOBO输出插件的一些信息
    任务
    说说谷歌在线电子表格
    EditGrid在线编辑Excel文档
    如何控制,textField的宽度,
    在线文档管理平台
    雅虎的这个效果,有机会实现一下
    推荐在线电子表格EditGrid
    我的台账录入界面
  • 原文地址:https://www.cnblogs.com/a-du/p/8298384.html
Copyright © 2011-2022 走看看