zoukankan      html  css  js  c++  java
  • Cannot sending data from mongodb into elasticsearch using logstash

    Question:

    Hi all, i have an issue when i try to get data from mongodb to elasticsearch using logstash-input-mongodb. There is no document that get into elastic but when i stdout using rubydebug, logstash show me the documents in mongodb. After that i check the indices in elastic, they create the index but there is no document. 0 Doc in index that i created.

    This my configuration file:

    input {
    mongodb {
    uri => 'mongodb://***:*********@localhost:27017/logs'
    placeholder_db_dir => 'E:ELKlogstash-5.5.0db_dir'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'tx_log'
    batch_size => 5000
    generateId => 'true'
    parse_method => "simple"
    }
    }
    output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "tx_logs"
    }
    stdout { codec => rubydebug }
    }

    and this is some output from logstash console after they print the data on the console:
    s
    D, [2017-09-18T14:20:16.807000 #26876] DEBUG -- : MONGODB | localhost:27017 | logs.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "filter"=>{:name=>{"$not"=>/system.|$/}}}
    D, [2017-09-18T14:20:16.812000 #26876] DEBUG -- : MONGODB | localhost:27017 | gosdev_logs.listCollections | SUCCEEDED | 0.004s

    and it looping forever.
    Thanks

    Answer:
    so ladies and gentlemen, i already solved this problem. just add this filter:

    filter {
    mutate {
    remove_field => [ "_id" ]
    }
    }

    because, logstash generate mongo_id and _id with the same value

  • 相关阅读:
    CISCO 过载NAT配置(小型网络)
    CISCO静态路由配置
    CISCO 动态路由(RIP)
    CISCO 动态路由(OSPF)
    20191315 2.3.1测试
    cat userlist课上练习
    xxd
    2021-2022-1 20191315《信息安全系统设计与实现(上)》学习笔记6
    团队作业(二):需求分析
    2021-2022-1 20191315《信息安全系统设计与实现(上)》学习笔记5
  • 原文地址:https://www.cnblogs.com/a-du/p/8298384.html
Copyright © 2011-2022 走看看