zoukankan      html  css  js  c++  java
  • elk 日志处理的一点思路

    zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_agent.conf 
    
    
    zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_indexer.conf
    
    
    你可以在每一个input插件设置一个type,根据type配置不同的filter....这样能节省点资源。logstash起多了,还是不太好的。毕竟每一个都是一个jvm进程....
    
    
    
    /*** 写入redis
    [elk@zjtest7-frontend config]$ cat logstash_agent.conf 
    input {
            file {
                    type => "zj_nginx_access"
                    path => ["/usr/local/nginx/logs/zj_access.log"]
            }
    
            file {
                    type => "wj_nginx_access"
                    path => ["/usr/local/nginx/logs/wj_access.log"]
            }
    }
    
    
    filter {
        grok {
            match => {
                "message" => "%{IPORHOST:clientip} [%{HTTPDATE:time}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:http_status_code} %{NUMBER:bytes} "(?
    
    <http_referer>S+)" "(?<http_user_agent>S+)" "(?<http_x_forwarded_for>S+)""
            }
        }   
    }
    output {
            redis {
                    host => "192.168.32.67"
                    data_type => "list"
                    key => "logstash:redis"
                    port=>"6379"
                    password => "1234567"
            }
    }
    
    
    
    output {
    if [type] == "xxx" {
     redis {xxxxx}
    }
    }
    
    
    根除不同的type 写入到redis
    
    
    
    /***从redis读取,发送到elasticsearch
    
    [elk@zjtest7-frontend config]$ cat logstash_indexer.conf 
    input {
            redis {
                    host => "192.168.32.67"
                    data_type => "list"
                    key => "logstash:redis"
                    type => "redis-input"
                    password => "1234567"
                    port =>"6379"
            }
    }
    
    
    
    
    
    output {
            elasticsearch {
                    hosts => "192.168.32.80:9200"
                    index => "logstash-nginx-%{+YYYY.MM.dd}"
            }
    		stdout {
    			codec => rubydebug
    		}
    }
    
    
    
    不同的 redis key 怎么发送到不同的elasticsearch 
    
    
    
    
    
    
    比如 我同步应用日志到logstash 的  /log/zjzc 下面有nginx-192.168.32.1.log  nginx-192.168.32.2.log  nginx-192.168.32.3.log 
    
    
    那么我    file {
                    type => "zj_nginx_access"
                    path => ["/log/zjzc/nginx-*.log"]
            }
    
    

  • 相关阅读:
    js用currentStyle和getComputedStyle获取css样式(非行间)
    XMLHttpRequest Level 2 使用指南
    image-set实现Retina屏幕下图片显示[转载]
    Png的秘密
    css清除&闭合浮动
    2016学习计划
    提高性能及操作硬件的能力
    新兵易学,老兵易用----C++(C++11的学习整理---如何减少代码量,加强代码的可读性)
    CV限制符--C++
    能ping通网络,也正常连接,就是打不开网页,无法访问网络
  • 原文地址:https://www.cnblogs.com/zhaoyangjian724/p/6199442.html
Copyright © 2011-2022 走看看