zoukankan      html  css  js  c++  java
  • 使用filebeat发送nginx日志到kafka

    1.配置filebeat_nginx.yml

    filebeat.modules:
    - module: nginx
      access:
        enabled: true
        var.paths: ["/var/log/nginx/access.log*"]
      error:
        enabled: true
        var.paths: ["/var/log/nginx/error.log*"]
    
    #----------------------------------Kafka output--------------------------------#
    output.kafka:
      version: "1.0.1"
      enabled: true
      hosts: ['xxx:9092', 'xxx:9092', 'xxx:9092']
      topic: 'temp'
      required_acks: 1  #default
      compression: gzip #default
      max_message_bytes: 1000000 #default
      codec.format:
        string: '%{[message]}'
    

    2.启动filebeat

    ./filebeat -e -c filebeat_nginx.yml
    

    3.访问nginx

    tail -f /var/log/nginx/access.log
    

    日志文件输出

    {"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    

    kafka输出

    {"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    {"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
    

     如果要采集多个log,并发送到不同的topic的话,参考

    https://blog.csdn.net/Z_GodGirl/article/details/81328000
    
  • 相关阅读:
    linux系统 ssh免密码登录服务器主机
    QPS TPS 并发数 响应时间
    mysqlmtop开源系统监控mysql数据库性能
    jenkins 自动化构建 部署 回滚配置
    实用Linux命令记录
    关于高并发的几个基础问题
    tcp四次挥手
    时序数据异常检测相关的问题记录
    判断时序数据的周期性
    最短路径问题
  • 原文地址:https://www.cnblogs.com/tonglin0325/p/11670456.html
Copyright © 2011-2022 走看看