zoukankan      html  css  js  c++  java
  • [elk]logstash统计api访问失败率

    处理原始日志

    日志从moogoo导出来的

    { "mobile" : "13612345678", "isp" : "中国移动_广东", "time" : ISODate("2017-10-16T18:39:51.245Z"), "success" : true }
    

    要解决时间问题:

    logstash修改时间

    发现日志已是json格式,想用date插件实现time字段赋值给@timestap字段,未果.(因为之前做的都是非json格式日志的时间匹配,先grok 后date)
    改造日志为:

    {"mobile" : "15812345606", "province": "上海", "isp": "中国移动","@timestamp" : "2017-12-06T09:30:51.244Z", "success" : "false"}
    {"mobile" : "15812345607", "province": "河北", "isp": "中国移动","@timestamp" : "2017-12-06T09:20:51.244Z", "success" : "true"}
    {"mobile" : "15812345607", "province": "河北", "isp": "中国联通","@timestamp" : "2017-12-06T09:22:51.244Z", "success" : "false"}
    {"mobile" : "15812345608", "province": "广东", "isp": "中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "true"}
    {"mobile" : "15812345608", "province": "广东", "isp": "中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"}
    {"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"}
    
    

    这样日志@timestamp的时间就是日志文件的时间了.

    统一日志统计需求

    上面的几幅图基本说明了问题:三级分, 第一级: 按照省份分 第二级: 按照isp分 第三级: 每个isp的标出成功失败比例

    进一步处理日志:

    mutate 拆封字段

    input { stdin { codec => "json" } }
    
    filter {
        if [success] == "true" {   // 这里true必须是字符串,否则lg启动会报错
            mutate { rename => ["sucess", "status_true"] } 
        }
        else {
            mutate { rename => ["sucess", "status_false"] } 
        }
    }
    
    output {
        stdout { codec => rubydebug }
        elasticsearch {
            hosts => [ "localhost:9200" ]
        }
    }
    

    即把日志的 "success" : "false" 拆分成2个字段:
    status_true:true
    status_false:false

    中途遇到的问题:

    1. codec => json失效.
      原因是: json数据中间本来逗号 不小心少了个逗号

    2. mutate 没成功
      日志是

    {"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}
    

    改日志为

    # 最后一个字段改成字符串即可
    {"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"} 
    

    最终日志入库展示

    接下来就是kibana出图了

    但目标是

    目前还没实现百分比.

    既然他能分两级我就这样排序: 能看到个数了

    todo: 研究百分比

    {"mobile" : "15812345606", "isp": "上海_中国移动","@timestamp" : "2017-12-06T09:30:51.244Z", "success" : false}
    {"mobile" : "15812345607", "isp": "河北_中国移动","@timestamp" : "2017-12-06T09:20:51.244Z", "success" : true}
    {"mobile" : "15812345607", "isp": "河北_中国联通","@timestamp" : "2017-12-06T09:22:51.244Z", "success" : false}
    {"mobile" : "15812345608", "isp": "广东_中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : true}
    {"mobile" : "15812345608", "isp": "广东_中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}
    {"mobile" : "15812345608", "isp": "广东_中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}
    {"mobile" : "15812345608", "isp": "广东_中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : true}
    
    
    

    最后折衷了下,采用目前方案

    绘制方法:

  • 相关阅读:
    学生免费注册Pycharm
    CSS笔记
    加载CIFAR数据集时报错的大坑
    发布小程序
    微信中的动图如果发朋友圈
    安卓第一个小项目
    转换小写字母
    1小时搞定vuepress快速制作vue文档/博客+免费部署预览
    干货满满!如何优雅简洁地实现时钟翻牌器(支持JS/Vue/React)
    JavaScript 加减危机——为什么会出现这样的结果?
  • 原文地址:https://www.cnblogs.com/iiiiher/p/7993771.html
Copyright © 2011-2022 走看看