zoukankan      html  css  js  c++  java
  • ELK——Logstash 2.2 date 插件【翻译+实践】

    官网地址

    本文内容

    • 语法
    • 测试数据
    • 可配置选项
    • 参考资料

    date 插件是日期插件,这个插件,常用而重要。

    如果不用 date 插件,那么 Logstash 将处理时间作为时间戳。时间戳字段是 Logstash 自己添加的内置字段 @timestamp,在ES中关于时间的相关查询,必须使用该字段,你当然也可以修改该字段的值。

    迁移到:http://www.bdata-cap.com/newsinfo/1712677.html

    语法


    该插件必须是用 date 包裹,如下所示:

    date {
    }

    可用的配置选项如下表所示:

    设置 输入类型 是否为必填 默认值
    add_field hash No {}
    add_tag array No []
    locale string No  
    match array No []
    periodic_flush boolean No false
    remove_field array No []
    remove_tag array No []
    tag_on_failure array No ["_dateparsefailure"]
    target string No "@timestamp"
    timezone string No  

    其中,add_field、remove_field、add_tag、remove_tag 是所有 Logstash 插件都有。它们在插件过滤成功后生效。这四个选项不多说。参见 ELK——Logstash 2.2 mutate 插件

    测试数据


    假设有 Tomcat access 日志:

    192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/goLogin" "" 8080 200 1692 23 "http://10.1.8.193:8080/goMain" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
    192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/js/common/jquery-1.10.2.min.js" "" 8080 304 - 67 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
    192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/css/common/login.css" "" 8080 304 - 75 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
    192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/js/system/login.js" "" 8080 304 - 53 "http://10.1.8.193:8080/goLogin" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"

    它是按如下 Tomcat 配置产生的:

    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
                   prefix="localhost_access_log." suffix=".txt"
                   pattern="%h %l %u %t %m &quot;%U&quot; &quot;%q&quot; %p %s %b %D &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;" />

    若用如下 Grok 表达式解析该日志:

    %{IPORHOST:clientip} %{NOTSPACE:identd} %{NOTSPACE:auth} [%{HTTPDATE:timestamp}] %{WORD:http_method} %{NOTSPACE:request} %{NOTSPACE:request_query|-} %{NUMBER:port} %{NUMBER:statusCode} (%{NOTSPACE:bytes}|-) %{NUMBER:reqTime} %{QS:referer} %{QS:userAgent}

    会得到如下结果:

    {
              "message" => "192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/goLogin" "" 8080 200 1692 23 "http://10.1.8.193:8080/goMain" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"",
             "@version" => "1",
           "@timestamp" => "2016-05-17T08:26:07.794Z",
                 "host" => "vcyber",
             "clientip" => "192.168.6.25",
               "identd" => "-",
                 "auth" => "-",
            "timestamp" => "24/Apr/2016:01:25:53 +0800",
          "http_method" => "GET",
              "request" => ""/goLogin"",
        "request_query" => """",
                 "port" => "8080",
           "statusCode" => "200",
                "bytes" => "1692",
              "reqTime" => "23",
              "referer" => ""http://10.1.8.193:8080/goMain"",
            "userAgent" => ""Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0""
    }

    注意,简单起见,日志拆分到各个字段后的数据类型全是字符串。

    可配置选项


    match

    • 值是数组 array
    • 默认值为 []

    The date formats allowed are anything allowed by Joda-Time (java time library). You can see the docs for this format here:

    joda.time.format.DateTimeFormat

    An array with field name first, and format patterns following, [ field, formats... ]

    如果你的时间字段可能有多个格式,则可指定多个可能的日期格式:

    match => [ "timestamp", "MMM dd YYY HH:mm:ss", "MMM  d YYY HH:mm:ss", "ISO8601" ]

    Logstash 支持四种日期格式:

    • ISO8601 - should parse any valid ISO8601 timestamp, such as 2011-04-19T03:44:01.103Z
    • UNIX - will parse float or int value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
    • UNIX_MS - will parse int value expressing unix time in milliseconds since epoch like 1366125117000
    • TAI64N - will parse tai64n time values

    例如,如果你有时间字段 timestamp,可能是 Aug 13 2010 00:03:44,你应该使用如下配置:

    filter {
      date {
        match => [ "logdate", "MMM dd YYYY HH:mm:ss" ]
      }
    }

    如果字段是嵌套结构,那么你可以使用嵌套语法(nested syntax) [foo][bar] 来匹配值。更多信息,参考 the section called “Field Referencesedit

    periodic_flush

    • 值是 boolean
    • 默认值为 false

    Call the filter flush method at regular interval. Optional.

    tag_on_failure

    • 值是 array
    • 默认值为 ["_dateparsefailure"]

    Append values to the tags field when there has been no successful match

    target

    • 值是 string
    • 默认值为 "@timestamp"

    把 match 的时间字段保存到指定字段。若为指定,默认更新到 @timestamp。

    示例:

    input {
            stdin {
            }
    }
    filter {
            grok {
                    match=>["message","%{IPORHOST:clientip} %{NOTSPACE:identd} %{NOTSPACE:auth} [%{HTTPDATE:timestamp}] %{WORD:http_method} %{NOTSPACE:request} %{NOTSPACE:request_query|-} %{NUMBER:port} %{NUMBER:statusCode} (%{NOTSPACE:bytes}|-) %{NUMBER:reqTime} %{QS:referer} %{QS:userAgent}"]
            }
            date {
                    match=>["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
                    target=>"@timestamp"
            }
    }
    output{
            stdout{
                    codec=>rubydebug
            }
    }

    得到如下结果:

    {
              "message" => "}192.168.6.25 - - [24/Apr/2016:01:25:53 +0800] GET "/goLogin" "" 8080 200 1692 23 "http://10.1.8.193:8080/goMain" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"",
             "@version" => "1",
           "@timestamp" => "2016-04-23T17:25:53.000Z",
                 "host" => "vcyber",
             "clientip" => "192.168.6.25",
               "identd" => "-",
                 "auth" => "-",
            "timestamp" => "24/Apr/2016:01:25:53 +0800",
          "http_method" => "GET",
              "request" => ""/goLogin"",
        "request_query" => """",
                 "port" => "8080",
           "statusCode" => "200",
                "bytes" => "1692",
              "reqTime" => "23",
              "referer" => ""http://10.1.8.193:8080/goMain"",
            "userAgent" => ""Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0""
    }

    timezone

    Specify a time zone canonical ID to be used for date parsing. The valid IDs are listed on the Joda.org available time zones page. This is useful in case the time zone cannot be extracted from the value, and is not the platform default. If this is not specified the platform default will be used. Canonical ID is good as it takes care of daylight saving time for you For example, America/Los_Angeles or Europe/Paris are valid IDs. This field can be dynamic and include parts of the event using the %{field} syntax

  • 相关阅读:
    struts2配置文件详解
    实体类和数据库映射--配置文件
    Hibernate常用配置文件详解
    本地计算机上的OracleOraDb11g_home1TNSListener服务启动后停止。某些服务在未由其他服务或程序使用时将自动停止。——Oracle监听器服务无法启动!
    关于远程访问Oracle数据库的设置(共享数据库)
    Log4J日志配置详解
    Java compiler level does not match the version of the installed Java project facet.问题
    Genymotion出现Unable to load VirtualBox engine问题--100%解决
    eclipse注解——作者,创建时间,版本
    web工程下的html中引用其他目录下的文件的path
  • 原文地址:https://www.cnblogs.com/liuning8023/p/5505744.html
Copyright © 2011-2022 走看看