ELK 记录 java log4j 类型日志

ELK 记载  java log4j 时，一个报错会生成很多行，阅读起来很不方便。

类似这样

解决这个问题的方法

1.使用多行合并

合并多行数据(Multiline)

有些时候，应用程序调试日志会包含非常丰富的内容，为一个事件打印出很多行内容。这种日志通常都很难通过命令行解析的方式做分析。

而 logstash 正为此准备好了 codec/multiline 插件！

小贴士：multiline 插件也可以用于其他类似的堆栈式信息，比如 linux 的内核日志。

参考文章：logstash 安装插件multiline

配置文件

input {
  file {
    path => "/root/error.log"
    codec => multiline {
      pattern => "^["
      negate => true
      what => "previous"
    }
    start_position => "beginning"
    sincedb_path => "/dev/null"
    ignore_older=>0
  }
}
output {
  elasticsearch { hosts => ["10.10.15.90:9200"] 
                  index => "testjava"
  }
  stdout { codec => rubydebug }
}

以上配置文件，将 不是以 " [ "  这个符号开头的行并入上一个事件中。 

测试文本error.log

[2018-06-05 17:23:57]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.
[2018-06-05 17:23:57]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.
[2018-06-05 17:23:59]ERROR com.alibaba.dubbo.container.Main(line:86) - [DUBBO] Error creating bean with name 'YSWPurchaseListService': Cannot resolve reference to bean 'ySWPurchaseListServiceImpl' while setting bean property 'ref'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' available, dubbo version: 2.5.3, current host: 127.0.0.1
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'YSWPurchaseListService': Cannot resolve reference to bean 'ySWPurchaseListServiceImpl' while setting bean property 'ref'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' available
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1613)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' available
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
    ... 17 more
[2018-06-05 17:25:05]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.
[2018-06-05 17:26:05]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.hello world
[hello world
111111111111111hello world
[222222222hello world
[333333hello world

运行结果

{
    "@timestamp" => 2018-08-29T02:02:56.340Z,
       "message" => "[2018-06-05 17:23:57]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}
{
    "@timestamp" => 2018-08-29T02:02:56.264Z,
       "message" => "[2018-06-05 17:23:57]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}
{
          "tags" => [
        [0] "multiline"
    ],
          "host" => "localhost.localdomain",
      "@version" => "1",
    "@timestamp" => 2018-08-29T02:02:56.349Z,
       "message" => "[2018-06-05 17:23:59]ERROR com.alibaba.dubbo.container.Main(line:86) - [DUBBO] Error creating bean with name 

'YSWPurchaseListService': Cannot resolve reference to bean 'ySWPurchaseListServiceImpl' while setting bean property 'ref'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' available, dubbo version: 2.5.3, 

current host: 127.0.0.1
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'YSWPurchaseListService': 

Cannot resolve reference to bean 'ySWPurchaseListServiceImpl' while setting bean property 'ref'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' 

available
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
	at 

org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1613)
Caused by: 

org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ySWPurchaseListServiceImpl' available
	at 

org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
	... 17 more",
          "path" => "/root/error.log"
}
{
    "@timestamp" => 2018-08-29T02:02:56.353Z,
       "message" => "[2018-06-05 17:25:05]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}
{
    "@timestamp" => 2018-08-29T02:02:56.353Z,
       "message" => "[2018-06-05 17:26:05]ERROR com.alibaba.druid.pool.DruidAbstractDataSource(line:1134) -oracle.jdbc.driver.OracleDriver is deprecated.hello world",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}
{
          "tags" => [
        [0] "multiline"
    ],
          "host" => "localhost.localdomain",
      "@version" => "1",
    "@timestamp" => 2018-08-29T02:02:56.354Z,
       "message" => "[hello world
111111111111111hello world",
          "path" => "/root/error.log"
}
{
    "@timestamp" => 2018-08-29T02:02:56.354Z,
       "message" => "[222222222hello world",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}
q^H^C[WARN ] 2018-08-29 10:13:19.902 [SIGINT handler] runner - SIGINT received. Shutting down.
{
    "@timestamp" => 2018-08-29T02:13:20.848Z,
       "message" => "[333333hello world",
          "path" => "/root/error.log",
          "host" => "localhost.localdomain",
      "@version" => "1"
}

测试结果：
测试文本中的语句，按照  "[" 这个符号，被分割成了 7 个事件（总共8个事件），最后一个事件没有显示是正常的

这是因为你最后输入的回车符   并不匹配设定的 ^[ 正则表达式，

logstash 还得等下一行数据直到匹配成功后才会输出这个事件。

解释

其实这个插件的原理很简单，就是把当前行的数据添加到前面一行后面，，直到新进的当前行匹配 ^[ 正则为止。

这个正则还可以用 grok 表达式，稍后你就会学习这方面的内容。

2.使用插件input/log4j

logstash 还提供了另一种处理 log4j 的方式：input/log4j。
与 codec/multiline 不同，这个插件是直接调用了 org.apache.log4j.spi.LoggingEvent 处理 TCP 端口接收的数据。