1、 采集tomcat确实比之前的需求复杂很多，我在搭建了一个tomcat的环境，然后产生如下报错先贴出来：
Jan 05, 2017 10:53:35 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8088"]
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["ajp-bio-8009"]
java.net.BindException: Address already in use (Bind failed) <null>:8009
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:413)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:665)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:452)
at org.apache.catalina.startup.Catalina.load(Catalina.java:667)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:400)
... 16 more
2、 分析我们需要的结构：
通过上面的分析，我们需要的数据有:时间戳、类名、日志信息。
我们需要的操作就是先把相同时间和的多行日志数据合并到同一个事件里面再分析。
###提示，因为tomcat日志比较困难，我们可以参考默认的日志结构：
[root@monitor patterns]# pwd
/test/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.0.2/patterns
[root@monitor patterns]# cat java
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*.)*[a-zA-Z$_][a-zA-Z$_0-9]*
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
JAVAFILE (?:[A-Za-z0-9_. -]+)
#Allow special <init>, <clinit> methods
JAVAMETHOD (?:(<(?:cl)?init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
#Line number is optional in special cases 'Native method' or 'Unknown source'
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}.%{JAVAMETHOD:method}(%{JAVAFILE:file}(?::%{NUMBER:line})?)
# Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[d]+)
JAVACLASS (?:[a-zA-Z0-9-]+.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVASTACKTRACEPART at %{JAVACLASS:class}.%{WORD:method}(%{JAVAFILE:file}:%{NUMBER:line})
JAVALOGMESSAGE (.*)
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} | %{LOGLEVEL:level} | %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
通过对比我们可以很简单的先把日志相同时间的合并：
[root@controller etc]# cat tomcat.conf
input{stdin{}}
filter {
multiline {
pattern => "(^%{CATALINA_DATESTAMP})"
negate => true
what => "previous"
}
if "_grokparsefailure" in [tags] {
drop { }
}
grok {
match => [ "message", "%{CATALINALOG}" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd, yyyy HH:mm:ss a" ]
}
}
output{stdout{codec=>rubydebug}}
##先看测试数据，要比较小一点的：
Jan 05, 2017 10:53:35 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 728 ms
Jan 05, 2017 10:53:35 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 05, 2017 10:53:35 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.73
测试效果：
"@timestamp" => 2017-01-05T03:45:46.749Z,
"@version" => "1",
"host" => "controller",
"message" => "Jan 05, 2017 10:53:35 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 728 ms",
"tags" => [
[0] "multiline"
]
}
{
"@timestamp" => 2017-01-05T03:45:46.760Z,
"@version" => "1",
"host" => "controller",
"message" => "Jan 05, 2017 10:53:35 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina",
"tags" => [
[0] "multiline"
]
}
{
"@timestamp" => 2017-01-05T03:45:46.780Z,
"@version" => "1",
"host" => "controller",
"message" => "Jan 05, 2017 10:53:35 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.73",
"tags" => [
[0] "multiline"
]
}
3、 之前用的都是系统默认的catalina文件管理日志，通过简化的方式我们可以使用log4j的方式。
1、 安装log4j:
1、下载与Tomcat相应版本的tomcat-juli.jar 和 tomcat-juli-adapters.jar，及log4j-1.2.17.jar，放在tomcat/lib目录中 附上网址：http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.73/bin/extras/ 下载时注意你的 TOMCAT 版本
再将tomcat-juli.jar 复制到tomcat/bin目录中，替换掉原来的
2、修改 Tomcat 的 conf/context.xml 文件，将<Context>改为<Context swallowOutput="true">这步很重要。很多人会忘。
3、创建log4j.properties放在tomcat/lib中
[root@controller lib]# cat log4j.properties
log4j.rootLogger=info,Console,R
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
#log4j.appender.Console.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
log4j.appender.Console.layout.ConversionPattern=%d{yy-MM-dd HH:mm:ss} %5p %c{1}:%L - %m%n
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.File=${catalina.home}/logs/tomcat.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy.MM.dd HH:mm:ss} %5p %c{1}(%L):? %m%n
log4j.logger.org.apache=info, R
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=DEBUG, R
log4j.logger.org.apache.catalina.core=info, R
log4j.logger.org.apache.catalina.session=info, R
4、 重启看到log目录下生成tomcat.log文件说明已经安装成功了。
5、 log4j当然可以指定生成日志文件的格式：
log4j.appender.R.layout.ConversionPattern={"debug_level":"%p","debug_timestamp":"%d{ISO8601}","debug_thread":"%t","debug_file":"%F", "debug_line":"%L","debug_message":"%m"}%n
##生成日志之后直接解析成json即可。
6、 当然也有一个比较优秀的插件，也是我们推荐的方式：log4j-jsonevent-layout：
这玩意儿的作用相当于我们在nginx中干的事儿，直接将log4j的日志格式定义成json的，有助于性能提升~
7、安装：
先上传一下几个包，已经从官方打包了几个jar包，确实的话很容易失败和报错：
commons-lang-2.6.jar
jsonevent-layout-1.8-SNAPSHOT.jar
json-smart-1.1.1.jar
7、 修改log4j.properties，直接把日志发送到Logstash：
[root@controller lib]# cat log4j.properties
log4j.rootCategory=info, RollingLog ###为了方便出日志我们用Info，线上大家可以用WARN
log4j.appender.RollingLog=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingLog.Threshold=TRACE
log4j.appender.RollingLog.File=${catalina.home}/logs/api.log
log4j.appender.RollingLog.DatePattern=.yyyy-MM-dd
log4j.appender.RollingLog.layout=net.logstash.log4j.JSONEventLayoutV1
###备注：重启后我们生成了相关日志在api.log下面，下面我们用json格式可以直接解析他了。
看一下我们需要做的匹配文件：
[root@controller etc]# cat tomcat_log4j_layout.conf
input {
file {
codec => json
path => "/usr/local/src/apache-tomcat-7.0.73/logs/api.log"
type => "log4j"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output{
if[type] == "log4j"{
redis {
host => "192.168.0.46"
port => 6379
data_type => "list"
key => "logstash:log4j"
}
}
}
本文出自：http://www.roncoo.com/course/view/3c0710458fe347c2a0b31135bbbcb57b