重点参考:
http://blog.csdn.net/qq1032355091/article/details/52953837
logstash的精髓:
grok插件原理
date插件原理
kv插件原理
日志默认情况
默认将日志内容赋给了message字段, logstash附加了@timestamp @version host 3个字段
{
"@timestamp" => 2017-11-30T06:09:09.625Z,
"@version" => "1",
"host" => "lb-212-222.above.com",
"message" => "sad"
}
match匹配原则
参考: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
date插件匹配过程解析
input { stdin { } }
filter {
grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }
date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output {
stdout { codec => rubydebug }
}
##用正则HTTPDATE匹配message,将结果赋给[@metadata][timestamp]字段
grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }
##date插件将[@metadata][timestamp]的值赋给 @timestamp字段
date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }
下面是一个完整例子:
参考: http://blog.csdn.net/xiaoyu_bd/article/details/52531051
input {
stdin{}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
}
date {
match => ["logdate", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "@timestamp" ## 默认target就是"@timestamp
}
}
output{
stdout{
codec=>rubydebug{}
}
}
date {
match => [“timestamp”, “dd/MMM/yyyy:HH:mm:ss Z”]
#默认目标就是@timestamp
target => "@timestamp"
"locale" => "en"
}
mutate插件
- 修改字段类型
参考(修改时间格式): http://blog.csdn.net/wang_zhenwei/article/details/49760975
mutate {
convert => { "dest_Port" => "integer" }
convert => { "source_Port" => "integer" }
}
- 添加字段
input { stdin { } }
filter {
mutate { add_field => { "show" => "This data will be in the output" } }
}
output {
stdout { codec => rubydebug }
}
- 还可以转换字段大小写
kibana 查询结果csv导出
table类型的导出:
饼图统计结果导出