Logstash(二) 使用

基本用法

测试配置文件是否在安装路径

./filebeat -configtest -e

如下输入：

2017/06/23 10:04:00.515550 beat.go:285: INFO Home path: [/home/map/tools/filebeat-5.4.1-linux-x86_64] Config path: [/home/map/tools/filebeat-5.4.1-linux-x86_64] Data path: [/home/map/tools/filebeat-5.4.1-linux-x86_64/data] Logs path: [/home/map/tools/filebeat-5.4.1-linux-x86_64/logs]
2017/06/23 10:04:00.515612 beat.go:186: INFO Setup Beat: filebeat; Version: 5.4.1
2017/06/23 10:04:00.515722 logstash.go:90: INFO Max Retries set to: 3
2017/06/23 10:04:00.515784 metrics.go:23: INFO Metrics logging every 10s
2017/06/23 10:04:00.515871 outputs.go:108: INFO Activated logstash as output plugin.
2017/06/23 10:04:00.516005 publish.go:295: INFO Publisher name: cp01-map-2016-52.epc.baidu.com
2017/06/23 10:04:00.516164 async.go:63: INFO Flush Interval set to: 1s
2017/06/23 10:04:00.516180 async.go:64: INFO Max Bulk Size set to: 2048
Config OK

简单的例子

cd logstash-5.4.2
bin/logstash -e 'input { stdin { } } output { stdout {} }'

从文件中读取数据

1. 写 taxi-pipeline.conf, 如下

input {
    file {
        path => "/home/map/data/taxi.txt"
    }
}
filter {
}
output {
    stdout {
    }
}

2. 测试配置文件是否OK

map@cp01-map-2016-52.epc.baidu.com ~/tools/logstash-5.4.1 16:34:42 $
bin/logstash -f taxi-pipeline.conf --config.test_and_exit
Sending Logstash's logs to /home/map/tools/logstash-5.4.1/logs which is now configured via log4j2.properties
Configuration OK
[2017-06-22T16:35:01,525][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

The --config.test_and_exit option parses your configuration file and reports any errors.

3. 运行

bin/logstash -f taxi-pipeline.conf --config.reload.automatic

使用Filebeat搜集日志

配置filebeat.yml文件

启动服务

./filebeat -e -c filebeat.yml -d "publish"

修改Filebeat检测log的时间

scan_frequency: 3s

Filebeat

Command Line Options

https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html

How Filebeat Works

https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html

Filebeat consists of two main components: prospectors and harvesters. These components work together to tail files and send event data to the output that you specify.

Configuring Filebeat

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

Configure File Format
Configuration Options (Reference)

Logstash

Monitoring APIs

Retrieve general information about the Logstash instance

curl -XGET 'localhost:9600/?pretty'
{
  "host" : "cp01-map-2016-52.epc.baidu.com",
  "version" : "5.4.1",
  "http_address" : "127.0.0.1:9600",
  "id" : "b5d32f3d-903c-42f8-a59c-0f0567db1c12",
  "name" : "cp01-map-2016-52.epc.baidu.com",
  "build_date" : "2017-05-29T16:40:20Z",
  "build_sha" : "cf39b7a82225994a0a3e716021c66f7a45fae46c",
  "build_snapshot" : false
}

Retrieve information about the node

curl -XGET 'localhost:9600/_node?pretty'
{
  "host" : "cp01-map-2016-52.epc.baidu.com",
  "version" : "5.4.1",
  "http_address" : "127.0.0.1:9600",
  "id" : "b5d32f3d-903c-42f8-a59c-0f0567db1c12",
  "name" : "cp01-map-2016-52.epc.baidu.com",
  "pipeline" : {
    "workers" : 12,
    "batch_size" : 125,
    "batch_delay" : 5,
    "config_reload_automatic" : true,
    "config_reload_interval" : 3,
    "id" : "main"
  },
  "os" : {
    "name" : "Linux",
    "arch" : "amd64",
    "version" : "2.6.32_1-16-0-0_virtio",
    "available_processors" : 12
  },
  "jvm" : {
    "pid" : 6909,
    "version" : "1.8.0_111",
    "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
    "vm_version" : "1.8.0_111",
    "vm_vendor" : "Oracle Corporation",
    "start_time_in_millis" : 1498462270980,
    "mem" : {
      "heap_init_in_bytes" : 268435456,
      "heap_max_in_bytes" : 1037959168,
      "non_heap_init_in_bytes" : 2555904,
      "non_heap_max_in_bytes" : 0
    },
    "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ]
  }
}

Plugins Info API

curl -XGET 'localhost:9600/_node/plugins?pretty'

Node Stats API

curl -XGET 'localhost:9600/_node/stats/<types>'
curl -XGET 'localhost:9600/_node/stats/jvm?pretty'
curl -XGET 'localhost:9600/_node/stats/pipeline?pretty'
curl -XGET 'localhost:9600/_node/stats/os?pretty'

Hot Threads API

curl -XGET 'localhost:9600/_node/hot_threads?pretty'

参考

1. Logstash Reference
2. Filebeat Reference
3. Grok Debug
4. Grok Patterns