Fluentd 例子

0. 安装、启动

安装

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

更新：最近貌似会安装最新版本2.3.2安装会有问题，可以手动下载td-agent-2.2.1-0.el6.x86_64.rpm，手动安装

sudo rpm -ivh td-agent-2.2.1-0.el6.x86_64.rpm

启动

/opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config --group td-agent --daemon /var/run/td-agent/td-agent.pid

1. http输入，stdout输出

例子

<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

<match td3.**>
    type stdout
</match>

请求

curl http://10.218.139.216:8888/td3 -d 'json={"hi":"abc"}'

结果（/var/log/td-agent/td-agent.log）

2016-02-03 10:17:00 +0800 td3: {"hi":"abc"}

格式：

<match td2.**>
    @type mongo
    host 10.218.139.216
    port 27017
    database db_log
    collection db_col
    time_format %H-%M-%S:%s  #时-分-秒.毫秒
    localtime                                  #本地时间
    flush_interval 10s
</match>

2. http输入，文件输出

例子

## Source descriptions
# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

## Output
# File output
# match tag=td.*.* and output to file
<match td.**>
    @type file
    path /home/jihite/monitor/test.log
    flush_interval 10s
</match>

## match tag=td2.*.* and output to file
<match td2.**>
    @type file
    path /home/jihite/monitor/test_2.log
    flush_interval 10s
</match>

http请求：

curl http://10.218.139.216:8888/td2 -d 'json={"hi":1}'

linux命令：

发POST请求工具：

结果查看：

2016-02-01T15:46:06+08:00       td      {"hi":1}

3. http输入，mongoDB输出

例子

## Source descriptions
# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

## Output
# MongoDB output
# match tag=td2.*.* and output to file
<match td2.**>
    @type mongo
    host 10.218.139.216
    port 27017
    database db_log
    collection db_col
    time_key time
    flush_interval 10s
</match>

请求

curl 10.218.139.216:8888/td2 -d 'json={"hi":"123"}'

结果查询：

> show dbs
db_log  0.078GB
local   0.078GB

> use db_log
switched to db db_log

> show collections
db_col
system.indexes

> db.db_col.find()
{ "_id" : ObjectId("56af19dbdfb99f0f50000001"), "hi" : 2, "time" : ISODate("2016-02-01T08:39:47Z") }

4. python写日志

安装插件

sudo pip install fluent-logger

参考：https://github.com/fluent/fluent-logger-python

日志记录示例：

import logging
from fluent import handler

custom_format = {
  'host': '%(hostname)s',
  'where': '%(module)s.%(funcName)s',  #具体到文件、函数
  'type': '%(levelname)s',
  'stack_trace': '%(exc_text)s'
}

logging.basicConfig(level=logging.DEBUG)

l = logging.getLogger('fluent.test')

h = handler.FluentHandler('python.mongo', host='10.218.139.216', port=24224)
formatter = handler.FluentRecordFormatter(custom_format)
h.setFormatter(formatter)

l.addHandler(h)

def funcs():
    l.warning("hello")
    l.error("hello  error")

l.info('{"from": "1", "to": "2"}')

修改td-agent.conf

<source>
    @type forward
    port 24224
</source>

<match python.mongo.**>
    @type mongo
    host 10.218.139.216
    port  27017
    database db_python
    collection col_python
    time_key time
    flush_interval 10s
</match>