配置监控
-
修改flume.env.sh
export JAVA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false” -
使用以下脚本
flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-hdfs-partition.conf -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=1234
指标项整理
OS监控项
| objectName | 指标项 | 说明 |
|---|---|---|
| java.lang:type=OperatingSystem | FreePhysicalMemorySize | 空闲物理内存 |
| java.lang:type=OperatingSystem | SystemCpuLoad | 系统CPU利用率 |
| java.lang:type=OperatingSystem | ProcessCpuLoad | 进程CPU利用率 |
| java.lang:type=GarbageCollector,name=PS Scavenge | CollectionCount | GC次数 |
json数据格式
{
"SOURCE.src-1":{
"OpenConnectionCount":"0",
"Type":"SOURCE",
"AppendBatchAcceptedCount":"1355",
"AppendBatchReceivedCount":"1355",
"EventAcceptedCount":"28286",
"AppendReceivedCount":"0",
"StopTime":"0",
"StartTime":"1442566410435",
"EventReceivedCount":"28286",
"AppendAcceptedCount":"0"
},
"CHANNEL.ch-1":{
"EventPutSuccessCount":"28286",
"ChannelFillPercentage":"0.0",
"Type":"CHANNEL",
"StopTime":"0",
"EventPutAttemptCount":"28286",
"ChannelSize":"0",
"StartTime":"1442566410326",
"EventTakeSuccessCount":"28286",
"ChannelCapacity":"1000000",
"EventTakeAttemptCount":"313734329512"
},
"SINK.sink-1":{
"Type":"SINK",
"ConnectionClosedCount":"0",
"EventDrainSuccessCount":"28286",
"KafkaEventSendTimer":"482493",
"BatchCompleteCount":"0",
"ConnectionFailedCount":"0",
"EventDrainAttemptCount":"0",
"ConnectionCreatedCount":"0",
"BatchEmptyCount":"0",
"StopTime":"0",
"RollbackCount":"9",
"StartTime":"1442566411897",
"BatchUnderflowCount":"0"
}
}
指标项说明
source监控项
| objectName (会随实际情况而变化) |
指标项 | 说明 |
|---|---|---|
| org.apache.flume.source:type=r1 | OpenConnectionCount | 目前与客户端或sink保持连接的总数量 |
| org.apache.flume.source:type=r1 | AppendBatchAcceptedCount | 成功提交到channel的批次的总数量 |
| org.apache.flume.source:type=r1 | AppendBatchReceivedCount | 接收到事件批次的总数量 |
| org.apache.flume.source:type=r1 | AppendAcceptedCount | 逐条录入的次数 |
| org.apache.flume.source:type=r1 | AppendReceivedCount | 每批只有一个事件的事件总数量 |
| org.apache.flume.source:type=r1 | EventAcceptedCount | 成功写出到channel的事件总数量 |
| org.apache.flume.source:type=r1 | EventReceivedCount | 目前为止source已经接收到的事件总数量 |
| org.apache.flume.source:type=r1 | StartTime | source启动时的毫秒值时间 |
| org.apache.flume.source:type=r1 | StopTime | source停止时的毫秒值时间，为0表示一直在运行 |
channel监控项
| objectName (会随实际情况而变化) |
指标项 | 说明 |
|---|---|---|
| org.apache.flume.channel:type=c1 | EventPutAttemptCount | Source尝试写入Channe的事件总次数 |
| org.apache.flume.channel:type=c1 | EventPutSuccessCount | 成功写入channel且提交的事件总次数 |
| org.apache.flume.channel:type=c1 | EventTakeAttemptCount | sink尝试从channel拉取事件的总次数。 |
| org.apache.flume.channel:type=c1 | EventTakeSuccessCount | sink成功从channel读取事件的总数量 |
| org.apache.flume.channel:type=c1 | ChannelSize | 目前channel中事件的总数量 |
| org.apache.flume.channel:type=c1 | ChannelCapacity | channel的容量 |
| org.apache.flume.channel:type=c1 | ChannelFillPercentage | channel已填入的百分比 |
| org.apache.flume.channel:type=c1 | StartTime | channel启动时的毫秒值时间 |
| org.apache.flume.channel:type=c1 | StopTime | channel停止时的毫秒值时间，为0表示一直在运行 |
sink监控项
| objectName (会随实际情况而变化) |
指标项 | 说明 |
|---|---|---|
| org.apache.flume.sink:type=k1 | ConnectionCreatedCount | 创建的连接数量 |
| org.apache.flume.sink:type=k1 | ConnectionClosedCount | 关闭的连接数量 |
| org.apache.flume.sink:type=k1 | ConnectionFailedCount | 由于错误关闭的连接数量 |
| org.apache.flume.sink:type=k1 | BatchEmptyCount | 批量处理event的个数为0的数量-表示source写入数据的速度比sink处理数据的速度慢 |
| org.apache.flume.sink:type=k1 | BatchUnderflowCount | 批量处理event的个数小于批处理大小的数量 |
| org.apache.flume.sink:type=k1 | BatchCompleteCount | 批量处理event的个数等于批处理大小的数量 |
| org.apache.flume.sink:type=k1 | EventDrainAttemptCount | sink尝试写出到存储的事件总数量 |
| org.apache.flume.sink:type=k1 | EventDrainSuccessCount | sink成功写出到存储的事件总数量 |
| org.apache.flume.sink:type=k1 | StartTime | channel启动时的毫秒值时间 |
| org.apache.flume.sink:type=k1 | StopTime | channel停止时的毫秒值时间，为0表示一直在运行 |