hadoop之 flume1.6安装

flume 1.6安装
1.解压

2.复制

　　cp conf/flume-conf.properties.template conf/flume.conf

　　cp conf/flume-env.sh.template conf/flume-env.sh

3. 修改 JAVA_HOME

　　flume-env.sh

4.检验安装

　　flume-ng version

5.启动flume agent agent1

　　flume-ng agent --conf conf -f /bigdata/flume-1.6/conf/agent1.conf -n agent1 -Dlume.root.logger=DEBUG,console

6.创建指定文件

　　echo "hello world" > /home/hadoop/flume-1.5.0-bin/log.00

7.使用avro-client发送文件

　　flume-ng avro-client -c . -H localhost -p 41414 -F /bigdata/flume-1.6/log.00

8.在 agent1控制台会查看到相关信息

更多详细配置
　　http://www.aboutyun.com/thread-8917-1-1.html

二. 收集 udp

　　udp.conf

　　

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 8283
a1.sources.r1.host = cdh101
a1.sources.r1.channels = c1

 

#HDFS PATH
#a1.sinks.k1.hdfs.path = hdfs://nameservice:8020/user/root/udp/%Y%m%d/hour=%H
#HDFS thread numbers
a1.sinks.k1.hdfs.threadsPoolSize=100 
# HDFS the sink

#a1.sinks.k1.type = hdfs
a1.sinks.k1.type = logger
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.filePrefix = udp
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollSize = 118435456
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.idleTimeout = 1000
a1.sinks.k1.hdfs.fileType = DataStream 
a1.sinks.k1.hdfs.rollTimerPoolSize = 1

#memory size
a1.channels.c1.type = memory
a1.channels.c1.transactionCapacity = 100000
a1.channels.c1.capacity = 268435456

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1