zoukankan      html  css  js  c++  java
  • Flume监听文件目录sink至hdfs配置

    一:flume介绍

            Flume是一个分布式、可靠、和高可用的海量日志聚合的系统,支持在系统中定制各类数据发送方,用于收集数据;同时,Flume提供对数据进行简单处理,并写到各种数据接受方(可定制)的能力。,Flume架构分为三个部分 源-Source,接收器-Sink,通道-Channel

    二:配置文件

        此配置文件source为一个目录,注意,该目录下的文件应为只读,不可写,且文件名不能相同,采用的channels为file,sink为hdfs,此处往hdfs写的策略是当时间达到3600s或者文件大小达到128M。

    agent1.sources = spooldirSource
    agent1.channels = fileChannel
    agent1.sinks = hdfsSink
    
    agent1.sources.spooldirSource.type=spooldir
    agent1.sources.spooldirSource.spoolDir=/home/duanxz/flume/scan_logs
    agent1.sources.spooldirSource.channels=fileChannel
    
    agent1.sinks.hdfsSink.type=hdfs
    agent1.sinks.hdfsSink.hdfs.path=hdfs://192.168.1.105:9000/import/flume/%y-%m-%d
    agent1.sinks.hdfsSink.hdfs.filePrefix=flume
    agent1.sinks.sink1.hdfs.round = true
    # Number of seconds to wait before rolling current file (0 = never roll based on time interval)
    agent1.sinks.hdfsSink.hdfs.rollInterval = 3600
    # File size to trigger roll, in bytes (0: never roll based on file size)
    agent1.sinks.hdfsSink.hdfs.rollSize = 128000000
    agent1.sinks.hdfsSink.hdfs.rollCount = 0
    agent1.sinks.hdfsSink.hdfs.batchSize = 1000
    
    #Rounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than current time.
    agent1.sinks.hdfsSink.hdfs.roundValue = 1
    agent1.sinks.hdfsSink.hdfs.roundUnit = minute
    agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
    agent1.sinks.hdfsSink.channel=fileChannel
    agent1.sinks.hdfsSink.hdfs.fileType = DataStream
    
    
    agent1.channels.fileChannel.type = file
    agent1.channels.fileChannel.checkpointDir=/home/duanxz/flume/apache-flume-1.5.0-bin/checkpoint
    agent1.channels.fileChannel.dataDirs=/home/duanxz/flume/apache-flume-1.5.0-bin/dataDir

    三:启动命令

    duanxz@three:~/flume/apache-flume-1.8.0-bin/bin$ ./flume-ng agent --conf ../conf/ -f ../conf/flume_hdfs.conf -Dflume.root.logger=DEBUG,console -n agent1

    四、查看结果:

    或者通过hadoop的shell命令行查看文件如下:

    Q:启动flume后,控制台上输出warning信息:No configuration found for this host:flume-hdfs

    原因:flume启动时,如果没有通过参数(--name 或者 - n)指定代理,那么默认使用flume-hdfs作为代理名称,启动成功后,不会报错,但是可能会提示如题警告。

     

    Q:flume启动后,好像没有正常工作,查看日志,看到flume启动报错:org.apache.commons.cli.MissingOptionException: Missing required option: n

    解决方案:agent启动时需要用-n参数指定agent的名字(用于跟flume配置中的agent名字对应上)

  • 相关阅读:
    Quartz.net
    Perfview 分析进程性能
    windbg 分析cpu异常
    ansible-vault 教程
    ansible 自动化运维(2)
    简单生成随机测试数据
    基于 RabbitMQ-EasyNetQ 实现.NET与Go的消息调度交互
    自绘 TreeDataView 控件
    C# 创建音频WAVE文件头信息(*.wav)
    C# GOF 23种设计模式 学习Log【更新中】
  • 原文地址:https://www.cnblogs.com/duanxz/p/9178773.html
Copyright © 2011-2022 走看看