zoukankan      html  css  js  c++  java
  • 使用kafka作为生产者生产数据到hdfs(单节点)

    关键:查看kafka官网的userguide

    agent.sources = kafkaSource
    agent.channels = memoryChannel
    agent.sinks = hdfsSink

    agent.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource
    agent.sources.kafkaSource.zookeeperConnect = 192.168.57.11:2181
    agent.sources.kafkaSource.topic = test_pan
    agent.sources.kafkaSource.groupId = test-consumer-group
    agent.sources.kafkaSource.kafka.consumer.timeout.ms = 100

    agent.channels.memoryChannel.type = memory
    agent.channels.memoryChannel.capacity=100
    agent.channels.memoryChannel.transactionCapacity=100

    agent.sinks.hdfsSink.type = hdfs
    agent.sinks.hdfsSink.hdfs.path = hdfs://beicai/test/pan
    agent.sinks.hdfsSink.hdfs.writeFormat = Text
    agent.sinks.hdfsSink.hdfs.fileType = DataStream


    agent.sinks.hdfsSink.hdfs.rollSize = 1024
    agent.sinks.hdfsSink.hdfs.rollCount = 0
    agent.sinks.hdfsSink.hdfs.rollInterval = 60

    agent.sinks.hdfsSink.hdfs.filePrefix=test
    agent.sinks.hdfsSink.hdfs.fileSuffix=.data

    agent.sinks.hdfsSink.hdfs.inUserPrefix=_
    agent.sinks.hdfsSink.hdfs.inUserSuffix=
    agent.sinks.hdfsSink.hdfs.fileType = DataStream
    agent.sinks.hdfsSink.hdfs.writeFormat = TEXT
    agent.sinks.hdfsSink.hdfs.rollInterval = 1
    agent.sinks.sink1.hdfs.filePrefix =A

    agent.sources.kafkaSource.channels = memoryChannel
    agent.sinks.hdfsSink.channel = memoryChannel

    成就人
  • 相关阅读:
    js获取盒子scrollTop
    获取浏览器宽度,自适应屏幕
    js切割字符串
    有向图的欧拉路径POJ2337
    欧拉回路和欧拉路径
    HDU 4462(暴力枚举)
    HDU 4455(dp)
    鸽巢原理入门
    HDU 4819 Mosaic(二维线段树)
    POJ 1330 Nearest Common Ancestors(LCA模板)
  • 原文地址:https://www.cnblogs.com/pingzizhuanshu/p/9102602.html
Copyright © 2011-2022 走看看