zoukankan      html  css  js  c++  java
  • kafka与flume的集成

    kafka可以作为kafka的任意一个组件,source、channel、sink

    kafka(消息集群中间件,可重复消费,高吞吐量,暂存)和flume(落地,抓取源文件,监控目录,实时收集)

    kafka与flume的集成

    1.kafka作为source,从kafka中取数据,source.type=kafkaSource(消费者角色)

    复制代码
    a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
    //batch info
    a1.sources.r1.batchSize = 5000
    a1.sources.r1.batchDurationMillis = 2000
    //broker
    a1.sources.r1.kafka.bootstrap.servers = s202:9092
    //topicname
    a1.sources.r1.kafka.topics = topic1
    //groupname
    a1.sources.r1.kafka.consumer.group.id = g1
    复制代码

    2.kafka作为sink

    把消息推送到kafka,

    复制代码
    a1.sources=r1
    a1.channels=c1
    a1.sinks=k1
    
    a1.sources.r1.type=netcat
    a1.sources.r1.bind=localhost
    a1.sources.r1.port=8888
    
    a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
    a1.sinks.k1.kafka.topic = topic1
    a1.sinks.k1.kafka.bootstrap.servers = s202:9092
    a1.sinks.k1.kafka.flumeBatchSize = 20
    a1.sinks.k1.kafka.producer.acks = 1
    a1.sinks.k1.kafka.producer.linger.ms = 1
    
    a1.channels.c1.type=memory
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel= c1
    复制代码

    3.kafka作为channel

    既作为生产者,也作为消费者

    复制代码
    a1.sources=r1
    a1.channels=c1
    a1.sinks=k1
    
    a1.sources.r1.type=netcat
    a1.sources.r1.bind=localhost
    a1.sources.r1.port=8888
    
    a1.sinks.k1.type = logger
    
    a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
    a1.channels.c1.parseAsFlumeEvent = false
    a1.channels.c1.kafka.bootstrap.servers = s202:9092
    a1.channels.c1.kafka.topic = topic1
    a1.channels.c1.kafka.consumer.group.id = g5
    
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel= c1
    复制代码
  • 相关阅读:
    your account already has a valid ios distribution certificate
    uinavigation样式
    phonegap ios默认启动页
    git init出错
    ios assetlibrary
    uitableviewcell高度自适应笔记
    ios frame bounds applicationframe
    java学习笔记
    file not found while xcode archive
    c++回调
  • 原文地址:https://www.cnblogs.com/gaohuajie/p/12917710.html
Copyright © 2011-2022 走看看