zoukankan      html  css  js  c++  java
  • Flume下读取kafka数据后再打把数据输出到kafka,利用拦截器解决topic覆盖问题

    1:如果在一个Flume Agent中同时使用Kafka Source和Kafka Sink来处理events,便会遇到Kafka Topic覆盖问题,具体表现为,Kafka Source可以正常从指定的Topic中读取数据,但在Kafka Sink中配置的目标Topic不起作用,数据仍然会被写入到Source中指定的Topic中。

    2:Kafka Topic覆盖问题解决方案,利用flume的拦截器

    #拦截器处理,topic覆盖问题
    agent_log.sources.kafka0.interceptors = i1
    agent_log.sources.kafka0.interceptors.i1.type = static
    agent_log.sources.kafka0.interceptors.i1.key = topic
    agent_log.sources.kafka0.interceptors.i1.preserveExisting = false
    agent_log.sources.kafka0.interceptors.i1.value = testsongout

    3:flume config完整配置

    agent_log.sources = kafka0 
    agent_log.channels = ch0 
    agent_log.sinks = sink0
    
    agent_log.sources.kafka0.channels = ch0
    agent_log.sinks.sink0.channel = ch0
    
    #sources定义
    agent_log.sources.kafka0.type = org.apache.flume.source.kafka.KafkaSource
    agent_log.sources.kafka0.kafka.bootstrap.servers = localhost:9092
    #agent.sources.kafka-source.zookeeper.connect =127.0.0.1:2181
    agent_log.sources.kafka0.kafka.topics = testsong,songtest
    agent_log.sources.kafka0.kafka.group.id= test 
    
    #拦截器处理,topic覆盖问题
    agent_log.sources.kafka0.interceptors = i1
    agent_log.sources.kafka0.interceptors.i1.type = static
    agent_log.sources.kafka0.interceptors.i1.key = topic
    agent_log.sources.kafka0.interceptors.i1.preserveExisting = false
    agent_log.sources.kafka0.interceptors.i1.value = testsongout
    
    #channels定义
    agent_log.channels.ch0.type = memory
    agent_log.channels.ch0.capacity = 2048
    agent_log.channels.ch0.transactionCapacity = 1000
    
    #sink定义
    agent_log.sinks.sink0.channel = ch0
    agent_log.sinks.sink0.type = org.apache.flume.sink.kafka.KafkaSink  
    agent_log.sinks.sink0.brokerList = localhost:9092  
    agent_log.sinks.sink0.topic = testsongout  
  • 相关阅读:
    2018 校招在线编程 20题-01
    ubuntu 配置muduo库
    plsql远程访问配置
    web开发转发和重定向大比拼
    Eclipse中svn同步忽略设置
    静态方法、实例方法和域
    接口中的域
    屏蔽所有异常的方法
    使用axis2时在temp文件产生大量缓存
    spring boot redis分布式锁 (转)
  • 原文地址:https://www.cnblogs.com/songpingyi/p/7366087.html
Copyright © 2011-2022 走看看