zoukankan      html  css  js  c++  java
  • Using KafkaBolt to write to a kafka topic

    https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html

    --------------------------------------------------------------------------------------------------------------

    I'm building a kafka and storm based streaming application based on the usecase:

    1. Application produces JSON message to a kafka topic

    2. Kafka storm spout ingests the message and does processing

    3. Produce output to another kafka topic using a storm bolt

    Here are components and versions I'm using:

    Storm: 0.10.0

    Kafka_2.10: 0.9.0.1

    HDP: 2.3.4

    storm-kafka: 0.10.0

    I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka

    I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:

      

    Properties props = new Properties();
    props.put("metadata.broker.list", "192.168.56.102:9092");
    props.put("request.required.acks", "1");
    props.put("serializer.class", "kafka.serializer.StringEncoder");
     
    Config conf = new Config();        
    conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props);
     
    KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
                          .withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");

      

    "process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.

    1. Am I doing anything wrong ?
    2. How do I use "FieldNameBasedTupleToKafkaMapper"?
  • 相关阅读:
    Building Performant Expand & Collapse Animations
    选取图片上对应区域
    css绝对对齐
    如何在node.js中使用neo4j
    io.js的六大新特性
    npm-install once
    C# EF & linq &重定向等常用操作
    js 数组
    jquery/js iframe 元素操作
    js on 和 bind 绑定click的区别 事件的冒泡 捕获 委托
  • 原文地址:https://www.cnblogs.com/twodog/p/12139577.html
Copyright © 2011-2022 走看看