Introduction
MapR Ecosystem Package 2.0 (MEP) is coming with some new features related to MapR Streams:
Kafka REST Proxy for MapR Streams provides a RESTful interface to MapR Streams and Kafka clusters to consume and product messages and to perform administrative operations. Kafka Connect for MapR Streams is a utility for streaming data between MapR Streams and Apache Kafka and other storage systems.
MapR Ecosystem Packs (MEPs) are a way to deliver ecosystem upgrades decoupled from core upgrades - allowing you to upgrade your tooling independently of your Converged Data Platform. You can lean more about MEP 2.0 in this article .
In this blog we describe how to use the REST Proxy to publish and consume messages to/from MapR Streams. The REST Proxy is a great addition to the MapR Converged Data Platform allowing any programming language to use MapR Streams.
The Kafka REST Proxy provided with the MapR Streams tools, can be used with MapR Streams (default), but also used in a hybrid mode with Apache Kafka. In this article we will focus on MapR Streams.
Prerequisites MapR Converged Data Platform 5.2 with MEP 2.0 with MapR Streams Tools curl, wget or any HTTP/REST Client tool Create the MapR Streams and Topic
A stream is a collection of topics that you can manage as a group by:
Setting security policies that apply to all topics in that stream Setting a default number of partitions for each new topic that is created in the stream Set a time-to-live for messages in every topic in the stream
You can find more information about MapR Streams concepts in the documentation .
On your Mapr Cluster or Sandbox, run the following commands:
$ maprcli stream create -path /apps/iot-stream -produceperm p -consumeperm p -topicperm p $ maprcli stream topic create -path /apps/iot-stream -topic sensor-json -partitions 3 $ maprcli stream topic create -path /apps/iot-stream -topic sensor-binary -partitions 3
Start Kafka Console Producers and Consumers
Open two terminal windows and run the consumer Kafka utilities using the following commands:
Consumer Topic sensor-json
$ /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-consumer.sh --new-consumer --bootstrap-server this.will.be.ignored:9092 --topic /apps/iot-stream:sensor-json
Topic sensor-binary
$ /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-consumer.sh --new-consumer --bootstrap-server this.will.be.ignored:9092 --topic /apps/iot-stream:sensor-binary
This two terminal windows will allow you to see the messages posted on the different topics
Using Kafka REST Proxy Inspect Topic Metadata The endpoint /topics/[topic_name] allows you to get some informations about the topic. In MapR Streams, topics are part of a stream identified by a path; to use the topic using the REST API you have to use the full path, and encode it in the URL; for example: /apps/iot-stream:sensor-json will be encoded with %2Fapps%2Fiot-stream%3Asensor-json
Run the following command, to get information about the sensor-json topic
$ curl -X GET http://localhost:8082/topics/%2Fapps%2Fiot-stream%3Asensor-json
Note: For simplicity reason I am running the command from the node where the Kafka REST proxy is running, so it is possible to use localhost .
You can print JSON in a pretty way, by adding a python command such as :
$ curl -X GET http://localhost:8082/topics/%2Fapps%2Fiot-stream%3Asensor-json | python -m json.tool
Default Stream
As mentioned above, the Stream path is part of the topic name you have to use in the command; however it is possible to configure the MapR Kafka REST Proxy to use a default stream. For this you should add the following property in the /opt/mapr/kafka-rest/kafka-rest-2.0.1/config/kafka-rest.properties file:
streams.default.stream=/apps/iot-stream
When you change the Kafka REST proxy configuration, you must restart the service using maprcli or MCS.
The main reason to use the streams.default.stream properties is to simplify the URLs used by the application for example * with streams.default.stream you can use curl -X GET http://localhost:8082/topics/ * without this configuration, or if you want to use a specific stream you must specify it in the URL http://localhost:8082/topics/%2Fapps%2Fiot-stream%3Asensor-json
In this article, all the URLs contains the encoded stream name, like that you can start using the Kafka REST proxy without changind the configuration and also use it with different streams.
Publishing Messages
The Kafka REST Proxy for MapR Streams allows application to publish messages to MapR Streams. Messages could be send as JSON or Binary content (base64 encoding).
To send a JSON Message: the query should be a HTTP POST the Content-Type should be : application/vnd.kafka.json.v1+json the Body: { "records": [ { "value": { "temp" : 10 , "speed" : 40 , "direction" : "NW" } } ] }
The complete request is:
curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"value": {"temp" : 10 , "speed" : 40 , "direction" : "NW"} }]}' http://localhost:8082/topics/%2Fapps%2Fiot-stream%3Asensor-json
You should see the message printed in the terminal window where the /apps/iot-stream:sensor-json consumer is running.
To send a binary Message: the query should be a HTTP POST the Content-Type should be : application/vnd.kafka.binary.v1+json the Body: { "records": [ { "value":"SGVsbG8gV29ybGQ=" } ] }