What is the difference between message queue pattern and publish-subscribe? - Quora https://www.quora.com/What-is-the-difference-between-message-queue-pattern-and-publish-subscribe
A message queue is a stream of messages which typically so to exactly one consumer.
In Publish/Subscriber, you can have any number of subscribers (including zero) listening to the same messages.
A messaging system is a software interface, that maintains a stream of messages to transfer it from one application to another. This system maintains a queue in its in-memory or in its disk to store the messages that are produced by the producers. It will wipe out those messages once they are consumed by the consumers. It follows two kinds of patterns viz. message queueing pattern and pub-sub pattern.
Message queueing pattern is a kind of point-to-point messaging system, where the message from the queue will be wiped once it is consumed by any one of the consumer. It's similar to the default behavior of Post Office Protocol, where the message from the server will be deleted once it is consumed by any of the devices.
In Publish-Subscibe pattern, publishers are the set of producers that publishes different categories of messages and subscribers are set of consumers that consumes messages from the subscribed message categories. Unlike point-to-point messaging systems, the message will be wiped from the queue only if it is consumed by all of the subscribers. In certain messaging systems like kafka, there is a retention policy specifies how long the message should stay in the queue. So the message will be available in the queue for the specified duration even though it is consumed by all of the subscribers.
Kafka as a Messaging System
How does Kafka's notion of streams compare to a traditional enterprise messaging system?
Messaging traditionally has two models: queuing and publish-subscribe. In a queue, a pool of consumers may read from a server and each record goes to one of them; in publish-subscribe the record is broadcast to all consumers. Each of these two models has a strength and a weakness. The strength of queuing is that it allows you to divide up the processing of data over multiple consumer instances, which lets you scale your processing. Unfortunately, queues aren't multi-subscriber—once one process reads the data it's gone. Publish-subscribe allows you broadcast data to multiple processes, but has no way of scaling processing since every message goes to every subscriber.
The consumer group concept in Kafka generalizes these two concepts. As with a queue the consumer group allows you to divide up processing over a collection of processes (the members of the consumer group). As with publish-subscribe, Kafka allows you to broadcast messages to multiple consumer groups.
The advantage of Kafka's model is that every topic has both these properties—it can scale processing and is also multi-subscriber—there is no need to choose one or the other.
Kafka has stronger ordering guarantees than a traditional messaging system, too.
A traditional queue retains records in-order on the server, and if multiple consumers consume from the queue then the server hands out records in the order they are stored. However, although the server hands out records in order, the records are delivered asynchronously to consumers, so they may arrive out of order on different consumers. This effectively means the ordering of the records is lost in the presence of parallel consumption. Messaging systems often work around this by having a notion of "exclusive consumer" that allows only one process to consume from a queue, but of course this means that there is no parallelism in processing.
Kafka does it better. By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances in a consumer group than partitions.
Apache Kafka http://kafka.apache.org/intro#kafka_mq