Manual message knowledge on Kafka.

Message Broker

A message broker a computer program that translates a message from the sender messaging protocol to the receiver messaging protocol. Message brokers are elements in telecommunication or computer networks where software applications communicate by exchanging formally-defined messages. Then we look at how Kafka involving with message sending.

What is Kafka?

Before going that, we need to understand what Kafka is. Kafka offers much higher performance than message brokers. Kafka can work with limited resources because it uses sequential disk I/O to boost performance. It is a suitable method for implementing message queues.

Case 01

In Kafka, we can have one topic and multiple partitions. Above figure, Employee topic and three partitions. Also, we have a project consumer group with a single consumer. This single takes messages from all these three partitions. Now we scale the application.

Case 02

There is another consumer in the consumer group. Then Consumer 1 listens to Partitions 0 and 1. Consumer 2 listening to Partition 2.

Case 03

Now there are three partitions and three consumers, respectively.

Case 04

Kafka always looks at the maximum consumers in the consumer group to the number of the consumer concerning the maximum number of partitions you have The topic has three partitions, so Kafka does not send messages for Consumer 4. So Consumer 4 is idle.

Let’s consider if consumer one or two crashes, then there is one consumer already running, so Kafka immediately transfers over to the remaining consumers. There is one more use case like this.

Case 05

Suppose we open another consumer group to the same topic. The first consumer group has three consumers and the second consumer group has two consumers. So each consumer has each partition. So how do consumers know where I take the messages so consumer offset must be wanted to manage it.

The consumer offset is Kafka topics receive a way of tracking the sequential order in messages. It keeps track of the offset. All Kafka use cases are important and can be necessary for certain instances.

References: https://www.youtube.com/watch?v=wP-FMNuO3D0&ab_channel=KrishDinesh