Building Scalable Node.js APIs with Kafka Pub-Sub

Unlocking the Power of Kafka Pub-Sub in Node.js

In today’s fast-paced world of user-facing applications, handling data effectively is crucial. As the scale of daily active users grows, so does the complexity of managing data. That’s where Apache Kafka comes in – a durable, fault-tolerant, and scalable data pipeline solution. In this article, we’ll delve into the world of Kafka pub-sub, exploring its benefits and how to integrate it into your Node.js API.

What is Pub-Sub?

Pub-sub, short for publish-subscribe, is a messaging pattern that decouples the producer and consumer, allowing them to communicate asynchronously. This setup consists of publishers (pub) and subscribers (sub), where publishers broadcast events without targeting a specific subscriber. Subscribers then consume these events at their own pace.

A Brief Introduction to Kafka

Kafka offers three primary capabilities:

1. Pub-sub mechanism: Allows producers to publish events and consumers to subscribe to these events.
2. Storing data: Kafka stores events even after consumption, ensuring data durability.
3. Processing events: Kafka enables batch or real-time processing of events.

How Kafka Works

A Kafka cluster consists of brokers, which manage partitions and handle incoming requests. Producers write events to topics, which are divided into partitions. Consumers subscribe to topics and read events from partitions. Kafka ensures that published events are not deleted after consumption, allowing multiple consumers to process the same event.

Practical Constraints on Partition and Consumer Count

• A topic’s partition can only be consumed by a single consumer in a consumer group.
• Multiple consumers from different consumer groups can consume from the same partition.

Setting Up a Node.js Project with Kafka

To integrate Kafka into your Node.js project, follow these steps:

1. Clone the starter repository: git clone [email protected]:Rishabh570/kafka-nodejs-starter.git
2. Checkout to the starter branch: git checkout starter
3. Install packages: npm install
4. Run the server: npm start

Installing Kafka

To install Kafka, run the following command: npm install kafkajs

Creating a Kafka Topic

Create a Kafka topic using the Kafka CLI tool: kafka-topics.sh --create --bootstrap-server <bootstrap-server> --replication-factor 1 --partitions 1 demoTopic

Setting Up Kafka Brokers

Create two brokers in your index.js file before using the API:
“`javascript
const { KafkaClient } = require(‘kafkajs’);

const kafkaClient = new KafkaClient({
clientId: ‘demo-client’,
brokers: [‘localhost:9092’],
});

const producer = kafkaClient.producer();
const consumer = kafkaClient.consumer({ groupId: ‘demo-group’ });
“`
Writing Your First Producer

Create a producer that sends events to the demoTopic topic:
javascript
producer.send({
topic: 'demoTopic',
messages: ['Hello, World!'],
});

Configuring Your Kafka Consumer

Create a consumer that subscribes to the demoTopic topic:
“`javascript
consumer.subscribe({ topic: ‘demoTopic’ });

consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log(Received message: ${message.value});
},
});
“`
Kafka Consumer Customization Options

The Kafka consumer provides several customization options, including:

• autoCommit: Commits messages to disk after processing.
• maxBytes: Limits the size of messages fetched from Kafka.
• fromBeginning: Consumes messages from the beginning of the topic.
• seek: Consumes messages from a specific offset.

Why Kafka Pub-Sub is Better than Using HTTP Requests

Kafka pub-sub offers several advantages over HTTP requests, including:

• Decoupling of producers and consumers.
• Durability of events.
• Scal