Building Scalable Microservices with NestJS and Kafka

What are Microservices?

Microservices are an architectural pattern that follows the Service-Oriented Architecture (SOA) concept. In a microservices architecture, a complex system is broken down into smaller, independent services that communicate with each other using lightweight protocols. Each service is responsible for a specific business capability and can be developed, tested, and deployed independently of other services in the system.

Why Use NestJS and Kafka?

NestJS is a popular Node.js framework for building server-side applications, while Kafka is a distributed streaming platform designed for high-throughput and fault-tolerant data processing. Together, they provide a powerful combination for building scalable microservices. NestJS provides a robust framework for building server-side applications, while Kafka enables efficient communication between microservices.

Setting Up the Project Workspace

To get started, let’s set up the project workspace. First, ensure that you have Apache Kafka installed and running on your local machine. Next, create a new Nx workspace using the following command:

npx create-nx-workspace myworkspace --preset=empty

Specify the nest option while creating the workspace and name the application api-gateway. Once the workspace is created, install the project dependencies by running the following commands:

npm install
npm run build

Creating the Auth Microservice

Next, let’s create the auth microservice. Run the following command to create a new NestJS application:

npx nest new auth-microservice

In the main.ts file, remove the boilerplate code and replace it with the NestFactory.createMicroservice() method. Pass the AppModule in the first argument and the Kafka transport object in the second argument of the createMicroservice() method.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { KafkaTransport } from './kafka.transport';

async function bootstrap() {
  const app = await NestFactory.createMicroservice(AppModule, KafkaTransport);
  await app.listen();
}
bootstrap();

Adding the Payments Microservice

Now that the auth microservice is up and running, let’s add a payments microservice. Run the following command to create a new NestJS application:

npx nest new payments-microservice

In the main.ts file, remove the boilerplate code and replace it with the NestFactory.createMicroservice() method. Pass the AppModule in the first argument and the Kafka transport object in the second argument of the createMicroservice() method.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { KafkaTransport } from './kafka.transport';

async function bootstrap() {
  const app = await NestFactory.createMicroservice(AppModule, KafkaTransport);
  await app.listen();
}
bootstrap();

Building the Microservice App

To complete the payments microservice, add the event handler for the process_payment event in the AppController. Use the @EventPattern() decorator to consume the event and process the payment accordingly.

import { Controller, EventPattern } from '@nestjs/common';

@Controller()
export class AppController {
  @EventPattern('process_payment')
  async processPayment(data: any) {
    // Process payment logic here
  }
}

Running and Testing the Services

To test and see all the services in action, run the following commands individually on separate terminals:

npm run start:auth-microservice
npm run start:payments-microservice
npm run start:api-gateway

Use Postman to test the API endpoints. Select the /api/auth/signup API to create a user, and then call the /api/payments/pay API to process a payment with the user ID and amount.

• Test the /api/auth/signup API to create a user.
• Call the /api/payments/pay API to process a payment with the user ID and amount.

By following these steps, you can build a robust application using microservices architecture with NestJS and Kafka. This stack provides a scalable, reliable, and easy-to-maintain system that is perfect for building complex applications.