Node.js Cloud Pub/Sub Tutorial for Beginners

Unlocking the Power of Event-Driven Architecture with Cloud Pub/Sub

In recent years, the popularity of microservices has led to a surge in the adoption of event-driven architecture (EDA). This approach enables independent services to communicate with each other seamlessly, creating a robust and scalable application. However, maintaining resilient communication between these services can be a challenge. This is where message brokers come into play, and Cloud Pub/Sub is a prime example.

What is Cloud Pub/Sub?

Cloud Pub/Sub is a fully managed message broker offered by Google. It enables teams to set up communication channels between services easily, without worrying about individual service uptime, retry policies, and idempotency. In this article, we’ll explore how to implement inter-service communication using Cloud Pub/Sub in Node.js.

Prerequisites

To follow along, you’ll need:

• Basic knowledge of Node.js
• Yarn or NPM installed (we’ll be using Yarn)
• Git installed
• Ngrok installed
• A Google account
• A development machine configured to run Node.js

Setting Up a Service Account for Pub/Sub on GCP

To initialize Pub/Sub in our Node app, we need to set up a service account with Pub/Sub access on our GCP console. Here’s how:

1. Log in to the Google Cloud Console and select a project or create one if it’s your first time.
2. Navigate to the Pub/Sub section to enable the API for your project.
3. Head over to the Service Accounts section to select a project and create a service account.
4. Grant Pub/Sub admin access to the service account.
5. Click on Create Key, select the JSON option, and download the JSON object.

Defining Key Terms

Before we dive deeper, let’s define some essential terms:

• Message: The data entity relayed to subscribing services on a particular topic.
• Topic: A theme that represents a feed of messages.
• Subscription: A coalition of subscriber entities that receives published messages on a particular topic.
• Subscriber: An entity that receives and acknowledges messages from a particular topic.
• Publisher: An entity that creates and broadcasts a message to subscribing services on a topic.

Understanding Message Flow

To understand how a message is passed from a publisher to subscriber, let’s take a look at a flow diagram from the Pub/Sub documentation:

1. The publisher application sends a message to a topic.
2. Pub/Sub ensures messages are retained in storage until services subscribing to the topic acknowledge message receipt.
3. Pub/Sub fans out the message to all subscribing entities in the topic.
4. A subscriber receives the message from Pub/Sub, either by pushing to a subscriber-configured endpoint or by pulling the message from Pub/Sub.
5. The subscriber sends an acknowledgment to Pub/Sub for each message received and removes the acknowledged message from the message queue.

Problem Domain and Proposed Solution

Let’s consider an e-commerce software solution with multiple services: an order service, delivery service, and notification service. The order service receives the user’s order, processes it, and sends it to the delivery service for processing. The delivery service then processes it and notifies the notification service of the delivery status.

Implementing this software architecture using regular HTTP calls between services can lead to system anomalies when edge cases like service downtimes or adding more services are introduced. Cloud Pub/Sub can act as a relay between services, ensuring message retry policies and making it possible for teams to add more services to the application stack.

Building the Publisher and Subscriber

We’ll build out our publisher and subscribers (push and pull) logic, containing three entry-point files: orders.js (publisher), delivery-sub.js (subscriber A), and notification-sub.js (subscriber B). Each service has its respective routes and controllers but shares the same pub-sub repository.

Pub/Sub Repository

This is where we define all the functions that enable us to carry out Pub/Sub-related tasks, such as publishing a message and listening for push or pull subscriptions.

Building the Publisher

The base logic for our publisher lies in the src/controllers/orders-controllers.js file. This acts as the orders service, which accepts orders from users, processes the order, and then sends a message to concerned services (delivery and notifications) notifying them of a new order.

Building the Subscribers

We’ll build the delivery service alone in this section, as the same steps can be recreated for the notifications service.

Connecting the Dots

We’ll create a topic and subscribers in the Google Cloud Console. We’ll also learn how to run our three services respectively for a proof of concept.

Running Our Services

To demonstrate three microservices, we created three entry-points in our project directory. These files have already been created and bootstrapped in our project repository.

Before running our application, we need to create our.env file and copy in our service account key into our project directory.

Conclusion

In this tutorial, we’ve learned what Cloud Pub/Sub is and how to build a simple case to demonstrate its usage in a service-oriented/microservice architecture. You can visit the official documentation for more information on Cloud Pub/Sub. The source code for this tutorial is available on GitHub.