Unlock the Power of Search in Your Go Application with Elasticsearch

Getting Started

To follow along, you’ll need:

• Go (version >= 1.14) installed on your machine
• Docker and docker-compose installed
• Familiarity with Docker and the Go programming language

Create a new directory for your project and initialize a new Go module. Install the required dependencies, including the PostgreSQL driver, Elasticsearch client, HTTP framework, and logger.

Project Structure

Organize your project directory with the following structure:

cmd: Application binaries
db: Database package and migration files
.env: Environment variables
handler: API route handlers
logstash: Logstash pipeline configurations and Dockerfile
models: Golang structs for JSON objects

Database Setup

Create a .env file with your environment variables, including database credentials. Set up the Post struct and database connection in db/database.go. Implement database operations for the posts and post_logs tables in db/posts.go.

Database Migrations

Use golang-migrate to manage database migrations. Create migration files for the posts and post_logs tables. Apply the migrations to create the tables.

Elasticsearch and PostgreSQL as Docker Containers

Create a docker-compose.yml file to declare the services needed for your application, including PostgreSQL, API, and Elasticsearch. Set up the Dockerfile to build your application.

version: '3'
services:
  db:
    image: postgres
    environment:
      - POSTGRES_USER=myuser
      - POSTGRES_PASSWORD=mypassword
      - POSTGRES_DB=mydb
    volumes:
      -./db/data:/var/lib/postgresql/data
  elasticsearch:
    image: elasticsearch:7.9.2
    environment:
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      -./elasticsearch/data:/usr/share/elasticsearch/data
  api:
    build:.
    ports:
      - "8080:8080"
    depends_on:
      - db
      - elasticsearch
    environment:
      - DB_HOST=db
      - DB_PORT=5432
      - ELASTICSEARCH_URL=http://elasticsearch:9200

Route Handlers with Gin

Create route handlers for creating, reading, updating, and deleting posts, as well as searching posts using Elasticsearch. Implement the CreatePost and SearchPosts functions.

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/olivere/elastic"
)

func CreatePost(c *gin.Context) {
    // Create a new post
}

func SearchPosts(c *gin.Context) {
    // Search for posts using Elasticsearch
    es, err := elastic.NewClient(elastic.SetURL("http://elasticsearch:9200"))
    if err!= nil {
        // Handle error
    }
}

Sync Database to Elasticsearch with Logstash

Configure Logstash to use the PostgreSQL database as its input and Elasticsearch as output. Set up the Logstash pipeline to sync the database with Elasticsearch.

input {
  jdbc {
    jdbc_connection_string => "jdbc:postgresql://db:5432/mydb"
    jdbc_user => "myuser"
    jdbc_password => "mypassword"
    jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.18.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    statement => "SELECT * FROM posts"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "posts"
  }
}

Building the API Binary

Create the main.go file in cmd/api to set up the logger, database connection, and Elasticsearch connection. Initialize the route handler and start the API server.

package main

import (
    "github.com/gin-gonic/gin"
    _ "github.com/lib/pq"
    "github.com/olivere/elastic"
)

func main() {
    // Set up logger, database connection, and Elasticsearch connection
    r := gin.Default()
    r.POST("/posts", CreatePost)
    r.GET("/posts/search", SearchPosts)
    r.Run(":8080")
}

Testing the Search Application

Rebuild and start the docker-compose services. Create some posts using your favorite API testing tool. Test the search endpoint to see the results.

Visualize Elasticsearch with Kibana

Add the Kibana service to your docker-compose file. Access the Kibana dashboard to visualize and explore your Elasticsearch data.

kibana:
  image: kibana:7.9.2
  ports:
    - "5601:5601"
  depends_on:
    - elasticsearch
  environment:
    - ELASTICSEARCH_URL=http://elasticsearch:9200