Unlocking the Power of Robots.txt in Next.js

What is a Robots.txt File?

A robots.txt file is a web standard file that communicates with search engine crawlers, like Google bots, to specify which pages can be crawled and which should be ignored. It’s typically located at the root of your domain and can be accessed via the URL yourdomain.com/robots.txt. This file allows you to control what parts of your application are crawlable, ensuring that sensitive areas, such as admin pages, are not indexed by search engines.

Adding a Robots.txt File to Your Next.js Application

Adding a robots.txt file to your Next.js application is straightforward. Every Next.js project comes with a public folder, which allows you to store static assets that can be accessed from the root of your domain. To add a robots.txt file, simply create a new file named robots.txt in the public folder.

mkdir public
touch public/robots.txt

Alternatively, you can dynamically generate your robots.txt file using Next.js’s API routes and rewrites features. Create a robots.js file in the pages/api folder, and add a handler that returns your robots.txt content.

import { NextApiRequest, NextApiResponse } from 'next';

const robotsTxt = `
User-agent: *
Disallow: /admin
`;

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  res.setHeader('Content-Type', 'text/plain');
  res.write(robotsTxt);
  res.end();
}

Then, add a rewrite rule in your next.config.js file to redirect requests for /robots.txt to /api/robots.

module.exports = {
  //...
  rewrites() {
    return [
      {
        source: '/robots.txt',
        destination: '/api/robots',
      },
    ];
  },
};

Validating Your Robots.txt File

Once you’ve deployed your application to production, you can validate your robots.txt file using Google Search’s tester tool. If your file is valid, you should see a message indicating 0 errors.

In addition to validating your robots.txt file, consider setting up a sitemap generator in Next.js to further optimize your website’s SEO. This will help search engines understand the structure of your website and improve your page ranking.