What Is robots.txt and How Should You Configure It?

What Is robots.txt?

robots.txt is a small text file at the root of your website that tells search engine crawlers (like Googlebot) which URLs they may or may not crawl. It's a set of polite instructions, not a security tool — a blocked page can still be linked and visited by people.

How It Looks

A basic robots.txt file is simple:

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

This allows all crawlers and points them to your sitemap.

How Should You Configure It?

• Start with Allow: / so nothing important is blocked

• Block only what should stay out of search results, like admin areas or duplicate internal pages

• Reference your sitemap so crawlers find it

• Keep it minimal — the more rules, the more chances to make a mistake

Common Mistakes

• Blocking CSS or JavaScript: this can break rendering and hurt rankings.

• Blocking the whole site: a stray Disallow: / removes you from Google.

• Using it for security: robots.txt is public — anyone can read it. Private files need real access control.

Do You Need to Create One?

If your platform generates a sensible default (most do), leave it alone. You only need to edit robots.txt when you have a specific reason to restrict crawling — otherwise, simple is safe.

The Bottom Line

robots.txt is a small but powerful file. Configure it to allow crawling, point to your sitemap, block only what truly needs blocking — and never use it as a security measure.