What Is robots.txt?
robots.txt is a small text file at the root of your website that tells search engine crawlers (like Googlebot) which URLs they may or may not crawl. It's a set of polite instructions, not a security tool — a blocked page can still be linked and visited by people.
How It Looks
A basic robots.txt file is simple:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlThis allows all crawlers and points them to your sitemap.
How Should You Configure It?
Start with
Allow: /so nothing important is blockedBlock only what should stay out of search results, like admin areas or duplicate internal pages
Reference your sitemap so crawlers find it
Keep it minimal — the more rules, the more chances to make a mistake
Common Mistakes
Blocking CSS or JavaScript: this can break rendering and hurt rankings.
Blocking the whole site: a stray
Disallow: /removes you from Google.Using it for security: robots.txt is public — anyone can read it. Private files need real access control.
Do You Need to Create One?
If your platform generates a sensible default (most do), leave it alone. You only need to edit robots.txt when you have a specific reason to restrict crawling — otherwise, simple is safe.
The Bottom Line
robots.txt is a small but powerful file. Configure it to allow crawling, point to your sitemap, block only what truly needs blocking — and never use it as a security measure.