Yellow Toolsby Yellow Solutions

Robots.txt Generator

Create clear crawler instructions, catch common formatting mistakes, and download a ready-to-review robots.txt file.

  • Free to use
  • Runs in your browser
  • No website crawling

Crawler rules

Choose what crawlers may request

Create one group for each crawler that needs a different set of rules. Use * for all crawlers.

Crawler group 1

Result

Generated robots.txt

Review every rule before uploading this file to the root of your website.

Your result will appear here

Add your rules, then choose Generate robots.txt.

What does a robots.txt file do?

A robots.txt file gives automated crawlers instructions about which URL paths they may request from a particular website host. Search engines normally check this public file before crawling pages and other files.

The file must be named robots.txt and placed at the root of the website, such as https://example.com/robots.txt. Rules only apply to the same protocol, host, and port where that file is published.

How to use this generator

  1. Enter * to address all crawlers, or enter a supported crawler name such as Googlebot.
  2. Add an Allow or Disallow rule and enter a website path beginning with /.
  3. Add another crawler group when a bot needs different rules.
  4. Optionally enter the complete URL of your sitemap.
  5. Generate the file, review it, then copy or download it.

Allow and Disallow explained

Disallow: /private/ asks a matching crawler not to request URLs beginning with /private/. An Allow rule can make a more specific path crawlable inside a broader disallowed area. When rules conflict, crawlers generally use the most specific matching path.

Important limitations

  • Robots.txt is crawler guidance, not authentication or access control.
  • A blocked URL may still appear in search results when other pages link to it.
  • Do not list confidential paths as a substitute for protecting them with passwords and server permissions.
  • An accidental Disallow: / can stop a crawler from requesting the entire site.
  • Different crawlers may support additional non-standard directives differently.

Frequently asked questions

Does robots.txt remove a page from Google?

No. Blocking crawling is not a guaranteed removal method. Use the appropriate noindex or access-control method for your situation, and make sure the crawler can access a noindex directive when required.

Can robots.txt protect an admin area?

No. The file is public and does not block visitors. Protect private areas with authentication and server-side authorization.

Do I need a sitemap line?

It is optional, but a complete sitemap URL can help supported crawlers discover the sitemap associated with the site.

Need technical SEO help?

A robots.txt file is only one part of crawl management. Yellow Solutions can help review site structure, indexability, metadata, internal links, and technical SEO issues as one connected system.