Robots.txt Generator

A robots.txt file tells search engine crawlers which parts of your site they may request. It sits at the root of your domain, at yoursite.com/robots.txt, and it is the first file most crawlers look for before they fetch anything else. This tool generates a valid file for the most common case: allow every crawler, and point them at your sitemap so they can find your pages efficiently. That is genuinely what the majority of sites need. What robots.txt does not do is the single most misunderstood thing about it, and getting it wrong can remove pages from search or, worse, trap pages in search that you wanted gone. It controls crawling, not indexing, and the difference is the whole subject of the sections below. The file generates in your browser and nothing is sent anywhere.

How to Use the Robots.txt Generator - Step by Step

1. Choose "Default Access". "Allow All Robots" is selected by default and is the right choice for almost every site. It produces a file that lets crawlers request any page, which is what you want when your goal is to be found. Only choose "Disallow All Robots" for a site that should not appear in search at all, such as a private staging environment, and understand that this blocks crawling rather than guaranteeing removal from search.
2. Enter a "Crawl Delay" only if you have a specific reason. This asks crawlers to wait the given number of seconds between requests, which can reduce server load on a small host during heavy crawling. Leave it blank unless your host is struggling, because it slows how quickly your new pages are discovered. Note the important limitation covered below: Google ignores this directive entirely.
3. Enter your "Sitemap URL". This is the most valuable field. Point it at your full sitemap, for example https://yoursite.com/sitemap.xml, so crawlers can find every page you want indexed rather than relying on discovering them through links. Use the absolute URL including https, not a relative path.
4. Click "Generate Robots.txt". The tool assembles the directives and displays the file in a text box.
5. Read the output before using it. For the allow-all case it will show "User-agent: *" followed by "Disallow:" with nothing after the colon. That empty Disallow is correct and deliberate: it means disallow nothing, which is to say allow everything. It is not a mistake or an unfinished line.
6. Copy the file and save it as robots.txt, all lowercase. The filename must be exactly that.
7. Upload it to the root of your domain so it resolves at yoursite.com/robots.txt. It will not work in a subfolder. On most hosts this is the public root directory, the same place your homepage lives.
8. Verify it is live by visiting yoursite.com/robots.txt in a browser. Then check it in Google Search Console under the robots.txt report, which shows how Google reads it and flags any errors.

The Directives, and What Each One Means

The most important thing to understand about robots.txt is what it is not. It is a crawling directive, which means it tells cooperating crawlers which URLs they may request. It is not an indexing directive, and it does not control whether a page appears in search results. These sound like the same thing and are not. A page blocked in robots.txt can still appear in Google search results if other sites link to it, because Google can list a URL it has never crawled, showing the link with no description. Worse, and this is the trap that catches people, blocking a page in robots.txt prevents Google from removing it. If you have a page you want gone from search, the correct method is a noindex meta tag or an X-Robots-Tag header on the page. But Google can only see that noindex tag if it is allowed to crawl the page and read it. Block the page in robots.txt and Google never fetches it, never sees the noindex, and the page can linger in results indefinitely. The rule is counterintuitive but critical: to remove a page from search, allow crawling and add noindex. To keep something genuinely private, use authentication, because robots.txt is a public file that anyone can read and that malicious bots ignore completely. One more limitation to internalise. robots.txt is a set of requests, not enforcement. Googlebot, Bingbot, and other reputable crawlers honour it. Scrapers, spam bots, and anything malicious read it, sometimes specifically to find the folders you tried to hide, and then request them anyway.

Robots.txt Generator - Worked Examples

Who Uses the Robots.txt Generator?

Common Robots.txt Mistakes to Avoid

Robots.txt vs Noindex vs Authentication

These three are constantly confused, and choosing the wrong one is why pages get stuck in or fall out of search. Each does a different job. This table is the quickest way to pick the right one for what you actually want.

The key relationships, verified against Google and Bing documentation. An empty Disallow directive means allow everything; Disallow: / means block everything. Google publicly documents that it does not support the Crawl-delay directive and manages crawl rate itself, while Bing and Yandex do honour it. A URL blocked by robots.txt can still be indexed if linked externally, and blocking a URL prevents Google from reading a noindex tag on it, which is why removal requires allowing the crawl. robots.txt must resolve at the domain root to have any effect.