Robots.txt checker: what are crawlers allowed to fetch?
Enter a site or a specific URL. We read its robots.txt, list the rules and any problems, and tell you if that URL is blocked for Googlebot and for other crawlers.
Check robots.txt
Live checkTurn any long URL into a short, trackable link.
Once the destination checks out, shorten it with click analytics by source, location and device, all in one dashboard.
What robots.txt does and doesn't do.
A request to crawlers, published at the root of a site.
robots.txt is a plain text file at the root of a site, such as example.com/robots.txt. It lists which paths crawlers may or may not fetch, grouped by crawler name, and can point to your sitemap.
It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results if other sites link to it, just without a description. To keep a page out of search results, let crawlers fetch it and mark it noindex.
The file is a convention that well-behaved crawlers follow. It isn't access control, and it shouldn't be used to hide private content.
Directives
What each line does.
| Directive | Meaning | Example |
|---|---|---|
| User-agent | Starts a group of rules for a crawler. * means all. | User-agent: Googlebot |
| Disallow | A path the crawler shouldn't fetch. Empty allows everything. | Disallow: /private/ |
| Allow | An exception to a broader Disallow. The longest matching rule wins. | Allow: /private/help |
| Sitemap | The full URL of a sitemap. Can appear anywhere in the file. | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests. Bing and others honour it, Google ignores it. | Crawl-delay: 10 |
Common questions
Does Disallow remove a page from Google? +
No. It stops crawlers fetching the page, but the URL can still be indexed from links. Use a noindex directive on a crawlable page to remove it.
What happens if there is no robots.txt? +
Crawlers assume everything may be fetched. A 404 for robots.txt is normal and harmless.
Are the rules case-sensitive? +
Paths are, so /Admin and /admin are different. Directive names and user-agent names are not.
How do wildcards work? +
* matches any sequence of characters and $ marks the end of the URL. For example Disallow: /*.pdf$ blocks URLs ending in .pdf.