Question 1

What is robots.txt and what is it used for?

Accepted Answer

robots.txt is a plain-text file at the root of a domain that tells web crawlers which paths they can or cannot fetch. It's used to keep crawlers out of admin/auth/dashboard paths, block SEO-competitor crawlers from scraping your site, and make explicit allow/block decisions for AI bots like GPTBot and ClaudeBot — but it does NOT control whether a page can be indexed, only whether it can be crawled.

Question 2

Does robots.txt prevent indexing?

Accepted Answer

No — it prevents crawling. A blocked URL can still appear in search results if Google discovers the URL through external links (indexed without content). To remove from the index, use a noindex meta tag or HTTP header — and don't block the URL in robots.txt, because Google needs to crawl the page to see the noindex directive.

Question 3

What happens if I block a noindex page in robots.txt?

Accepted Answer

You create the exact trap the previous answer warns about, in reverse: if you Disallow a URL in robots.txt AND put a noindex tag on that page, Google can never crawl the page to see the noindex directive — so instead of being removed from the index, the URL can sit indexed with no content shown, indefinitely. The correct order is: let the page be crawled with noindex in place, wait for Google to process it and drop it from the index, THEN block it in robots.txt if you want to stop future crawl attempts.

Question 4

What's the difference between Allow and Disallow?

Accepted Answer

Disallow tells crawlers not to fetch matching paths. Allow overrides a Disallow for a more specific path (e.g., Disallow: /admin/ + Allow: /admin/public/). Most engines support both; legacy bots may only honor Disallow.

Question 5

Should I block AI bots?

Accepted Answer

Depends on your business model. Marketing-led companies should allow GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — being cited by AI engines is a top-of-funnel win. Content-licensing companies (publishers, paid courses) often block them to protect IP. SearchChamp defaults to allow on the marketing surface.

Question 6

How do I test robots.txt?

Accepted Answer

Google Search Console has a robots.txt tester. For each User-Agent, paste a URL and see whether it's blocked or allowed. SearchChamp's Site Auditor parses your robots.txt and flags syntax errors, conflicts, and blocked critical paths (e.g., accidentally blocking /sitemap.xml).

Question 7

How do I test robots.txt without any tools?

Accepted Answer

Fetch it directly — curl https://example.com/robots.txt or just visit the URL in a browser, since it's a plain-text file with no authentication. Read the rules manually: find the block matching the crawler's User-Agent (or the wildcard * block if there's no exact match) and check whether your target path falls under a Disallow rule. It's tedious for a large file with many User-Agent blocks, which is why an automated parser is worth it past a handful of rules.

Question 8

Do web crawlers have to respect robots.txt rules?

Accepted Answer

No — it's a voluntary convention, not an enforcement mechanism. Reputable crawlers (Googlebot, Bingbot, the major AI bots) honor it by policy, but robots.txt has no technical way to block a crawler that chooses to ignore it. If you need to actually prevent access rather than politely request it, that requires server-side blocking (IP allowlisting, authentication, a firewall rule) — robots.txt alone won't stop a bad actor.

Question 9

Can robots.txt be bypassed or overcome by web scrapers?

Accepted Answer

Yes, trivially — since it's a voluntary, unenforced convention (see the previous answer), any scraper that doesn't care about following the rules simply ignores the file. robots.txt is a signal for well-behaved crawlers, not a security control. Don't rely on it to protect sensitive content; use authentication or server-side access rules for anything that actually needs to be inaccessible.

Question 10

Where exactly should robots.txt be served?

Accepted Answer

At the root: example.com/robots.txt. Not /sub/robots.txt. Not subdomain.example.com/robots.txt unless that subdomain is a separate hostname (then each subdomain needs its own robots.txt).

Question 11

Does having a robots.txt file create a security vulnerability?

Accepted Answer

Indirectly, yes, in one specific way: because robots.txt is a public, unauthenticated file, listing a sensitive path in a Disallow rule to "hide" it from crawlers actually advertises that path's existence to anyone who reads the file. Don't use robots.txt as an obscurity measure for admin panels or staging environments — those need real authentication, not a polite request not to crawl them.

Question 12

Does SearchChamp's Site Audit check my robots.txt for AI-bot-specific issues, not just traditional crawler rules?

Accepted Answer

Yes — Site Audit parses your robots.txt for syntax errors, rule conflicts, and accidentally blocked critical paths across all User-Agent blocks, including the AI bot entries (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), not just the legacy Googlebot/Bingbot rules.

What is robots.txt?

robots.txt in context

Example

Related terms

Common questions about robots.txt.

Catch robots.txt mistakes before crawlers do.