Question 1

Is crawled but not cited worse than not being crawled at all?

Accepted Answer

It is generally better news, because the harder half of the problem is already solved. Not being crawled points at robots.txt rules, a blocked user agent, an orphaned URL, or a rendering issue — infrastructure work. Crawled but not cited points at the page itself, which is usually a rewrite you can do this week. It is also easier to verify afterwards, since you already know the crawler visits that URL.

Question 2

Why would an engine fetch a page and then not use it?

Accepted Answer

The usual reasons are extraction problems rather than quality judgements. The direct answer is buried below a long introduction, the substance lives in an image or a chart with no text equivalent, the quotable summary is inside a collapsed element, the claims are vague enough that nothing is safely quotable, or a competing page states the same thing far more plainly. Assistants prefer passages they can lift with low risk of being wrong, so an unhedged, specific, early sentence beats a well-argued page that never quite commits.

Question 3

Does fixing a crawled-but-not-cited page make an engine cite it?

Accepted Answer

It cannot be promised. Making the answer explicit, early and extractable removes a known obstacle, and it is a reasonable input to test — but engines weigh source authority, freshness, and what competitors published, none of which you control. The honest framing is that the edit improves your odds; the way to find out is to re-ask the same question after the engine has re-read the page and compare, treating any single comparison as one directional sample.

Question 4

How do I know a page was crawled by an AI bot at all?

Accepted Answer

Look in your server or CDN access logs for the published AI crawler user agents — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google’s AI training access. Filter by the exact URL, not the domain, because whole-site totals hide the fact that your key page is never fetched. Most hosting platforms and CDNs expose this without extra tooling; if yours does not, an edge log drain or an analytics rule keyed on user agent will.

Question 5

Which pages should I check for this first?

Accepted Answer

Start with the pages that map to questions you already track and already lose. Those are the ones where the mismatch between “we cover this” and “we are never credited for it” is provable rather than assumed. Commercial comparison pages and definitional pages tend to surface first, because they are the shapes assistants are most often asked to summarise, and they are also where buried answers and collapsed summaries are most common.

Question 6

What does the fix usually look like in practice?

Accepted Answer

Lead with a short, direct answer to the exact question in the first screenful, before any narrative. Convert any substance that lives only in an image, chart or table into text. Expand anything hidden behind a click, since content the crawler cannot reach without interaction may as well not exist. Add verifiable specifics — figures, dates, named constraints — because assistants favour passages they can attribute safely. FAQPage schema helps a parser find the Q&A block once the answer is actually there.

Question 7

How do I find these pages at scale rather than one at a time?

Accepted Answer

You need two joined datasets: AI crawler hits per URL from your logs, and per-question citation records from a tracker. Neither on its own identifies the class. At portfolio scale the join is worth automating — export crawler hits grouped by URL, export the credited URLs from your tracked answers, and look for URLs present in the first set and absent from the second for questions they clearly address.

Question 8

Does this apply to sites that block AI crawlers?

Accepted Answer

Not in the same way. If robots.txt or your CDN blocks GPTBot, ClaudeBot or their peers, the diagnosis is never-crawled by design, and no page rewrite will change it. That can be a deliberate, legitimate choice — some publishers block training access on purpose. The important thing is to know which situation you are in before spending on content, since the two look identical from a citation report alone.

Question 9

How often should the check be repeated?

Accepted Answer

Often enough to notice a regression, which in practice means on the same cadence as your citation tracking rather than continuously. Crawler behaviour and answer composition both shift week to week, so a page that is credited today can quietly stop being credited after a competitor publishes or an engine updates. A recurring check on a stable question set is more informative than an intensive one-off audit.

What is Crawled but Not Cited?

Crawled but Not Cited in context

Example

Why it matters in 2026

Related terms

Common questions about Crawled but Not Cited.

See which of your pages get read and passed over.