How does AI crawler verification work?
Crawler verification resolves each visit into one of three states: verified, spoofed, or unverifiable. We check the request against the crawler operator's published address ranges and reverse DNS. A user agent string alone never earns verified status, because that string is trivially copied by anyone who wants to look like a real engine.
Verified means the network evidence agrees with the claimed identity. Spoofed means it actively disagrees. Unverifiable means we could not establish either, which is a real and common third answer.
We report the third state rather than folding it into one of the first two. Collapsing unverifiable into verified would flatter the numbers; collapsing it into spoofed would invent an attack that may not exist.
What this answer is based on
- Crawler verification method and states
Last updated August 3, 2026