Checking it worked

01

Start with the server log

The access log is the only record of what actually happened. It shows which agent requested which path, when, and what status it received. Analytics will not show you this, because analytics runs JavaScript and crawlers generally do not, so a crawler visit is invisible to it by design.

The useful first pass is simply to count requests by user agent over a month and look for the tokens from the first chapter. Three outcomes are informative. Seeing a crawler you intended to block means the rule is not reaching it, and the group-matching trap from the previous chapter is the first thing to check. Seeing nothing at all from a crawler you allowed usually means it has not found reason to fetch you rather than that something is broken. Seeing a crawler hit paths you thought were denied is the specific signature of the named-group problem.

If your host does not give you raw logs, the equivalent is usually available in a CDN dashboard, which will typically break traffic down by user agent. What matters is that it is request-level data from the server side rather than anything measured in a browser.

• Count requests by user agent over a month, not a day. Crawl frequency is lumpy
• Check status codes too. A crawler receiving 403 or 404 is a different problem from one being denied
• Compare the paths fetched against what you intended to allow
• A crawler absent entirely is normal, and is not evidence of a misconfiguration

02

Do not trust the user agent string

A user agent is self-reported text. Anything can claim to be GPTBot, and scrapers routinely do, because it is a single header and there is no barrier to setting it. Treating that string as identity will give you a picture of AI crawler activity that is partly fiction, and it inflates in the direction people want to believe.

The check that works is to verify the requesting IP address against the ranges the operator publishes. The major operators publish these deliberately so that site owners can do exactly this. The workflow is unglamorous and reliable: take the IPs behind the requests claiming a given token, check them against the published ranges, and discard the ones that do not match. If a large share fails, you have learned something more useful than the original count.

This matters beyond tidiness. Decisions get made on these numbers, and a report claiming heavy AI crawler interest that turns out to be scrapers wearing a borrowed name is worse than no report, because it is confident.

03

What cannot be verified, and what to do instead

Google-Extended is the clean example. Google states it has no separate user agent string and that the robots.txt token is used in a control capacity, so there is nothing to find in a log. The directive either works as documented or it does not, and no amount of log analysis will tell you which. This is worth knowing before someone spends a day looking.

The larger thing you cannot check from logs is whether you were actually named in an answer. A retrieval crawler fetching your page does not mean the answer used it, and an answer naming you does not always involve a fresh fetch. The two are related but neither implies the other, so a log cannot settle it.

The honest method for that question is to run a fixed set of prompts on a schedule and record what comes back: whether you were named, how you were described, and which sources the answer drew on. It is manual and it is a weaker instrument than it appears, because answers to identical questions vary between runs. What makes it worth doing is the description rather than the count. Being named inaccurately is a fixable problem, and you can only fix it if you know about it.

04

Finding the traffic that did arrive

When an assistant links you and someone clicks, that visit lands in your analytics as an ordinary referral, with the assistant’s domain as the referrer. No special integration is required to see it, and no tool is required to segment it. Building a segment or filter for the assistant hostnames gives you the same view a paid dashboard would sell you.

Two cautions make the numbers honest. The volumes are usually small compared with search, and a low number is not evidence that something is broken. And some assistant traffic arrives without a referrer at all, for instance from a desktop application rather than a browser, which means what you can see is a floor rather than a total. Report it as a floor.

The pattern worth watching is not the daily figure but the direction over months, alongside what the prompt log says about how you are being described. Traffic tells you something arrived. The prompt log tells you what was said. Neither is sufficient alone, and together they are about as good as this gets without inventing precision.