Crawl activity

AI crawlers fetch your pages. None of those requests reach Google Analytics, because a crawler runs no JavaScript and Analytics needs JavaScript to run. Your edge or server logs are the only place the requests exist.

Crawl activity is Heading reading those logs. You point one log source at one endpoint, Heading filters and verifies what arrives, and the Crawl activity section on Tracking → Outcomes fills up.

What connecting a log source gives you

Three things, from one stream.

Crawl activity. Which AI crawlers fetched which pages, what status code your own origin served them, and whether the request really came from the operator its user agent claimed. A 403 served to a retrieval crawler is the single finding this feature exists to surface.

AI-referred entries. The same logs carry human arrivals with an AI referer. Analytics misses those it cannot run JavaScript for. The difference between the two figures is dark traffic, and it is measured rather than estimated.

A page inventory that costs no requests. Because the logs carry your human traffic too, Heading learns which pages exist and which are actually used. "This page takes human traffic every week and no retrieval crawler has ever fetched it" needs no outbound crawl of your site to say.

What it is not

Crawl activity does not predict visibility, and Heading will not tell you it does. It is a precondition and a fault signal. Retrieval precedes a citation, so a page no retrieval crawler ever fetched cannot be quoted in an answer, but a page that was fetched is not thereby cited.

Heading also reads no robots.txt and fetches no sitemap. That means it can say GPTBot has not visited in 40 days, and it cannot say why. That is a stated limit, not a gap we are about to close.

The three crawler functions

Every crawler in the catalogue is exactly one of these, and the three are never blended into one number or one line on a chart. A blended line moves for reasons the chart cannot separate.

Two names you will not see in your logs: Google-Extended and Applebot-Extended are robots.txt control tokens with no crawler behind them. Google states that Google-Extended has no separate request user agent, Apple states that Applebot-Extended does not crawl webpages. Heading keeps them in the catalogue and never matches a log line to them, because a catalogue that matched them would report crawls that did not happen.

Verified, unverified, unverifiable

Anyone can put GPTBot in a user agent string. Heading checks the source IP of every crawler request against the ranges the operator publishes, and then discards the IP.

Three states, never two. Reporting an unverifiable bot as unverified accuses it of spoofing on no evidence. Reporting it as verified launders it. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Mistral, DuckDuckGo and Common Crawl publish ranges. ByteDance, Meta and Amazon publish none, so their crawlers are permanently unverifiable and the page says so.

If your log source cannot send the client IP, or you choose not to send it, everything arrives as unverifiable. Nothing else breaks.

Entries and sessions are different units

A log-derived AI-referred entry is one arrival. A GA4 AI-referred session is a session. They come from different sources, they count different things, and Heading never sums them, averages them, or puts them in the same tile. The gap between them is the point: it is the traffic Analytics cannot see.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Choose a recipe

Every recipe speaks the same endpoint. Pick the one that matches where your requests are already being logged.

Two rules that apply to every recipe

Send 100% of requests. No sampling, at any layer. Heading counts crawler requests, and a 10% sample turns 40 visits from GPTBot into 4 with no way to correct it later: which requests were dropped is gone. A sampled stream is worse than none, because it produces a plausible number nobody questions.

Do not filter by user agent. Heading filters server-side, and it drops roughly a hundred lines for every one it keeps. That ratio is the whole reason ingest runs on its own infrastructure. Filtering at your end to "just the bots" also throws away the human traffic that produces the page inventory and every AI-referred entry, which is most of the value in the stream.

Filtering by host is different and sometimes necessary. See host_not_matched in the troubleshooting section of any recipe page.

Check it worked

Setup is not finished when you save the configuration. It is finished when a real event lands.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

Nothing is arriving. Most log sources send in batches on a timer, not per request, so wait a few minutes before deciding. Then check the endpoint URL character for character, including the property id, and check the source is actually enabled and pointed at production traffic.

Events arrive and all of them are rejected. A 401 is the token. Tokens are per property: the token for one property is rejected by another property's endpoint, and the property id in the URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately, so a rotation you forgot about looks exactly like a typo. There is no period in which both tokens work: update every sender carrying the old token as soon as you rotate, because events sent in between are refused and most log sources do not send them again. The delivery log records each refusal, so the gap is visible.

Every event is refused with a 403. Ingestion is switched off for this property: someone disconnected the connection in Heading. Nothing sent to a disconnected connection is stored, and the connection stays disconnected however many events arrive. Remove the sender at your edge, or reconnect the property from the same setup dialog, which starts the verification over.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. This is the normal case when one drain covers several sites, and when preview or deployment hostnames (*.vercel.app, *.netlify.app) are included. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Fix it by sending production traffic only, by filtering on host at the source where the source supports it, or by connecting the other site as its own property in Heading with its own endpoint and token.

Locked or lapsed properties stop ingesting. Crawl activity is a paid surface. If a property's subscription lapses, the endpoint stops accepting its events, the same way daily tracking stops collecting for it.