Fastly real-time logging
Stream a Fastly service's request logs to Heading with an HTTPS logging endpoint and a JSON log format
Fastly streams logs to an HTTPS endpoint, in a format you write yourself, with a custom header you set. That makes it configuration only, and it means you decide exactly which fields leave your edge.
Official reference: Log streaming: HTTPS and Useful log formats.
INGEST_HOST is a placeholder
INGEST_HOST stands in for the ingest hostname, which is not final yet. The setup dialog in Heading shows the real endpoint and property id. Copy them from there and replace INGEST_HOST everywhere it appears on this page.
Before you start
Get the endpoint and the bearer token from Heading. Open Tracking → Outcomes, find the Crawl activity card and select Set up. Treat the token as a server-side credential.
Create the logging endpoint
Open the service configuration
In the Fastly web interface, select the service, then Edit configuration, and clone the active version. Go to Logging, choose HTTPS, and select Create endpoint.
Write the log format
Paste this into Log format, as a single line. One request produces one JSON object.
{"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S%z}t","host":"%{json.escape(req.http.host)}V","path":"%{json.escape(req.url.path)}V","method":"%m","statusCode":%>s,"userAgent":"%{json.escape(req.http.User-Agent)}V","referer":"%{json.escape(req.http.Referer)}V","ip":"%h"}Keep it on one line. A newline inside the format splits one record across several lines, and every one of them is then unreadable.
json.escape is doing real work here. A user agent containing a double quote, and there are plenty, produces a broken line without it. req.url.path is the path without the query string, which is what Heading stores anyway.
Fill in the endpoint fields
| Field | Value |
|---|---|
| Name | Heading crawl activity |
| URL | https://INGEST_HOST/api/ingest/crawl/{propertyId} |
| Content type | application/x-ndjson |
| Method | POST |
| JSON log entry format | Newline delimited |
| Custom header name | Authorization |
| Custom header value | Bearer YOUR_TOKEN |
| Placement | Format Version Default |
Leave Maximum logs and Maximum bytes on their defaults. They control how much Fastly batches into one request, and the defaults are fine.
Array of JSON also works if you prefer it, since the endpoint accepts a JSON array batch. Set the content type to application/json if you choose it.
Do not sample and do not filter
Log every request. Heading counts crawler requests, and a sampled stream produces a number that looks plausible and is wrong, with no way to correct it afterwards.
If you are tempted to wrap the logging call in VCL that fires only for bot user agents, do not. Heading filters server-side, and the human traffic in this stream is what produces the page inventory and every AI-referred entry.
Activate the version
A logging endpoint on a draft version sends nothing. Activate the new version, then open the setup dialog in Heading and wait for the first delivery.
What Heading stores
The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.
| Data | Stored |
|---|---|
| Request path | Yes, with the query string stripped before anything is written |
| Host | Yes, and it has to match the property's domain |
| User agent | Yes. It is how a crawler is identified |
| Status code | Yes. What your origin served the crawler is half the value |
| Time of the request | Yes, as a UTC timestamp |
| Referer | Yes, matched against the AI platforms Heading already knows |
| IP address | No. Checked against the operator's published ranges at ingest, then discarded |
| Cookies | No. Never read, never stored |
| Request or response body | No. Never read, never stored |
| Query string | No. Stripped from the path on arrival |
On this recipe the log format is the whole payload, so the fields above are the only fields that ever leave your edge. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.
Check it worked
Setup is not finished when you activate the version. It is finished when a real event lands.
Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.
| Status | Meaning |
|---|---|
| Not verified | Nothing has arrived yet. Setup is not finished |
| Connected | Batches are arriving and the most recent one was accepted |
| Rejecting | Batches are arriving and every one is being turned away. Nothing is being recorded |
| Nothing arriving | Batches arrived before and none has arrived for 24 hours. Check the sender at your edge is still pointed here |
| Disconnected | Someone switched ingestion off for this property in Heading. Anything still arriving is refused with a 403 and nothing from it is stored. Reconnect from the same dialog to start accepting events again |
An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.
Troubleshooting
Nothing is arriving. The commonest cause is a version that was never activated. Check the active version has the endpoint on it. Then check the service is the one serving this domain, and give Fastly a few minutes to fill a batch on a quiet service.
Events arrive and all of them are rejected. A 401 is the token. Check Custom header name is exactly Authorization and Custom header value is Bearer then a space then the token. Tokens are per property: the token for one property is rejected by another property's endpoint, and the property id in the URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately.
The delivery log shows lines arriving but nothing readable. The log format is the thing to check. Paste it again as one line, and keep json.escape around the user agent and referer. A single unescaped quote in a user agent breaks the line it appears in.
host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. A Fastly service often fronts several hostnames. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Either connect the other hostname as its own property in Heading with its own endpoint and token, or add a condition on the logging endpoint so it fires only for req.http.host matching the property's domain. Filtering on host is fine here. Filtering on user agent is not.