Vercel log drain

If your site runs on Vercel, this is configuration only. Vercel sends every request your project serves to an HTTP endpoint of your choosing, in NDJSON, with headers you set.

Official reference: Using Drains and the Log Drains reference.

Before you start

Drains are available to teams on the Vercel Pro and Enterprise plans. A Hobby team cannot create one, so use the custom REST recipe from inside your app instead.

You need two values from Heading. Open Tracking → Outcomes, find the Crawl activity card and select Set up. The first step holds the endpoint and the bearer token. The token is derived from the property, so there is nothing to store on our side and nothing per-property to leak. Treat it as a server-side credential.

Create the drain

The test Vercel runs when you save

Vercel will not save a drain whose endpoint fails its test, so it is worth knowing what that test is.

Vercel posts sample log events to your endpoint, through the delivery configuration you just filled in. Your Custom Headers go with it, so the request carries the same Authorization: Bearer header a real delivery carries and authenticates the same way. Heading answers it 202, which is the 2xx Vercel is looking for.

Nothing is stored from it. The sample events carry Vercel's own hostnames and timestamps from years ago, so the host check and the timestamp window each drop them, and a sample event never becomes a row in your inventory.

It appears in the delivery log in Heading as an accepted delivery with lines received above zero and lines kept zero. That row is useful: it is proof the endpoint, the property id and the token are all right, before a single real log line has been drained. It does not finish setup. The connection stays Not verified until a real event lands, because a test payload is not evidence your site's traffic is reaching us.

A request that carries no lines at all is answered the same way, as an empty batch with received zero, and is recorded as an accepted delivery. Only a body in some other format, an HTML error page or plain text, is refused with a 400.

What Vercel sends

Vercel delivers newline-delimited JSON, one log object per line, batched. Heading reads the proxy object on request lines and ignores everything else.

Retried deliveries

Heading can recognise a repeat of a batch it has already stored, if the sender names the batch in an Idempotency-Key header. A Vercel log drain cannot do that. The Custom Headers box sets the same headers on every delivery, so there is no way to give one batch a name of its own, and there is nothing to configure here. That is a platform limit rather than a choice.

What it means in practice: if Vercel retries a delivery that in fact reached us, the lines in it are counted a second time. Crawl figures are daily counts read as a trend, so a rare repeat shifts a day slightly rather than changing what the page tells you. If exact counts matter more to you than a configuration-only setup, relay the drain through a small endpoint of your own that names each batch and posts it on, following the custom REST contract.

Do not put a fixed Idempotency-Key in the Custom Headers box. A key is a name for one batch. A static one names every batch the same, so the first delivery would be stored and every delivery after it answered as a duplicate and stored nowhere at all. The connection would look healthy and the data would stop.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when you save the drain. It is finished when a real event lands.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

"Your endpoint sent a 503 status instead of 2xx". Vercel says this when it tests the endpoint, and the drain is not saved. The 503 is ours: it means the ingest deployment you are pointing at has no signing secret set, so it can verify no token and reads nothing. There is nothing to change in your drain. If the endpoint is a Heading one, tell us and try again once we say it is set. If you are running the ingest deployment yourself, set CRAWL_INGEST_SIGNING_SECRET on it and then redeploy: an environment variable added in Vercel reaches a deployment when that deployment is built, so the endpoint keeps answering 503 until a new one goes out. Press Test again after the redeploy finishes rather than recreating the drain.

Nothing is arriving. Vercel batches lines into one request, so give it a few minutes on a quiet site. Then check the drain is not paused on the Drains page, check the project selection includes the project serving this domain, and press Test to make Vercel send immediately.

Vercel says the drain is erroring. Vercel emails you and flags the drain when more than 80% of deliveries fail in an hour, or when failures pass 50. Look at the delivery log in Heading: if the attempts are arriving and being refused, the reason is there. If nothing is arriving at all, the endpoint URL is wrong.

Events arrive and all of them are rejected. A 401 is the token. Tokens are per property: the token for one property is rejected by another property's endpoint, and the property id in the URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately, so a rotation you forgot about looks exactly like a typo. Check the Custom Headers box reads Authorization: Bearer and then the token, with one space and no trailing whitespace.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. On Vercel this is almost always preview deployments (turn the preview environment off) or one project serving several domains. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. If the other domains are sites you track, connect each one as its own property, with its own endpoint and token, and split them into one drain per project.

Lines received is high, lines kept is zero, all day. Check that static is among the selected sources. A site whose pages are all statically served sends nothing else that a crawler would touch.