Netlify edge function

Netlify log drains are an Enterprise feature. An edge function is not, so this is the Netlify recipe for everyone else. One file, deployed with your site, reporting each request to Heading after the response has been served.

Official reference: Edge functions overview and the edge function API.

Before you start

Get the endpoint and the bearer token from Heading. Open Tracking → Outcomes, find the Crawl activity card and select Set up. Put the token in a Netlify environment variable, not in the source file.

Deploy the function

const PROPERTY_ID = "YOUR_PROPERTY_ID";
const INGEST_URL = `https://INGEST_HOST/api/ingest/crawl/${PROPERTY_ID}`;

export default async (request, context) => {
  // Serve the visitor first. Nothing below is on the response path.
  const response = await context.next();

  const url = new URL(request.url);
  const event = {
    timestamp: new Date().toISOString(),
    host: url.hostname,
    path: url.pathname,
    method: request.method,
    statusCode: response.status,
    userAgent: request.headers.get("user-agent") ?? "",
    referer: request.headers.get("referer") ?? "",
    ip: context.ip,
  };

  context.waitUntil(
    fetch(INGEST_URL, {
      method: "POST",
      headers: {
        "content-type": "application/x-ndjson",
        authorization: `Bearer ${Netlify.env.get("HEADING_CRAWL_TOKEN")}`,
      },
      body: `${JSON.stringify(event)}\n`,
    }).catch(() => {
      // A reporting failure must never affect the visitor's response.
    })
  );

  return response;
};

export const config = { path: "/*" };

Match every path

path: "/*" is deliberate. Heading filters server-side and drops roughly a hundred lines for every one it keeps, and the lines it keeps are not only the crawler ones: your human traffic builds the page inventory and produces every AI-referred entry.

Do not add a user agent check to the function either. A function that reports only what its own list already believes to be a crawler can only confirm that list, and the list goes stale the week a new crawler appears.

Edge function invocations count against your Netlify plan. Check your plan's included volume before putting this on a high-traffic site.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

The function above sends url.pathname, so the query string never leaves your edge in the first place. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when the function deploys. It is finished when a real event lands.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

Nothing is arriving. Confirm the deploy that includes the file is the published one, and check the function is listed under the deploy's edge functions. netlify dev runs it locally, and a console.log in the handler tells you whether it is being invoked at all.

Events arrive and all of them are rejected. A 401 is the token. Check HEADING_CRAWL_TOKEN is set for the production context, not only for previews, and that it is the token for this property. Tokens are per property: the token for one property is rejected by another property's endpoint, and the PROPERTY_ID in the URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. Edge functions also run on deploy previews and branch deploys, which answer on *.netlify.app. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Either scope the environment variable to production only, so previews send an unauthenticated request that is refused at the door, or return early in the function when url.hostname is not your production domain.

Requests got slower. The reporting call is inside context.waitUntil, so it is not on the response path. If you see added latency, check you have not awaited the fetch to Heading before returning the response.