Cloudflare Worker

This is the Cloudflare recipe for everyone. Workers run on every Cloudflare plan, Free included, so you do not need Enterprise and you do not need Logpush. You deploy one Worker on a route, it passes each request through to your origin untouched, and on the side it reports the requests Heading keeps, in batches.

The Worker is the only recipe that sits in your visitors' request path, so it is the only one where the cost of reporting is yours as well as ours. That shapes the code below: it matches each request against the crawler and AI referer lists first, and it posts what survives in batches rather than one request at a time.

Official reference: Get started with Workers, Routes and Secrets.

If you are on Cloudflare Enterprise and already run Logpush jobs, Cloudflare Logpush is configuration only and costs you no Worker requests. Everyone else wants this page.

Before you start

You need the endpoint and the bearer token from Heading. Open Tracking → Outcomes, find the Crawl activity card and select Set up. The token is derived from the property, so there is nothing per-property stored on our side. Treat it as a server-side credential and put it in a Worker secret, not in the source file.

You also need the zone in Cloudflare with proxied DNS, so requests actually pass through Cloudflare.

Deploy the Worker

npm create cloudflare@latest -- heading-crawl

const PROPERTY_ID = "YOUR_PROPERTY_ID";
const INGEST_URL = `https://INGEST_HOST/api/ingest/crawl/${PROPERTY_ID}`;

// Only a request Heading would keep ever leaves your edge: a user agent in
// Heading's crawler catalogue, or a referer from an AI platform. On a normal
// site that is about one request in a hundred. Both lists were written into
// this file when you copied it. Copy it again from the setup dialog to pick up
// crawlers Heading has added since.
const CRAWLER_USER_AGENT = /\bgptbot\b|\boai-searchbot\b|\bchatgpt-user\b|\bclaudebot\b|\bclaude-searchbot\b|\bclaude-user\b|\banthropic-ai\b|\bperplexitybot\b|\bperplexity-user\b|\bgooglebot\b|\bbingbot\b|\bccbot\b|\bbytespider\b|\bamazonbot\b|\bmeta-externalagent\b|\bduckassistbot\b|\bmistralai-user\b|\bapplebot\b(?!-extended)/i;
const AI_REFERER_HOSTS = [
  "chatgpt.com",
  "chat.openai.com",
  "claude.ai",
  "gemini.google.com",
  "bard.google.com",
  "perplexity.ai",
  "copilot.microsoft.com",
];

// What survives the filter is batched, never posted one request at a time. A
// batch goes when it reaches 20 events or when the oldest event in it is
// 10 seconds old, whichever comes first.
const FLUSH_AT_EVENTS = 20;
const FLUSH_AFTER_MS = 10000;
const RETRY_AFTER_MS = 2000;

let queued = [];
let oldestQueuedAt = 0;

function refererHost(referer) {
  try {
    const host = new URL(referer).hostname.toLowerCase();
    return host.startsWith("www.") ? host.slice(4) : host;
  } catch {
    // A referer that is not a URL is not an AI referer.
    return "";
  }
}

function isAiReferer(referer) {
  const host = refererHost(referer);
  if (host === "") {
    return false;
  }
  return AI_REFERER_HOSTS.some(
    (known) => host === known || host.endsWith("." + known)
  );
}

function isReportable(userAgent, referer) {
  return CRAWLER_USER_AGENT.test(userAgent) || isAiReferer(referer);
}

// Hands back the whole buffer when either threshold is reached, and empties it
// in the same step, so two invocations can never send the same event twice.
//
// The batch is named here, once, and keeps that name for every attempt at it.
// That is what lets Heading recognise a retry of a batch that had in fact
// landed, instead of counting every event in it a second time.
function takeBatch(now) {
  if (queued.length === 0) {
    return null;
  }
  const full = queued.length >= FLUSH_AT_EVENTS;
  const stale = now - oldestQueuedAt >= FLUSH_AFTER_MS;
  if (!(full || stale)) {
    return null;
  }
  const batch = { events: queued, key: crypto.randomUUID() };
  queued = [];
  oldestQueuedAt = 0;
  return batch;
}

// NDJSON: one JSON object per line, which is what the endpoint parses.
//
// One attempt, then one more if the first got no answer or a 5xx, both under
// the same Idempotency-Key. A repeat that Heading has already stored comes
// back 202 with "duplicate": true, which is a success: the batch is in, and
// sending it again would be the double count this key exists to prevent.
// Anything below 500 is final, including that duplicate and including a 401 a
// retry would only repeat.
async function send(batch, key, token) {
  for (let attempt = 0; attempt < 2; attempt++) {
    if (attempt > 0) {
      await new Promise((resolve) => setTimeout(resolve, RETRY_AFTER_MS));
    }
    let response = null;
    try {
      response = await fetch(INGEST_URL, {
        method: "POST",
        headers: {
          "content-type": "application/x-ndjson",
          "idempotency-key": key,
          authorization: `Bearer ${token}`,
        },
        body: `${batch.map((event) => JSON.stringify(event)).join("\n")}\n`,
      });
    } catch {
      // No answer at all. It may still have landed, which is exactly why the
      // retry carries the same key.
    }
    if (response && response.status < 500) {
      return;
    }
  }
  // Two attempts, no acceptance. The batch is dropped, and a reporting failure
  // never reaches the visitor.
}

export default {
  async fetch(request, env, ctx) {
    // Answer the visitor first. Nothing below is on the response path.
    const response = await fetch(request);

    const userAgent = request.headers.get("user-agent") ?? "";
    const referer = request.headers.get("referer") ?? "";
    const now = Date.now();

    if (isReportable(userAgent, referer)) {
      const url = new URL(request.url);
      if (queued.length === 0) {
        oldestQueuedAt = now;
      }
      queued.push({
        timestamp: new Date(now).toISOString(),
        host: url.hostname,
        path: url.pathname,
        method: request.method,
        statusCode: response.status,
        userAgent,
        referer,
        ip: request.headers.get("cf-connecting-ip") ?? "",
      });
    }

    // The age is checked on every request, not only on the kept ones, so a
    // queued event leaves within 10 seconds on a zone with traffic.
    const batch = takeBatch(now);
    if (batch) {
      ctx.waitUntil(send(batch.events, batch.key, env.HEADING_CRAWL_TOKEN));
    }

    return response;
  },
};

{
  "name": "heading-crawl",
  "main": "src/index.js",
  "compatibility_date": "2026-08-01",
  "routes": [
    { "pattern": "acmedental.com/*", "zone_name": "acmedental.com" },
    { "pattern": "www.acmedental.com/*", "zone_name": "acmedental.com" }
  ]
}

npx wrangler secret put HEADING_CRAWL_TOKEN

npx wrangler deploy

What the Worker sends, and what it does not

A request leaves your edge only if one of two things is true.

• Its user agent matches a crawler in Heading's catalogue, GPTBot and ClaudeBot and the 16 others in the list at the top of the file.
• Its referer is a host Heading knows as an AI platform, so the arrival is a person who came from an AI answer.

Everything else is answered and forgotten. No buffer entry, no subrequest, no row.

We chose that over sending you the whole firehose, and here is what we gave up. The page inventory on this recipe covers only the pages a crawler or an AI-referred visitor touched. Your ordinary human traffic normally tells Heading which pages exist and get used, which is what makes "this page has never been retrieved" a finding rather than a guess. On the Worker recipe you do not get that half. If you want it, use a Vercel log drain or Cloudflare Logpush instead: the platform batches those for you, they run nowhere near your response path, and they cost you no Worker subrequests.

The reason is arithmetic. One POST per visitor request on a zone serving 100 requests a second is 100 writes a second into the database Heading runs everything else on, and it fills the 50-row delivery log below in half a second, which leaves you nothing to diagnose with. Filtering first removes about 99 requests in 100 before any of that starts.

The lists go stale, and re-copying fixes it

The crawler patterns and the AI referer hosts are written into your file at the moment you copy it out of the setup dialog. They do not update themselves. When Heading adds a crawler, your deployed Worker keeps ignoring it until you copy the snippet again and redeploy. Copy it from the dialog rather than from this page, and treat it as a five-minute job worth doing when we tell you the catalogue has changed.

Batching, and what a lost batch costs you

Kept events are held in the Worker and posted together. A batch goes when it reaches 20 events, or when the oldest event in it is 10 seconds old, whichever comes first. The age is checked on every request the Worker sees, including the ones it does not keep, so on a zone with traffic nothing waits longer than about 10 seconds. The buffer cannot grow past 20 events, because every request that queues one also tests the thresholds.

Cloudflare can evict a Worker isolate at any time, and it takes whatever is buffered with it. A batch that has not been posted yet can be lost. In practice that is at most 20 crawler hits, or 10 seconds of them, and only on eviction. Two other cases lose the same way: an ingest outage that outlasts both attempts drops the batch in flight, and a zone that goes quiet holds its last few events until the next request arrives to push them out.

Retries, and the name on a batch

A batch is tried once, and once more 2 seconds later if the first attempt got no answer or a 5xx. Anything else is final: a 202 is done, and a 401 will be refused the same way however many times you send it.

The retry is only safe because of the name. When Heading answers a 5xx it genuinely cannot tell you whether your batch was stored, because a write can commit and then lose its answer on the way back. A plain retry of a batch that had landed would add every event in it twice. So takeBatch mints a crypto.randomUUID() when it forms the batch, both attempts send it as Idempotency-Key, and a repeat Heading has already stored is answered with the original delivery's numbers and a duplicate flag, storing nothing.

You do not have to do anything about this. It is worth knowing about because of what it does not cover: the key is recognised from the delivery log, which holds the last 50 attempts per property, and two attempts genuinely overlapping in time can both miss it. This is a retry that is safe to make, not a delivery guarantee. The custom REST page sets out the whole contract if you are writing your own sender.

We think that is the right trade for a monitoring signal you read in daily counts, and we would rather write it down than let you find it. If you cannot accept any loss, the fix is Cloudflare Queues or a Durable Object holding the buffer instead of the isolate. Both are paid features and both turn a paste-this-in recipe into something you maintain, which is why neither is the default here.

Worker subrequests still count against your plan's allowance, though only for the requests that pass the filter now. Check the Workers limits for your plan before you put this on a high-traffic zone.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

The Worker above sends url.pathname, so the query string never leaves your edge in the first place. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when the Worker deploys. It is finished when a real event lands.

Because the Worker only reports crawlers and AI referrals, browsing your own site proves nothing. Send yourself a crawler request instead, twice, about 15 seconds apart. The first queues an event and the second pushes the batch out, because a Worker has no timer of its own and flushes on the next request it sees.

curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://acmedental.com/
sleep 15
curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://acmedental.com/

That lands one delivery of two lines, both kept. It is a real crawl event carrying your own IP, so it will show as unverified on the page: it did not come from OpenAI's published range. That is the check working, not a fault.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

On the other recipes an accepted delivery that stores nothing is normal, because they send the whole firehose and Heading discards most of it. On this one the filter has already run at your edge, so lines kept should sit close to lines received. A delivery here where kept is zero is worth reading: the usual cause is host_not_matched, and the note on the row will say so.

Troubleshooting

Nothing is arriving. First rule out the filter: a browser visit is not a reportable request, so use the two curl commands above rather than loading the site. Then check the Worker is actually running: in the Cloudflare dashboard, open the Worker, then Settings → Domains & Routes, and confirm the route is there and the pattern ends in /*. Confirm the DNS record for the hostname is proxied (orange cloud) rather than DNS only. npx wrangler tail shows live invocations and any error thrown inside the Worker.

One event arrived and then nothing. Events wait for a batch, and a batch waits for the next request. On a quiet site the last few events sit in the Worker until someone visits, and if Cloudflare evicts the isolate first they are gone. Send another request and they land. If your site is quiet enough for this to be a daily annoyance, a log drain recipe suits you better than a Worker.

A crawler you know visited is missing. The crawler list is a copy taken when you copied the snippet. If Heading has added that crawler since, your Worker is still ignoring it. Copy the snippet again from the setup dialog and redeploy.

Events arrive and all of them are rejected. A 401 is the token. Tokens are per property: the token for one property is rejected by another property's endpoint, and the PROPERTY_ID in the URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately, so a rotation you forgot about looks exactly like a typo. Re-run npx wrangler secret put HEADING_CRAWL_TOKEN to replace it.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. On Cloudflare this happens when the Worker route covers hostnames beyond the property, such as a staging subdomain on the same zone. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Narrow the routes, or connect the other hostname as its own property in Heading with its own endpoint and token.

Your site broke. The Worker sits in the request path, so treat it as production code. fetch(request) passes the request through unchanged, and the reporting call is wrapped in catch, so a failure at Heading returns nothing to the visitor. If you need to switch it off in a hurry, delete the route in Settings → Domains & Routes: traffic goes straight to your origin again.