Custom REST API

Every recipe on this site posts to the same endpoint. This page is that endpoint, written out in full, for when you are sending the events yourself: from nginx, from your application, from a log shipper, or from a platform we have not written a recipe for.

The endpoint

POST https://INGEST_HOST/api/ingest/crawl/{propertyId}

One endpoint per property. The property id is in the path, and it has to be the property the token belongs to.

Headers

If your sender cannot set an Authorization header, put the token in a token query parameter instead:

POST https://INGEST_HOST/api/ingest/crawl/{propertyId}?token=YOUR_TOKEN

The header wins when both are present. Use the header wherever you can: a credential in a URL lives in your sender's configuration and can end up in an intermediary's logs. The parameter exists because three platforms cannot use the header at all — Akamai DataStream 2 refuses a custom Authorization header, Google Cloud Pub/Sub fills that header with its own signed token, and Cloudflare Logpush has no header field.

The token comes from Tracking → Outcomes → Crawl activity → Set up. It is derived from the property rather than stored, so there is no per-property secret in our database to leak, and rotation is a version bump that invalidates the old token immediately. Rotation is a revocation, so there is no period in which both tokens work: send the new token from every sender as soon as you rotate, or the endpoint answers 401 until you do. It is a server-side credential: never put it in a browser, a mobile app, or anything a visitor can read.

There is no request signature. A log drain cannot compute an HMAC over its own body, so this endpoint authenticates on a static bearer token, which is the one thing every log source can attach. That is a deliberate trade, and it is why the token has to be treated carefully.

Body

Two shapes, both accepted.

Newline-delimited JSON, one event per line. This is what the log drains send and what you should send from a shipper:

{"timestamp":"2026-08-15T09:14:02.412Z","host":"acmedental.com","path":"/treatments/implants","method":"GET","statusCode":200,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot","referer":"","ip":"20.171.207.1"}
{"timestamp":"2026-08-15T09:14:03.008Z","host":"acmedental.com","path":"/pricing","method":"GET","statusCode":200,"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36","referer":"https://chatgpt.com/","ip":"81.2.69.142"}

A JSON array of the same objects, for senders that build one document:

[
  {
    "timestamp": "2026-08-15T09:14:02.412Z",
    "host": "acmedental.com",
    "path": "/treatments/implants",
    "method": "GET",
    "statusCode": 200,
    "userAgent": "Mozilla/5.0 ... GPTBot/1.2; +https://openai.com/gptbot",
    "referer": "",
    "ip": "20.171.207.1"
  }
]

Send every request your site serves, including static assets and including human traffic. Heading filters server-side and keeps roughly one line in a hundred. The lines it keeps are not only crawler lines: your human traffic is what builds the page inventory and produces every AI-referred entry.

Fields

Any other key is ignored. Send the eight above and nothing else: a field Heading does not read is bandwidth you are paying for twice.

Worked examples

ENDPOINT="https://INGEST_HOST/api/ingest/crawl/YOUR_PROPERTY_ID"

curl -X POST "$ENDPOINT" \
  -H "authorization: Bearer $HEADING_CRAWL_TOKEN" \
  -H "idempotency-key: $(uuidgen)" \
  -H 'content-type: application/x-ndjson' \
  --data-binary $'{"timestamp":"2026-08-15T09:14:02.412Z","host":"acmedental.com","path":"/pricing","method":"GET","statusCode":200,"userAgent":"GPTBot/1.2","referer":"","ip":"20.171.207.1"}\n'

# {"kept":1,"received":1}

import { randomUUID } from "node:crypto";

const ENDPOINT = process.env.HEADING_INGEST_URL;
const TOKEN = process.env.HEADING_CRAWL_TOKEN;

// Collect events in memory and post them on a timer. One POST per request to
// your own site would work, and would be wasteful.
const buffer = [];

// A starting point, not the endpoint's limit. Lower it if you see a 413.
const BATCH_SIZE = 500;

// A batch that was sent and got no definite answer. It is held whole, under
// the name it was sent with, so the next flush sends the same batch again
// rather than a differently composed one.
let unconfirmed = null;

export function recordRequest(event) {
  buffer.push(event);
}

// A batch is named once, when it is formed, and keeps that name for every
// attempt at it. That is what lets Heading recognise a retry of a batch that
// had in fact landed, instead of counting every event in it a second time.
function nextBatch(batchSize) {
  if (unconfirmed) {
    return unconfirmed;
  }
  if (buffer.length === 0) {
    return null;
  }
  return { events: buffer.splice(0, batchSize), key: randomUUID() };
}

export async function flush(batchSize = BATCH_SIZE) {
  const batch = nextBatch(batchSize);
  if (!batch) {
    return;
  }
  const body = `${batch.events
    .map((event) => JSON.stringify(event))
    .join("\n")}\n`;

  let response;
  try {
    response = await fetch(ENDPOINT, {
      body,
      headers: {
        authorization: `Bearer ${TOKEN}`,
        "content-type": "application/x-ndjson",
        "idempotency-key": batch.key,
      },
      method: "POST",
    });
  } catch {
    // No answer, so it may still have landed. Same batch, same key, next time.
    unconfirmed = batch;
    return;
  }

  if (response.status === 413 && batch.events.length > 1) {
    // Too big for one request. Put it back and send it in smaller pieces.
    // Each half is a new batch and gets a new key of its own.
    unconfirmed = null;
    buffer.unshift(...batch.events);
    await flush(Math.ceil(batch.events.length / 2));
    return;
  }

  if (response.status >= 500) {
    // Transient, and possibly already stored. The same batch under the same
    // key goes again on the next flush.
    unconfirmed = batch;
    return;
  }
  unconfirmed = null;

  if (!response.ok) {
    const result = await response.json();
    throw new Error(`${result.error}. ${result.hint}`);
  }

  // { duplicate: true, ... } means this batch was already stored. That is a
  // success, and it must not be sent again.
  return await response.json();
}

setInterval(() => {
  flush().catch(() => {
    // Never let reporting break the thing being reported on.
  });
}, 5000);

import json
import os
import urllib.request
import uuid

ENDPOINT = os.environ["HEADING_INGEST_URL"]
TOKEN = os.environ["HEADING_CRAWL_TOKEN"]


def send_batch(events, key=None):
    """Post a list of event dicts as newline-delimited JSON.

    Pass the same key back in when you retry a batch, so the retry is
    recognised instead of counted twice.
    """
    body = ("\n".join(json.dumps(event, separators=(",", ":")) for event in events) + "\n").encode()

    request = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={
            "authorization": f"Bearer {TOKEN}",
            "content-type": "application/x-ndjson",
            "idempotency-key": key or str(uuid.uuid4()),
        },
    )
    with urllib.request.urlopen(request) as response:
        return json.load(response)


send_batch([
    {
        "timestamp": "2026-08-15T09:14:02.412Z",
        "host": "acmedental.com",
        "path": "/pricing",
        "method": "GET",
        "statusCode": 200,
        "userAgent": "GPTBot/1.2",
        "referer": "",
        "ip": "20.171.207.1",
    }
])
# {'kept': 1, 'received': 1}

From a web server

If you are shipping nginx logs, write them as JSON with the field names above and any shipper that can POST a batch with a bearer header will do:

log_format heading escape=json
  '{"timestamp":"$time_iso8601","host":"$host","path":"$uri",'
  '"method":"$request_method","statusCode":$status,'
  '"userAgent":"$http_user_agent","referer":"$http_referer",'
  '"ip":"$remote_addr"}';

access_log /var/log/nginx/heading.log heading;

$uri is the path without the query string, and escape=json is what keeps a user agent containing a quote from breaking the line.

Batching

Each POST is a batch. Post on a timer rather than per request: a request to your site should never wait on a request to us.

Order does not matter, and neither does grouping. Events carry their own timestamps, so a batch spanning several minutes is read correctly.

Send each event once. Heading does not deduplicate individual events, so a batch you replay without naming it counts twice. Name your batches and a retry of one is recognised: see Retries and the idempotency key.

There is a cap on the size of one body. Past it the endpoint answers 413 and stores nothing from that request, so split the batch and send both halves rather than dropping it. Splitting by line count is enough: no single log line comes close to the cap on its own.

Responses

An accepted request answers with how many lines arrived and how many of them Heading kept. Both figures count log lines, not bytes and not records: a line that could not be read still arrived, and it is counted in received.

{ "kept": 3, "received": 412 }

kept well below received is the normal case. Most requests to a site are not crawler requests, and Heading is not going to store a line about a stylesheet fetched by a browser.

Retry a 500 or a 503. Do not retry a 400, 401, 402, 403, 404 or 413: the same request will be refused the same way, and a retry loop against a 401 fills your delivery log with identical rejections.

A 500 deserves a word, because the honest answer is less tidy than it looks. A write here can fail before it commits, in which case nothing landed and a retry costs you nothing. It can also commit and then lose its answer on the way back to you, a timeout or a reset, in which case the batch is stored and a plain retry adds every one of its counters a second time. From inside the endpoint those two look identical, so it does not claim to know which happened, and it will not tell you nothing was stored. Retrying under the same idempotency key is what makes the difference safe.

Every rejection carries a stable reason code, a human error, and a hint saying what to change. The delivery log in Heading renders the same three, so someone who cannot see your server's response can still see why an event was refused.

Retries and the idempotency key

Send an Idempotency-Key header naming the batch, and a repeat of that batch is recognised rather than stored again:

Idempotency-Key: 7f3c2a18-6b21-4d0e-9d55-1c4f2f0a9e3b

A repeat is answered with the original delivery's numbers and a duplicate flag, and nothing is written:

{ "duplicate": true, "kept": 3, "received": 412 }

Treat that as success. The batch is stored, it is stored once, and sending it again achieves nothing. The kept and received in a duplicate answer are the figures from the attempt that landed, not from the one you just sent.

Naming a batch

The key is yours to choose, and only two properties matter.

It must not change when you retry. Mint it once, when the batch is formed, keep it with the batch, and send the same value on every attempt at that batch. A key drawn fresh per request, or built from a clock read at send time, makes every attempt a new batch and the header does nothing at all.

It must not name two different batches the same. A random UUID per batch is the simple answer, and it is what the recipes here use: randomUUID() from node:crypto in Node, crypto.randomUUID() in a Cloudflare Worker, uuidgen in a shell, uuid.uuid4() in Python. Do not use a fixed string, and do not derive the key from the batch's contents: two batches that happen to be identical would take the same name, and the second one would be answered as a duplicate and stored nowhere. If you split a batch, the halves are new batches and each one needs its own key.

Keep it short. A key over 200 characters is refused: the endpoint answers 400 with the reason idempotency_key_too_long, nothing from that request is stored, and the attempt is recorded in your delivery log with the length it saw. Shorten the key and send the batch again under the new one. A UUID is 36 characters.

It is refused rather than shortened, because two batches cut to the same 200 characters would become one batch, and the second would be answered as a duplicate and stored nowhere. And it is refused rather than ignored, because an ignored key leaves you sending batches you believe are protected and are not. You would find that out the next time a response was lost, in counters that had been added twice, with nothing to trace it back to.

What it does not cover

This is not exactly-once delivery, and it would be dishonest to sell it as that. Three gaps are worth knowing about.

A crash between the write and the log. The key is recognised because the accepted delivery was written to the delivery log. If the process dies after the batch commits but before that row is written, the key is unknown to us, and your retry is stored as a new batch and counted twice.

Two attempts genuinely in flight at the same time. The lookup happens before the body is read. Two identical retries overlapping closely enough can both find no stored key and both go on to store the batch. Retry in sequence with a backoff rather than firing a second attempt while the first is still open.

The window is the delivery log, not forever. Heading keeps the last 50 attempts per property and prunes older ones as new ones arrive. Once the original attempt's row has aged out, its key is unknown again and a repeat is stored as new. On a busy connection that can be a matter of minutes. The mechanism is for retrying a batch now, not for replaying one tomorrow.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when your first POST returns 202. It is finished when a real event lands and matches.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

Nothing is arriving. Your sender is not reaching us. Run the cURL example above from the same machine: it either returns 202, or it names the problem, or it fails to connect and the problem is between your network and the endpoint.

Events arrive and all of them are rejected. Read the reason on the response. A 401 is the token: tokens are per property, so the token for one property is rejected by another property's endpoint, and a token rotated in Heading stops working immediately. A 403 is the connection: someone disconnected it in Heading, and nothing sent to a disconnected connection is stored. A 400 is the body or the key, and the reason says which: unreadable_body means the content type does not match what you are sending, and idempotency_key_too_long means your sender is naming batches with something longer than 200 characters.

connection_disconnected. Ingestion is switched off for this property. Every attempt is refused and every one is recorded in the delivery log, so a sender nobody removed stays visible rather than silent. Remove the sender, or reconnect the property under Tracking → Outcomes → Crawl activity → Set up. Reconnecting starts the verification over: the connection waits for its first event again.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. This is the normal case when one sender covers several sites. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Send each site's events to its own property endpoint with its own token, or drop the non-matching lines before you send.

Everything is accepted and nothing is kept. Check userAgent is populated on the events you are sending, and that it is the raw header rather than something your framework has already parsed or truncated. Then check host: an event whose host matches would otherwise have been rejected, so if you are seeing acceptances with nothing kept, the user agent is the usual culprit.

Every crawler event shows as unverifiable. You are not sending ip, or you are sending the IP of your own proxy rather than the client's. Behind a load balancer, take the client address from the header your balancer sets rather than the socket address.