AWS CloudFront

CloudFront real-time logs go to one place: a Kinesis data stream. There is no HTTP destination and no custom header, so this recipe is a log configuration plus a Lambda that reads the stream and posts to Heading. The Lambda is about thirty lines and it is all on this page.

Official reference: Use real-time access logs.

Before you start

Get the endpoint and the bearer token from Heading. Open Tracking → Outcomes, find the Crawl activity card and select Set up. Store the token in AWS Secrets Manager or as an encrypted Lambda environment variable, not in the function source.

CloudFront charges for real-time logs, and Kinesis Data Streams charges for the stream. Both are your costs, and both scale with your traffic.

Set it up

// Fields arrive in the order AWS documents, NOT the order you ticked them.
// If you add a field to the log configuration later, add it here in the
// documented position or every value after it shifts one place left.
const FIELDS = [
  "timestamp",
  "c-ip",
  "sc-status",
  "cs-method",
  "cs-host",
  "cs-uri-stem",
  "cs-user-agent",
  "cs-referer",
];

const decode = (value) => {
  if (!value || value === "-") {
    return "";
  }
  try {
    return decodeURIComponent(value);
  } catch {
    return value;
  }
};

export const handler = async (event) => {
  const lines = [];

  for (const record of event.Records) {
    const payload = Buffer.from(record.kinesis.data, "base64").toString("utf8");

    for (const line of payload.split("\n")) {
      if (!line.trim()) {
        continue;
      }
      const values = line.split("\t");
      const row = Object.fromEntries(FIELDS.map((f, i) => [f, values[i]]));

      lines.push(
        JSON.stringify({
          timestamp: new Date(Number(row.timestamp) * 1000).toISOString(),
          host: row["cs-host"],
          // Real-time logs put the query string on cs-uri-stem. Drop it here
          // so it never leaves your account.
          path: row["cs-uri-stem"].split("?")[0],
          method: row["cs-method"],
          statusCode: Number(row["sc-status"]),
          userAgent: decode(row["cs-user-agent"]),
          referer: decode(row["cs-referer"]),
          ip: row["c-ip"],
        })
      );
    }
  }

  if (lines.length === 0) {
    return;
  }

  // Names this batch, so the Lambda retry below is recognised as a repeat
  // rather than counted twice. Lambda retries the same records, so the
  // first record's sequence number is the same on every attempt, and it is
  // different for every other batch on the stream.
  const idempotencyKey = event.Records[0].kinesis.sequenceNumber;

  const response = await fetch(process.env.HEADING_INGEST_URL, {
    method: "POST",
    headers: {
      "content-type": "application/x-ndjson",
      "idempotency-key": idempotencyKey,
      authorization: `Bearer ${process.env.HEADING_CRAWL_TOKEN}`,
    },
    body: `${lines.join("\n")}\n`,
  });

  if (!response.ok) {
    // Throwing makes Lambda retry the batch from the stream.
    throw new Error(`Heading ingest returned ${response.status}`);
  }
};

A cheaper alternative, with a delay

If real-time is more than you need, CloudFront standard logs written to S3 and read by an S3-triggered Lambda gets you the same events for less money. The same POST body works. What you give up is timeliness: standard logs arrive in batched files, so "did the crawler get a 403 an hour ago" becomes "did it get a 403 at some point today".

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

On this recipe the Lambda strips the query string before the request leaves AWS, and the eight fields above are the only ones the log configuration collects. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when the Lambda deploys. It is finished when a real event lands.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

Nothing is arriving, and the Lambda is not running. Work backwards. Is the real-time log configuration attached to a cache behaviour on the distribution? Is the distribution taking traffic? Is the Kinesis trigger enabled on the function? CloudWatch metrics on the stream tell you whether CloudFront is writing to it at all.

The Lambda runs and every record looks wrong. Fields shifted. AWS delivers the fields in its own documented order, not the order you selected them in, and adding a field later moves everything after it. Line up the FIELDS array with the field list in your log configuration.

Events arrive and all of them are rejected. A 401 is the token. Tokens are per property: the token for one property is rejected by another property's endpoint, and the property id in HEADING_INGEST_URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. On CloudFront this is usually the *.cloudfront.net domain being hit directly, or a distribution with several alternate domain names. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Connect the other domain as its own property in Heading with its own endpoint and token, or skip the line in the Lambda when cs-host is not the property's domain.

Records are being dropped between CloudFront and Kinesis. AWS delivers real-time logs on a best-effort basis and will throttle if the stream is undersized. Add shards, or move the stream to on-demand capacity.