Google Cloud logging

If your requests are logged by Cloud Run, App Engine or an external Application Load Balancer, they are already in Cloud Logging. This recipe routes them to a Pub/Sub topic and relays them to Heading with a function of about thirty lines.

Official reference: Configure and manage sinks, LogEntry and Application Load Balancer logging.

Why there is a relay function

A Pub/Sub push subscription cannot carry Heading's bearer token. Pub/Sub puts its own signed JWT in the Authorization header, and there is no second header to add. It also wraps every message in its own envelope, with the log entry base64-encoded inside it.

So something has to sit between Pub/Sub and Heading, unwrap the message and add the token. That is all the function below does.

Set it up

gcloud compute backend-services update BACKEND_SERVICE \
  --global \
  --enable-logging \
  --logging-sample-rate=1.0

gcloud pubsub topics create heading-crawl

# External Application Load Balancer
gcloud logging sinks create heading-crawl \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/heading-crawl \
  --log-filter='resource.type="http_load_balancer" AND httpRequest.requestUrl!=""'

# Cloud Run
gcloud logging sinks create heading-crawl \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/heading-crawl \
  --log-filter='resource.type="cloud_run_revision" AND logName="projects/PROJECT_ID/logs/run.googleapis.com%2Frequests"'

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=serviceAccount:WRITER_IDENTITY \
  --role=roles/pubsub.publisher

import { cloudEvent } from "@google-cloud/functions-framework";

cloudEvent("relay", async (event) => {
  const message = event.data?.message;
  if (!message?.data) {
    return;
  }

  const entry = JSON.parse(Buffer.from(message.data, "base64").toString("utf8"));
  const request = entry.httpRequest;
  if (!request?.requestUrl) {
    return;
  }

  // requestUrl carries the query string. Building a URL and taking pathname
  // drops it here, so it never leaves your project.
  const url = new URL(request.requestUrl);

  const line = JSON.stringify({
    timestamp: entry.timestamp,
    host: url.hostname,
    path: url.pathname,
    method: request.requestMethod,
    statusCode: request.status,
    userAgent: request.userAgent ?? "",
    referer: request.referer ?? "",
    ip: request.remoteIp ?? "",
  });

  const response = await fetch(process.env.HEADING_INGEST_URL, {
    method: "POST",
    headers: {
      "content-type": "application/x-ndjson",
      // Pub/Sub keeps the message id across redeliveries, so a redelivery
      // is recognised as a repeat rather than counted twice.
      "idempotency-key": message.messageId,
      authorization: `Bearer ${process.env.HEADING_CRAWL_TOKEN}`,
    },
    body: `${line}\n`,
  });

  if (!response.ok) {
    // Throwing lets Pub/Sub redeliver the message.
    throw new Error(`Heading ingest returned ${response.status}`);
  }
});

{
  "name": "heading-crawl-relay",
  "type": "module",
  "main": "index.js",
  "dependencies": {
    "@google-cloud/functions-framework": "^3.4.0"
  }
}

gcloud functions deploy heading-crawl-relay \
  --gen2 \
  --runtime=nodejs22 \
  --region=REGION \
  --source=. \
  --entry-point=relay \
  --trigger-topic=heading-crawl \
  --set-env-vars=HEADING_INGEST_URL=https://INGEST_HOST/api/ingest/crawl/PROPERTY_ID \
  --set-secrets=HEADING_CRAWL_TOKEN=heading-crawl-token:latest

One POST per request, and what to do about it

This relay posts one event per logged request, because a Pub/Sub trigger hands the function one message at a time. On a quiet site that is fine and costs almost nothing. On a busy one it is a lot of invocations and a lot of small requests.

If that matters to you, replace the trigger with a pull subscriber: read a batch of messages, build one NDJSON body with a line per message, post it once, and acknowledge the batch. The endpoint takes many lines per request, which is what every other recipe on this site does. The tradeoff is that you are now running a process rather than a function, and you own its uptime.

What Heading stores

The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.

On this recipe the relay drops the query string before anything leaves your project, and sends eight fields and nothing else. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.

Check it worked

Setup is not finished when the function deploys. It is finished when a real event lands.

Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.

An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.

Troubleshooting

Nothing is arriving, and the function is not running. Work backwards along the chain. Does Logs Explorer return entries for your filter? Does the topic show published messages? Did the sink's writer identity get roles/pubsub.publisher? A sink without that permission fails silently, which is the usual answer.

The function runs and returns early every time. It skips entries with no httpRequest.requestUrl. Application logs your service writes go through the same sink unless the filter excludes them, and they have no httpRequest. Tighten the filter.

Events arrive and all of them are rejected. A 401 is the token. Check the secret is bound to the function and the value is the token for this property. Tokens are per property: the token for one property is rejected by another property's endpoint, and the property id in HEADING_INGEST_URL has to be the property the token belongs to. Rotating the token in Heading invalidates the old one immediately.

host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. A load balancer usually fronts more than one hostname, and Cloud Run services answer on a *.run.app URL as well as your domain. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Add a host condition to the sink filter, or return early in the relay when url.hostname is not the property's domain.