Akamai DataStream 2
Stream an Akamai delivery configuration's request logs to Heading over a custom HTTPS destination
DataStream 2 pushes your edge logs to a custom HTTPS destination every 30 or 60 seconds. It is configuration only, and you choose exactly which fields leave Akamai.
Official reference: Stream logs to a custom HTTPS endpoint and Data set parameters.
INGEST_HOST is a placeholder
INGEST_HOST stands in for the ingest hostname, which is not final yet. The setup dialog in Heading shows the real endpoint and property id. Copy them from there and replace INGEST_HOST everywhere it appears on this page.
The trap to read first
DataStream 2 lets you add one custom header, and it will not let that header be Authorization. Akamai's documentation states that custom header values containing Content-Type, Encoding, Authorization, Host or Akamai are not supported. Basic authentication is offered instead, which is a username and a password, and Heading issues neither.
So on this recipe the token travels in the destination URL, percent-encoded:
https://INGEST_HOST/api/ingest/crawl/PROPERTY_ID?token=YOUR_TOKENPercent-encode anything in the token outside A-Z a-z 0-9 - _ . ~. The whole credential now lives in your stream configuration, so treat that configuration as a secret store.
Create the stream
Start a new stream
In Akamai Control Center, open DataStream and create a stream. Give it a name, choose the group, and select the delivery configurations (properties) that serve the domain you are connecting.
Select the data set parameters
Select exactly these eight:
| Parameter | Used for |
|---|---|
reqTimeSec | When the request happened |
reqHost | The host check against the property's domain |
reqPath | The path |
reqMethod | Recorded with the request |
statusCode | What your edge served the crawler |
UA | Crawler identification |
referer | AI-referred entries |
cliIP | Crawler verification only. Checked against published operator ranges, then discarded |
Leave queryStr off. Heading strips query strings on arrival, so sending them only moves data that is going to be thrown away.
DataStream 2 URL-encodes UA and referer, so they arrive looking like Mozilla%2F5.0+%28…%29. That is expected and Heading decodes them.
Choose the JSON log format
Set the log file format to JSON. The structured format is space or tab delimited and carries no field names, so a change to your parameter selection would silently shift every value.
Configure the destination
| Field | Value |
|---|---|
| Destination | Custom HTTPS |
| Endpoint URL | https://INGEST_HOST/api/ingest/crawl/PROPERTY_ID?token=YOUR_TOKEN |
| Authentication | None |
| Content type | application/json |
| Send compressed data | Leave unchecked |
| Push frequency | 30 or 60 seconds, whichever you prefer |
| Sampling rate | 100 |
Sampling has to be 100. Heading counts crawler requests, and at a lower rate a crawler that fetched 40 pages reads as a fraction of that, with no way to correct it later.
Activate the stream, and the behaviour
Activate the stream from the Summary tab. Then, in Property Manager, make sure the DataStream behaviour is enabled on the delivery configuration and that the version carrying it is activated.
An active stream on a property whose DataStream behaviour is off sends nothing, and nothing in the DataStream interface says so. This is the most common reason a correctly configured stream stays silent.
What Heading stores
The same answer appears on every recipe page, because it is the question your client's security reviewer will ask.
| Data | Stored |
|---|---|
| Request path | Yes, with the query string stripped before anything is written |
| Host | Yes, and it has to match the property's domain |
| User agent | Yes. It is how a crawler is identified |
| Status code | Yes. What your origin served the crawler is half the value |
| Time of the request | Yes, as a UTC timestamp |
| Referer | Yes, matched against the AI platforms Heading already knows |
| IP address | No. Checked against the operator's published ranges at ingest, then discarded |
| Cookies | No. Never read, never stored |
| Request or response body | No. Never read, never stored |
| Query string | No. Stripped from the path on arrival |
On this recipe the eight parameters above are the only ones the stream carries. Raw events are kept for a short window and pruned as new ones arrive. The daily counts built from them are what the page reads afterwards. Because no IP, cookie, query string or body is retained, connecting a log source adds no personal data to Heading and brings in no new subprocessor.
Check it worked
Setup is not finished when you activate the stream. It is finished when a real event lands.
Open Tracking → Outcomes, find the Crawl activity card, and select Set up. The delivery log there lists the attempts the endpoint has seen, accepted and rejected alike, and it updates within a few seconds of one arriving. Each row carries the response code, how many log lines were in the payload, how many of them matched, and, for a rejection, the reason and the fix. Two attempts leave no row at all: a request carrying no token, and a request sent to a property id that does not exist. An empty log under a sender you know is running means one of those two, so check the header and the URL before anything else.
| Status | Meaning |
|---|---|
| Not verified | Nothing has arrived yet. Setup is not finished |
| Connected | Batches are arriving and the most recent one was accepted |
| Rejecting | Batches are arriving and every one is being turned away. Nothing is being recorded |
| Nothing arriving | Batches arrived before and none has arrived for 24 hours. Check the sender at your edge is still pointed here |
| Disconnected | Someone switched ingestion off for this property in Heading. Anything still arriving is refused with a 403 and nothing from it is stored. Reconnect from the same dialog to start accepting events again |
An accepted delivery that stores nothing is normal. Most requests to a site are not crawler requests. What is worth looking at is a full day of deliveries where lines received is high and lines kept is zero.
Troubleshooting
Nothing is arriving. Check the DataStream behaviour is enabled on the delivery configuration and that the property version carrying it is activated, on the network your traffic uses. Then check the stream itself is activated and the delivery configurations selected on it are the ones serving this domain.
Events arrive and all of them are rejected. A 401 is the token. On this recipe the usual cause is encoding: a token pasted into the URL with a character that needed escaping. Rebuild the endpoint URL with the token percent-encoded. Tokens are also per property, so the property id in the path has to be the property the token belongs to, and a token rotated in Heading stops working immediately.
host_not_matched. A log line whose host is not the property's domain is dropped rather than counted as this property's traffic. The batch around it is still accepted, because the other lines in it may be yours, so this shows up as a note on an accepted delivery and never as a rejected one. An Akamai delivery configuration often serves several hostnames. The note counts the dropped lines and names the hosts they were for, up to three, most frequent first. Narrow the stream to the delivery configurations that serve this domain, or connect the other hostname as its own property in Heading with its own endpoint and token.
Deliveries look empty or unreadable. Check Send compressed data is unchecked and the log file format is JSON rather than structured.