Skip to main content
A log drain is a buffered, at-least-once pipeline. This page is the contract: what is promised, what is not, and what you see when something gives.

​
At-least-once

Every record is delivered at least once, and may be delivered more than once. Duplicates happen for ordinary reasons: a delivery that timed out after your receiver had already stored the batch, a worker that restarted mid-flight, a retry that overlaps a slow success. They are not a fault condition.
Consumers dedup on records[].id, never on X-Lua-Batch-Id. A retried record re-enters the queue and is re-sent under a new batch id, alongside whatever else is due at that moment — so the set of records in a batch is not stable across attempts. Keep the record id for at least six hours, the retry horizon, and you cannot receive an old duplicate you have forgotten.
Ordering is not promised either. Records are claimed oldest-first and a failed batch goes back to the queue, so a retried record can arrive after a newer one. Sort by timestamp, and within one execution by lua.execution.seq, rather than trusting arrival order.

​
Batching

Records are collected continuously and flushed about every 2 seconds. A batch is cut when the destination’s record or byte cap is reached, whichever comes first. One batch carries exactly one resource. Records from three agents become three batches — never one batch with a mixed resource — so your ingest never has to reconcile a resource per record. Bodies are always gzipped. A record larger than a destination’s per-record cap has its body cut and suffixed …[truncated for destination] rather than being dropped — the event, its severity, and its attributes still arrive. At most 4 requests are in flight to any one drain at a time. That is what stops a slow destination starving the others: one struggling drain can occupy four delivery slots, never the pool.

​
Retry and backoff

Each attempt gets 10 seconds. A failure is either retryable or terminal. Retryable: a network error, a DNS failure, a timeout, and any status in the destination’s retried list above. The batch goes back to the queue with a delay drawn uniformly from zero up to a ceiling that doubles per attempt and stops at 60 seconds — full jitter, so a destination coming back up is not hit by every drain’s backlog at the same instant. A Retry-After header — delta-seconds or an HTTP date — overrides that and is honoured exactly, clamped at one hour. Terminal: every 3xx (redirects are refused, never followed) and every 4xx other than 429 — and, for OTLP, 400 and 500 explicitly. A terminal failure drops the batch, counts it under rejected, and does not retry, because the same bytes would be refused again. OTLP is the one destination where a 500 is terminal: its specification allows a client to retry 429, 502, 503, and 504, and nothing else. The other three destinations retry a 500 like any server error.

​
The six-hour horizon

A record that has been waiting more than 6 hours is dropped, whatever the reason for the wait. That bounds the buffer in time: an endpoint that comes back after a long outage receives the last six hours, not a six-day flood that would knock it over again.

​
Auto-pause

A drain with no successful delivery for 24 hours, and at least one failure in that window, pauses itself with reason auto. It keeps buffering — it just stops trying. Fix the destination, then:
Resuming always lands in degraded, never straight in healthy.

​
Error classes

Every attempt is recorded with one class, which is what lua drains deliveries prints and what a support conversation starts from.
Output
Delivery records are kept for 7 days. They hold the status code, latency, byte count, error class, and up to 1 KB of the destination’s response, scrubbed. They never hold what was sent.

​
Backpressure and drops

The buffer is bounded in size as well as in time. When a drain’s queue exceeds its limit, the oldest pending records are dropped — keeping the newest is what you want during an incident, because the newest records are the ones describing it. You are told, in band. On the first successful delivery after a drop, the drain sends one synthetic record:
dropped_from and dropped_to bound the gap by record id, so you can reconcile against lua logs — the platform’s own copy is unaffected by a drain drop and is still readable for the full retention window. Alert on lua.drain.dropped: it means your pipeline lost data and is the one event that cannot be inferred from the absence of something else. The monitor packs include it.

​
Quota degradation

Going over the daily quota degrades the drain in steps rather than cutting it off. Dropped-for-quota records count the same way, and error records are the last to go — the ladder never drops severity WARN or ERROR before it pauses the drain. The 80% notification is one-shot by crossing, not by state: it fires on the batch that takes the drain from under 0.8 of its allowance to at or over it, and not again until the window rolls. A drain sitting at 0.9 all afternoon is not a drain that notifies all afternoon. Counters reset at 00:00 UTC, and a drain the ladder paused with reason quota resumes itself at that boundary — you do not have to run lua drains resume. Like every resume it lands in degraded rather than straight back in healthy. The drain’s own synthetic records — lua.drain.test, lua.drain.heartbeat, lua.drain.dropped, and lua.truncated.warn — do not count against the quota, so a heartbeat every five minutes never eats an allowance you are paying for, and a drain cannot be pushed over the ladder by its own bookkeeping. They do count in health.deliveredCount24h and appear in the deliveries log, which is why that number can be larger than the records your agents produced. lua drains status --json carries quota.usedEvents, quota.usedBytes, quota.resetsAt, and quota.degradation.

​
Heartbeat

Every 5 minutes, a drain in state healthy sends one synthetic record:
It exists so that silence is not ambiguous. Without it, “no agent errors in the last hour” and “the pipeline has been down for an hour” look identical at the destination. With it, the absence of a heartbeat is itself an alertable event — and it is the first monitor in every pack here. Two limits on what it proves:
  • It is sent only in state healthy. A degraded or failing drain stops heartbeating, which is intended: the heartbeat going missing is exactly the signal you want when deliveries start failing.
  • It proves the delivery path, not the selection path. A heartbeat arriving while a selector excludes every one of your agents still means no log records will come.
lua.drain.backlog is how many records are waiting. A backlog that climbs across successive heartbeats means your destination is slower than your agents.

​
Health states

The transitions: paused keeps collecting records — within the six-hour horizon and the buffer size — so a drain paused for 20 minutes during a deploy loses nothing. disabled stops collecting: nothing is queued while a drain is off, and turning it back on starts from that moment.
lua drains status exits 2 when any drain is failing. 2 is also the usage exit code, so a guard that must tell a failing drain from a mistyped command reads the state field rather than the exit status.

​
Next steps

Datadog monitor pack

Alerts for heartbeat absence, error spikes, and drops.

Grafana dashboard

The same signals over Loki.

Event schema

The synthetic records named on this page.

lua drains

status, deliveries, pause, and resume.