Skip to main content
A log drain is a buffered, at-least-once pipeline. This page is the contract: what is promised, what is not, and what you see when something gives.

​
At-least-once

Every record is delivered at least once, and may be delivered more than once. Duplicates happen for ordinary reasons: a delivery that timed out after your receiver had already stored the batch, a worker that restarted mid-flight, a retry that overlaps a slow success. They are not a fault condition.
Consumers dedup on records[].id, never on X-Lua-Batch-Id. A retried record re-enters the queue and is re-sent under a new batch id, alongside whatever else is due at that moment — so the set of records in a batch is not stable across attempts. Keep the record id for at least six hours, the retry horizon, and you cannot receive an old duplicate you have forgotten.
Ordering is not promised either. Records are claimed oldest-first and a failed batch goes back to the queue, so a retried record can arrive after a newer one. Sort by timestamp, and within one execution by lua.execution.seq, rather than trusting arrival order.

​
Batching

Records are collected continuously and flushed about every 2 seconds. A batch is cut when the destination’s record or byte cap is reached, whichever comes first. Which of the two binds depends on how large your records are, and it is worth knowing which you are near. A record averages a few hundred bytes, so the record cap is usually what cuts a batch. Add several kilobytes of your own fields per record and the byte cap cuts it instead, well short of the record count. The flush cadence does not change, so that shows up as more batches of fewer records each — not as delay, and not as backlog. One batch carries exactly one resource. Records from three agents become three batches — never one batch with a mixed resource — so your ingest never has to reconcile a resource per record. Every destination but OTLP retries 408, 429, and every 5xx: a server-side failure is worth retrying whether or not that particular status is on anyone’s list. OTLP is narrower because its own specification names the statuses a client may retry, and 400 and 500 are not among them. Object storage is the one destination that does not batch on the 2-second flush. It accumulates for 5 minutes or 50 MB, whichever comes first, because no query engine reads a partition of thousands of tiny objects at a sensible cost. A drain holding its window shows a non-zero backlog by design — watch the backlog’s age against the window rather than its depth. Bodies are always gzipped. A record larger than a destination’s per-record cap has its body cut and suffixed rather than being dropped — the event, its severity, and its attributes still arrive. The marker is …[truncated for destination] where the platform cuts a record to fit a destination’s cap, and ... [truncated by lua] where a vendor encoder cuts it to fit that vendor’s per-event ceiling. At most 4 requests are in flight to any one drain at a time. That is what stops a slow destination starving the others: one struggling drain can occupy four delivery slots, never the pool.

​
Retry and backoff

Each attempt gets 10 seconds. A failure is either retryable or terminal. Retryable: a network error, a DNS failure, a timeout, and any status in the destination’s retried list above. The batch goes back to the queue with a delay drawn uniformly from zero up to a ceiling that doubles per attempt and stops at 60 seconds — full jitter, so a destination coming back up is not hit by every drain’s backlog at the same instant. A Retry-After header — delta-seconds or an HTTP date — overrides that and is honoured exactly, clamped at one hour. Terminal: every 3xx (redirects are refused, never followed) and every 4xx other than 429 — and, for OTLP, 400 and 500 explicitly. A terminal failure drops the batch, counts it under rejected, and does not retry, because the same bytes would be refused again. OTLP is the one destination where a 500 is terminal: its specification allows a client to retry 429, 502, 503, and 504, and nothing else. The other three destinations retry a 500 like any server error.

​
The six-hour horizon

A record that has been waiting more than 6 hours is dropped, whatever the reason for the wait. That bounds the buffer in time: an endpoint that comes back after a long outage receives the last six hours, not a six-day flood that would knock it over again. The horizon is measured on the record’s own timestamp, not on when it was queued or claimed. That matters for a replay, below: a record put back in the queue is judged on how old it really is.

​
Replaying a batch that was lost

A terminal refusal drops the whole batch. When the reason has since been fixed — a receiver bug, a WAF rule, one malformed record — the records are still there for a while, and you can put them back in the queue:
Output
The records are re-queued, due immediately, and delivered in a new batch with a new id. The attempt that carries them is marked replayOf: "<the batch id you asked for>" on the deliveries list, which is how you find it. The command does not wait for the delivery — it returns as soon as the records are queued. The REST route is the same operation.
A replay re-sends the whole batch, including the records your receiver already accepted. A partially accepted batch arrives with its 499 good records again alongside the one you repaired.That is safe under the contract you already have — consumers dedup on records[].id, and a replay does not change a record id — and it is unsafe if you deduplicated on the batch id instead, which is the one thing this page asks you not to do.
The window is six hours, not the seven days a delivery row is kept. The horizon above is measured on each record’s own timestamp, so replaying something older would quietly re-lose it on the next flush. It is refused instead: Only kind: "batch" rows are replayable. A test, a verify, a heartbeat, a dropped or a truncated row is the drain talking about itself, and each has its own verb. Every replay writes a logs.drain.replayed entry to your organization’s audit log, naming the batch and the record count.

​
Auto-pause

A drain with no successful delivery for 24 hours, and at least one failure in that window, pauses itself with reason auto. It keeps buffering — it just stops trying. Fix the destination, then:
Resuming always lands in degraded, never straight in healthy.

​
Error classes

Every attempt is recorded with one class, which is what lua drains deliveries prints and what a support conversation starts from.
Output
Delivery records are kept for 7 days. They hold the status code, latency, byte count, error class, and up to 1 KB of the destination’s response, scrubbed. They never hold what was sent.

​
Backpressure and drops

The buffer is bounded in size as well as in time. When a drain’s queue exceeds its limit, the oldest pending records are dropped — keeping the newest is what you want during an incident, because the newest records are the ones describing it. You are told, in band. On the first successful delivery after a drop, the drain sends one synthetic record:
dropped_from and dropped_to bound the gap by record id, so you can reconcile against lua logs — the platform’s own copy is unaffected by a drain drop and is still readable for the full retention window. Alert on lua.drain.dropped: it means your pipeline lost data and is the one event that cannot be inferred from the absence of something else. The monitor packs include it.

​
Quota degradation

Going over the daily quota degrades the drain in steps rather than cutting it off. Dropped-for-quota records count the same way, and error records are the last to go — the ladder never drops severity WARN or ERROR before it pauses the drain. The 80% notification is one-shot by crossing, not by state: it fires on the batch that takes the drain from under 0.8 of its allowance to at or over it, and not again until the window rolls. A drain sitting at 0.9 all afternoon is not a drain that notifies all afternoon. Counters reset at 00:00 UTC, and a drain the ladder paused with reason quota resumes itself at that boundary — you do not have to run lua drains resume. Like every resume it lands in degraded rather than straight back in healthy. The drain’s own synthetic records — lua.drain.test, lua.drain.heartbeat, lua.drain.dropped, and lua.truncated.warn — do not count against the quota, so a heartbeat every five minutes never eats an allowance you are paying for, and a drain cannot be pushed over the ladder by its own bookkeeping. They do count in health.deliveredCount24h and appear in the deliveries log, which is why that number can be larger than the records your agents produced. lua drains status --json carries quota.usedEvents, quota.usedBytes, quota.resetsAt, and quota.degradation.

​
Heartbeat

Every 5 minutes, a drain in state healthy sends one synthetic record:
It exists so that silence is not ambiguous. Without it, “no agent errors in the last hour” and “the pipeline has been down for an hour” look identical at the destination. With it, the absence of a heartbeat is itself an alertable event — and it is the first monitor in every pack here. Two limits on what it proves:
  • It is sent only in state healthy. A degraded or failing drain stops heartbeating, which is intended: the heartbeat going missing is exactly the signal you want when deliveries start failing.
  • It proves the delivery path, not the selection path. A heartbeat arriving while a selector excludes every one of your agents still means no log records will come.
lua.drain.backlog is how many records are waiting. A backlog that climbs across successive heartbeats means your destination is slower than your agents.

​
Health states

The transitions: paused keeps collecting records — within the six-hour horizon and the buffer size — so a drain paused for 20 minutes during a deploy loses nothing. disabled stops collecting: nothing is queued while a drain is off, and turning it back on starts from that moment.
lua drains status exits 2 when any drain is failing. 2 is also the usage exit code, so a guard that must tell a failing drain from a mistyped command reads the state field rather than the exit status.

​
Next steps

Datadog monitor pack

Alerts for heartbeat absence, error spikes, and drops.

Grafana dashboard

The same signals over Loki.

Event schema

The synthetic records named on this page.

lua drains

status, deliveries, pause, and resume.