> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heylua.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Delivery guarantees
> At-least-once delivery, batching, retry and backoff, the six-hour horizon, drops and how you are told about them, the heartbeat, and the health state machine
A [log drain](/drains/overview) is a buffered, at-least-once pipeline. This page is the contract: what is promised, what is not, and what you see when something gives.
## At-least-once
**Every record is delivered at least once, and may be delivered more than once.**
Duplicates happen for ordinary reasons: a delivery that timed out after your receiver had already stored the batch, a worker that restarted mid-flight, a retry that overlaps a slow success. They are not a fault condition.
**Consumers dedup on `records[].id`, never on `X-Lua-Batch-Id`.** A retried record re-enters the queue and is re-sent under a **new** batch id, alongside whatever else is due at that moment — so the set of records in a batch is not stable across attempts. Keep the record id for at least six hours, the retry horizon, and you cannot receive an old duplicate you have forgotten.
Ordering is not promised either. Records are claimed oldest-first and a failed batch goes back to the queue, so a retried record can arrive after a newer one. Sort by `timestamp`, and within one execution by `lua.execution.seq`, rather than trusting arrival order.
## Batching
Records are collected continuously and flushed about every 2 seconds. A batch is cut when the destination's record or byte cap is reached, whichever comes first.
**One batch carries exactly one `resource`.** Records from three agents become three batches — never one batch with a mixed resource — so your ingest never has to reconcile a resource per record.
| Destination | Records per batch | Uncompressed bytes | Per record | Retried statuses | Terminal |
| -------------------------------------- | ----------------- | ------------------ | ---------- | ---------------------------------------- | -------------------------------------------- |
| [Generic HTTPS](/drains/generic-https) | 500 | 1 MiB | — | `408`, `429`, `500`, `502`, `503`, `504` | Every other `4xx`, every `3xx` |
| [OTLP](/drains/opentelemetry) | 500 | 1 MiB | — | `429`, `502`, `503`, `504` | `400`, `500`, every other `4xx`, every `3xx` |
| [Datadog](/drains/datadog) | 1,000 | 5 MiB | 1 MiB | `408`, `429`, `500`, `502`, `503`, `504` | Every other `4xx`, every `3xx` |
| [Better Stack](/drains/better-stack) | 500 | 1 MiB | — | `408`, `429`, `500`, `502`, `503`, `504` | Every other `4xx`, every `3xx` |
Bodies are always gzipped. A record larger than a destination's per-record cap has its `body` cut and suffixed `…[truncated for destination]` rather than being dropped — the event, its severity, and its attributes still arrive.
At most 4 requests are in flight to any one drain at a time. That is what stops a slow destination starving the others: one struggling drain can occupy four delivery slots, never the pool.
## Retry and backoff
Each attempt gets 10 seconds. A failure is either **retryable** or **terminal**.
Retryable: a network error, a DNS failure, a timeout, and any status in the destination's retried list above. The batch goes back to the queue with a delay drawn uniformly from zero up to a ceiling that doubles per attempt and stops at 60 seconds — full jitter, so a destination coming back up is not hit by every drain's backlog at the same instant.
| Consecutive failures | Delay drawn from |
| -------------------- | ---------------- |
| 1 | 0–2 s |
| 2 | 0–4 s |
| 3 | 0–8 s |
| 4 | 0–16 s |
| 5 | 0–32 s |
| 6 and after | 0–60 s |
A `Retry-After` header — delta-seconds or an HTTP date — overrides that and is honoured exactly, clamped at one hour.
Terminal: every `3xx` (redirects are refused, never followed) and every `4xx` other than `429` — and, for OTLP, `400` and `500` explicitly. A terminal failure drops the batch, counts it under `rejected`, and does not retry, because the same bytes would be refused again.
OTLP is the one destination where a `500` is terminal: its specification allows a client to retry `429`, `502`, `503`, and `504`, and nothing else. The other three destinations retry a `500` like any server error.
### The six-hour horizon
A record that has been waiting more than **6 hours** is dropped, whatever the reason for the wait. That bounds the buffer in time: an endpoint that comes back after a long outage receives the last six hours, not a six-day flood that would knock it over again.
### Auto-pause
A drain with no successful delivery for **24 hours**, and at least one failure in that window, pauses itself with reason `auto`. It keeps buffering — it just stops trying. Fix the destination, then:
```bash theme={null}
lua drains resume drn_9f31a7c04b2e615d8a03cc71
```
Resuming always lands in `degraded`, never straight in `healthy`.
### Error classes
Every attempt is recorded with one class, which is what `lua drains deliveries` prints and what a support conversation starts from.
| Class | Meaning |
| ---------- | -------------------------------------------------------------------------------------------------------- |
| `network` | The connection failed or was reset |
| `timeout` | No response within 10 seconds |
| `dns` | The hostname did not resolve |
| `ssrf` | The endpoint resolved to a private, loopback, link-local, or metadata address |
| `redirect` | A `3xx`. Never followed |
| `http_4xx` | A terminal client error |
| `http_429` | Rate limited. Retried |
| `http_5xx` | A server error. Retried — except on OTLP, where only `502`, `503`, and `504` are and a `500` is terminal |
| `rejected` | The destination accepted the request but refused the records |
| `partial` | Delivered, with some records refused — counted, not retried |
| `encode` | The batch could not be encoded for this destination |
```bash theme={null}
lua drains deliveries drn_9f31a7c04b2e615d8a03cc71 --limit 5
```
```text Output theme={null}
STARTED KIND RECORDS BYTES STATUS LATENCY RESULT
2026-09-21T09:14:02Z batch 412 284.1 KB 202 138 ms ok
2026-09-21T09:13:58Z heartbeat 1 412 B 202 96 ms ok
2026-09-21T09:13:44Z batch 500 331.7 KB 503 9 s http_5xx
2026-09-21T09:13:31Z batch 500 330.2 KB 503 9 s http_5xx
2026-09-21T09:08:47Z verify 1 508 B 200 121 ms ok
```
Delivery records are kept for 7 days. They hold the status code, latency, byte count, error class, and up to 1 KB of the destination's *response*, scrubbed. They never hold what was sent.
## Backpressure and drops
The buffer is bounded in size as well as in time. When a drain's queue exceeds its limit, the **oldest** pending records are dropped — keeping the newest is what you want during an incident, because the newest records are the ones describing it.
You are told, in band. On the first successful delivery after a drop, the drain sends one synthetic record:
```json theme={null}
{
"eventName": "lua.drain.dropped",
"severityNumber": 13,
"severityText": "WARN",
"attributes": {
"lua.source": "drain",
"lua.drain.id": "drn_9f31a7c04b2e615d8a03cc71",
"lua.drain.dropped_count": 8412,
"lua.drain.dropped_from": "1789217102008-a1c4ppzqx",
"lua.drain.dropped_to": "1789217994301-k0mzr7tly"
}
}
```
`dropped_from` and `dropped_to` bound the gap by record id, so you can reconcile against `lua logs` — the platform's own copy is unaffected by a drain drop and is still readable for the full [retention window](/concepts/security-and-data#retention).
Alert on `lua.drain.dropped`: it means your pipeline lost data and is the one event that cannot be inferred from the absence of something else. The [monitor packs](/drains/packs/datadog-monitors) include it.
### Quota degradation
Going over the daily quota degrades the drain in steps rather than cutting it off. Dropped-for-quota records count the same way, and `error` records are the last to go — the ladder never drops severity `WARN` or `ERROR` before it pauses the drain.
| Usage | Effect |
| ----- | ------------------------------------------------ |
| 80% | A notification, once per window. Nothing dropped |
| 100% | `DEBUG` records dropped |
| 125% | `INFO` records dropped too |
| 150% | The drain pauses, with reason `quota` |
The 80% notification is one-shot **by crossing**, not by state: it fires on the batch that takes the drain from under 0.8 of its allowance to at or over it, and not again until the window rolls. A drain sitting at 0.9 all afternoon is not a drain that notifies all afternoon.
Counters reset at 00:00 UTC, and a drain the ladder paused with reason `quota` **resumes itself at that boundary** — you do not have to run `lua drains resume`. Like every resume it lands in `degraded` rather than straight back in `healthy`.
The drain's own [synthetic records](/drains/event-schema#eventname) — `lua.drain.test`, `lua.drain.heartbeat`, `lua.drain.dropped`, and `lua.truncated.warn` — do **not** count against the quota, so a heartbeat every five minutes never eats an allowance you are paying for, and a drain cannot be pushed over the ladder by its own bookkeeping. They do count in `health.deliveredCount24h` and appear in the deliveries log, which is why that number can be larger than the records your agents produced.
`lua drains status --json` carries `quota.usedEvents`, `quota.usedBytes`, `quota.resetsAt`, and `quota.degradation`.
## Heartbeat
Every 5 minutes, a drain in state `healthy` sends one synthetic record:
```json theme={null}
{
"eventName": "lua.drain.heartbeat",
"severityNumber": 9,
"severityText": "INFO",
"attributes": {
"lua.source": "drain",
"lua.drain.id": "drn_9f31a7c04b2e615d8a03cc71",
"lua.drain.backlog": 0
}
}
```
It exists so that **silence is not ambiguous**. Without it, "no agent errors in the last hour" and "the pipeline has been down for an hour" look identical at the destination. With it, the absence of a heartbeat is itself an alertable event — and it is the first monitor in every pack here.
Two limits on what it proves:
* It is sent only in state `healthy`. A `degraded` or `failing` drain stops heartbeating, which is intended: the heartbeat going missing is exactly the signal you want when deliveries start failing.
* It proves the delivery path, not the selection path. A heartbeat arriving while a selector excludes every one of your agents still means no log records will come.
`lua.drain.backlog` is how many records are waiting. A backlog that climbs across successive heartbeats means your destination is slower than your agents.
## Health states
| State | Delivering? | Buffering? | Meaning |
| ---------------------- | ----------- | ---------- | ------------------------------------------------------------------ |
| `pending_verification` | No | Yes | Created, or the endpoint or type changed. Ownership not yet proved |
| `healthy` | Yes | — | Verified and succeeding. The only state that heartbeats |
| `degraded` | Yes | — | Failing more than 10% of attempts, or just resumed |
| `failing` | Yes | — | No success for 15 minutes. Still trying |
| `paused` | No | Yes | Stopped by you, by the quota ladder, or automatically |
| `disabled` | No | No | Off. Nothing is queued while it is off |
The transitions:
| From | On | To |
| -------------------------------- | -------------------------------------------------------------------------- | ----------------------- |
| `pending_verification` | Ownership verified | `healthy` |
| `healthy` | More than 10% of the last 5 minutes' attempts failed (at least 5 attempts) | `degraded` |
| `degraded` | Failure ratio back at or below 10% over 5 minutes | `healthy` |
| `degraded` | No success for 15 minutes | `failing` |
| `failing` | Any `2xx` | `degraded` |
| `failing` | 24 hours without a success | `paused` (`auto`) |
| Any | Quota at 150% | `paused` (`quota`) |
| `paused` (`quota`) | The daily counters reset at 00:00 UTC | `degraded` |
| Any | `lua drains pause` | `paused` (`manual`) |
| Any | Paused by Lua operations | `paused` (`operations`) |
| `paused` | `lua drains resume` | `degraded` |
| `healthy`, `degraded`, `failing` | Endpoint or type changed | `pending_verification` |
| `disabled` | Re-enabled, endpoint unchanged | `degraded` |
| `disabled` | Re-enabled, endpoint or type changed | `pending_verification` |
`paused` keeps collecting records — within the six-hour horizon and the buffer size — so a drain paused for 20 minutes during a deploy loses nothing. `disabled` stops collecting: nothing is queued while a drain is off, and turning it back on starts from that moment.
```bash theme={null}
lua drains status --json | jq -r '.drains[] | [.name, .state, .health.backlog, .health.lastSuccessAt] | @tsv'
```
`lua drains status` exits `2` when any drain is `failing`. `2` is also the usage exit code, so a guard that must tell a failing drain from a mistyped command reads the `state` field rather than the exit status.
## Next steps
Alerts for heartbeat absence, error spikes, and drops.
The same signals over Loki.
The synthetic records named on this page.
`status`, `deliveries`, `pause`, and `resume`.