> ## Documentation Index > Fetch the complete documentation index at: https://docs.heylua.ai/llms.txt > Use this file to discover all available pages before exploring further. # Object storage > Write agent logs into a bucket you own — S3, Cloudflare R2, or Google Cloud Storage — as gzipped NDJSON you can query with Athena or BigQuery An `s3` drain writes your agents' log records into **a bucket you own**, in your own region, as gzipped NDJSON under a Hive-style key layout. It is the destination a compliance owner asks for, and it is the only one where Lua is not the last hop: retention, encryption, object lock, and the query engine on top are all yours. All three providers are reached through the S3 API — Cloudflare R2 and Google Cloud Storage over their S3-compatible endpoints — so there is one set of rules for the three of them. Object-storage drains are switched on for a deployment as a whole. Where the type is not open yet, a create naming it answers `422 DRAIN_TYPE_UNAVAILABLE`, and the message names the types that are. **It may not be enabled on your deployment yet** — ask [support@heylua.ai](mailto:support@heylua.ai). **Before you begin** * A bucket, and a prefix inside it that only Lua writes to. * Either a key pair or — better — a role Lua can assume. The grant is one action: write into that prefix. * Azure Blob Storage is out of scope: it has no S3-compatible surface. ## What to enter | Field | Flag | Value | | ----------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------- | | Name | `--name` | 1–64 characters, unique in the organization | | Type | `--type s3` | | | Provider | `--provider` | `aws` (the default), `r2`, or `gcs` | | Bucket | `--bucket` | A plain DNS label with **no dots** | | Region | `--region` | The bucket's region; `auto` for R2 | | Prefix | `--prefix` | The key prefix everything is written under | | Endpoint | `--endpoint` | R2 and GCS only. **Refused for `aws`**, whose URL is derived from the bucket and the region | | Credentials | `--credentials-from-env =` | Two environment variable **names**. Neither value touches the command line | | Role | `--role-arn` with `--external-id` | AWS only, instead of a key pair | A dotted bucket name is refused: a dot in a virtual-hosted URL breaks TLS certificate verification, and every provider recommends against it. `--role-arn` and `--credentials-from-env` are mutually exclusive — a drain either assumes a role at send time or holds a key pair, never both. ```bash theme={null} # AWS — the endpoint is derived from the bucket and the region export AWS_KEY_ID=AKIA… AWS_SECRET=… lua drains create --name archive --type s3 \ --bucket acme-compliance-logs --region eu-west-1 --prefix lua/logs \ --credentials-from-env AWS_KEY_ID=AWS_SECRET # An assumable role instead — nothing long-lived is stored lua drains create --name archive --type s3 \ --bucket acme-compliance-logs --region eu-west-1 --prefix lua/logs \ --role-arn arn:aws:iam::123456789012:role/LuaLogDrain --external-id org_4f2c9a1b-archive # Cloudflare R2 — path style, region 'auto' lua drains create --name archive --type s3 --provider r2 \ --bucket acme-logs --region auto \ --endpoint https://<32-hex-account-id>.r2.cloudflarestorage.com \ --credentials-from-env R2_KEY_ID=R2_SECRET # Google Cloud Storage over its S3 interoperability API (an HMAC key pair) lua drains create --name archive --type s3 --provider gcs \ --bucket acme-logs --region EU --endpoint https://storage.googleapis.com \ --credentials-from-env GCS_KEY_ID=GCS_SECRET ``` **The secret access key is never a flag.** A shell history keeps one and so does a CI log. `--credentials-from-env` names two environment variables and reads them in the process that runs the command; neither value appears in `argv`. The access key **id** is stored and shown — SigV4 puts it in the `Authorization` header of every request in clear — while the secret goes to the encrypted credential store and appears in no response, no audit entry, and no error message. ### Which hosts are allowed The host is checked against the three providers' own shapes before any DNS lookup happens, so an arbitrary S3-compatible endpoint is refused: | `--provider` | Host | Style | | ------------ | ----------------------------------------------------------------------- | -------------- | | `aws` | `.s3..amazonaws.com`, derived — `--endpoint` is refused | virtual-hosted | | `r2` | `<32-hex-account-id>.r2.cloudflarestorage.com` | path | | `gcs` | `storage.googleapis.com` | path | Anything else answers `422 DRAIN_ENDPOINT_NOT_ALLOWED`. A deployment can widen the list for its own environment, but a widened host is still required to be `https:` and to resolve entirely to public addresses, re-checked on every send — the same posture as [every other destination](/drains/protecting-your-destination). ## What lands in the bucket ### Object keys ```text theme={null} {prefix}/org={orgId}/agent={agentId}/env={environment}/dt=YYYY-MM-DD/hour=HH/{batchId}.ndjson.gz ``` Hive-style partitioning, because every query engine discovers `key=value` directories as partition columns with no DDL of its own. Each object is **gzipped NDJSON**: one line per record, and every line carries `schemaUrl`, `batchId`, the batch's `resource` block, and one `record`. Each line's `record` is the same shape the [event schema](/drains/event-schema) describes and the same encoder the generic HTTPS drain uses in `ndjson` format, so the two cannot drift. A consumer that splits a file and drops lines still has the tenancy of every line it kept. `format` is always `ndjson` for a bucket drain; `--format json` is not honoured here. Dedup on `record.id`, never on `batchId`. **`dt=` and `hour=` are the *observation* hour — when Lua wrote the file — not the record's own event time.** A batch whose records straddle an hour boundary has to have one home, and a partition a query engine can prune has to be derivable from the key. Every record keeps its own `timestamp` and `observedTimestamp`, so nothing is lost. But a query that reads `dt=` as the event day **will under-count the boundary**: filter on the partition to prune, then filter on `record.timestamp` to be exact. ```sql theme={null} WHERE dt BETWEEN '2026-09-21' AND '2026-09-23' -- prunes AND record.timestamp >= '2026-09-22T00:00:00Z' -- is exact AND record.timestamp < '2026-09-23T00:00:00Z' ``` ### Batching Object storage batches on a **window**, not on a flush tick: **5 minutes or 50 MB**, whichever comes first. The 500-record, 1 MiB ceilings that suit an HTTP intake would cut a five-minute window into twenty half-megabyte objects, and no query engine reads a partition of 43,000 tiny files at a sensible cost. A single flush writes at most 25,000 records, so a very busy drain gets a second object in the window rather than one enormous one. At roughly a kilobyte a record that sits well inside 50 MB, so most windows are one file. A drain holding its window looks like **backlog, not failure**: `lua drains status` shows a non-zero backlog for up to five minutes at a time, by design. A `test` or a `verify` never waits out the window. ### The hourly manifest Every hour partition gets a manifest beside its objects: ```text theme={null} {prefix}/org=.../agent=.../env=.../dt=.../hour=HH/_manifest.json ``` ```json theme={null} { "manifestVersion": 1, "schemaUrl": "https://docs.heylua.ai/schemas/logs/1.0", "orgId": "org_4f2c9a1b", "agentId": "agent_1789214224176_2vta8rnyn", "environment": "production", "dt": "2026-09-22", "hour": "14", "objects": [ { "object": "01K5XJ9R7Q2M4V8B0N3C6D1F5H.ndjson.gz", "batchId": "01K5XJ9R7Q2M4V8B0N3C6D1F5H", "records": 412, "bytes": 918273, "bytesCompressed": 71422, "sha256": "9f2c…", "firstRecordId": "1789217102008-a1c4ppzqx", "lastRecordId": "1789217994301-k0mzr7tly", "writtenAt": "2026-09-22T14:07:33.412Z" } ], "records": 412, "bytes": 918273, "updatedAt": "2026-09-22T14:07:33.412Z" } ``` **To reconcile an hour**: list the partition, read `_manifest.json`, and for each row confirm the object exists, its length equals `bytesCompressed`, and the SHA-256 of the **downloaded bytes** equals `sha256`. The hash is over the stored, gzipped object, so you never have to decompress anything to verify it. ```bash theme={null} aws s3 cp s3://acme-compliance-logs/lua/logs/org=org_4f2c9a1b/agent=agt_7ac1/env=production/dt=2026-09-22/hour=14/_manifest.json - \ | jq -r '.objects[] | "\(.sha256) \(.object)"' > hour-14.sha256 aws s3 sync s3://acme-compliance-logs/lua/logs/org=org_4f2c9a1b/agent=agt_7ac1/env=production/dt=2026-09-22/hour=14/ ./hour-14/ (cd hour-14 && sha256sum -c hour-14.sha256) ``` Four properties worth knowing: * **The merge is idempotent on the file name.** A re-run that writes the same object replaces its row rather than appending a second one, and the totals are recomputed from the rows. * **Entries are sorted by file name**, which is a ULID, which is time order. * **Past 5,000 entries a manifest keeps the oldest rows and sets `"truncated": true`.** The objects are all still there; only the index is short. * **The manifest is best effort.** A delivery is never retried because a manifest write failed — a retry would mint a fresh batch id and write the same records to a second file. If you suspect a manifest is short, rebuild it by listing the partition: every column but `sha256` is derivable from the objects themselves, and `sha256` is derivable from their bytes. Concurrent manifest writes use conditional puts. A provider that ignores them degrades to last-write-wins, which can lose a manifest **row** — never an object. Rebuild from the listing if a partition's manifest and its objects disagree; the objects are the record. ## Prove you own the bucket Writing to a bucket proves somebody gave Lua a key. It does not prove the bucket is **yours**. So before an object-storage drain delivers anything, it writes a challenge object under your prefix and has to see the token come back. ```bash theme={null} lua drains verify drn_2b7f10d9ac4e83615702ffab ``` Lua mints a one-time token and writes `{prefix}/_lua-verify-` into the bucket, with the token as its content. Verification finishes on its own and the drain goes to `healthy`. This is the path when the grant includes `s3:GetObject`. With the write-only grant we recommend, Lua cannot read the object back. The drain stays in `pending_verification`, and the delivery row says `token_not_echoed`. **List your prefix, open the object, and echo the token:** ```bash theme={null} aws s3 ls s3://acme-compliance-logs/lua/logs/ | grep _lua-verify- lua drains confirm drn_2b7f10d9ac4e83615702ffab --token ``` ```text Output theme={null} ✅ Ownership confirmed for drn_2b7f10d9ac4e83615702ffab verified at 2026-09-22T14:11:07.244Z ``` Lua stores only a hash of the token and returns it from **no route** — not the read model, not the delivery row, not the audit trail. Finding it in the bucket *is* the proof: a control plane that could hand the token back would let anyone holding `logs:manage` confirm a bucket they cannot even read. The challenge expires with the verification token, after **10 minutes**. Run `lua drains verify` again to mint a new one; verification is limited to 5 attempts per drain per hour. The read model carries a `challenge.keyPattern` telling you **where to look, not what to send**. There is no route that returns the token. ### When verification fails | Delivery row | What it means | Fix | | -------------------------- | ------------------------------------------------------ | ------------------------------------------------------- | | `s3:AccessDenied` | The grant does not allow `s3:PutObject` on this prefix | Apply the [policy below](#the-bucket-policy) | | `s3:NoSuchBucket` | Wrong bucket name, or the right name in another region | Check `--bucket` and `--region` | | `s3:InvalidAccessKeyId` | The key pair is wrong or deactivated | Re-enter it with `--credentials-from-env` | | `s3:SignatureDoesNotMatch` | The same | The same | | `s3:PermanentRedirect` | The bucket lives in a different region | Fix `--region` | | `url:allowlist` | The endpoint is not one of the three providers' hosts | See [which hosts are allowed](#which-hosts-are-allowed) | | `token_not_echoed` | The object landed; ownership is not proved yet | `lua drains confirm` — this is the normal path | Errors carry the provider's own error **code**, never the SDK's message. An SDK message quotes the signed request, and a signed request carries a credential. ## The bucket policy The grant is deliberately small: write into one prefix, and nothing else. `s3:GetObject` is **optional** — without it the ownership challenge takes the `lua drains confirm` path and the hourly manifest degrades from read-modify-write to write-only. ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "LuaLogDrainWrite", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/lua-log-drain" }, "Action": ["s3:PutObject"], "Resource": "arn:aws:s3:::acme-compliance-logs/lua/logs/*" }, { "Sid": "LuaLogDrainReadBackOptional", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/lua-log-drain" }, "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::acme-compliance-logs/lua/logs/*" } ] } ``` Nothing long-lived is stored. The trust policy on your role — `sts:ExternalId` is the confused-deputy guard, and it is required: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` The permissions policy on the same role is the two statements from the other tab, as `Action`/`Resource` without the `Principal` block. `` is the AWS account whose principal you trust. It is specific to the deployment your organization is on — ask [support@heylua.ai](mailto:support@heylua.ai) for the value, and pass the same `--external-id` you put in the condition. ### Terraform ```hcl theme={null} resource "aws_s3_bucket" "lua_logs" { bucket = "acme-compliance-logs" } resource "aws_s3_bucket_public_access_block" "lua_logs" { bucket = aws_s3_bucket.lua_logs.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } resource "aws_s3_bucket_server_side_encryption_configuration" "lua_logs" { bucket = aws_s3_bucket.lua_logs.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } } } resource "aws_iam_role" "lua_log_drain" { name = "LuaLogDrain" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { AWS = "arn:aws:iam::${var.lua_account_id}:root" } Action = "sts:AssumeRole" Condition = { StringEquals = { "sts:ExternalId" = var.lua_external_id } } }] }) } resource "aws_iam_role_policy" "lua_log_drain" { role = aws_iam_role.lua_log_drain.id policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = ["s3:PutObject", "s3:GetObject"] Resource = "${aws_s3_bucket.lua_logs.arn}/lua/logs/*" }] }) } ``` **Object Lock is your setting, not Lua's.** Lua writes each data object exactly once and never deletes or overwrites one, so compliance mode is compatible with everything above **except the manifest**, which is rewritten as each hour fills. Either keep the manifest outside the locked prefix, or accept that a locked hour's manifest reflects only its first object — the objects themselves are complete either way. ## Query it The Hive layout means a query engine discovers `org`, `agent`, `env`, `dt`, and `hour` as partition columns. ```sql theme={null} CREATE EXTERNAL TABLE lua_logs ( schemaUrl string, batchId string, resource map, record struct< id: string, timestamp: string, observedTimestamp: string, eventName: string, severityNumber: int, severityText: string, body: string, attributes: map > ) PARTITIONED BY (org string, agent string, env string, dt string, hour string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' LOCATION 's3://acme-compliance-logs/lua/logs/'; MSCK REPAIR TABLE lua_logs; ``` ```sql theme={null} -- tool errors in one day, by tool SELECT record.attributes['gen_ai.tool.name'] AS tool, count(*) AS errors FROM lua_logs WHERE env = 'production' AND dt = '2026-09-22' AND record.eventName = 'lua.skill.error' GROUP BY 1 ORDER BY 2 DESC; ``` `_manifest.json` is not NDJSON, so exclude it — either keep manifests in a prefix of their own or add `AND "$path" NOT LIKE '%_manifest.json'` to the queries that would otherwise read one. A `map` coerces every attribute value to a string, which is what you want for the long tail. Declare the handful you compare numerically (`lua.duration_ms`, say) as named struct fields instead. ```sql theme={null} CREATE EXTERNAL TABLE lua_logs WITH PARTITION COLUMNS OPTIONS ( format = 'NEWLINE_DELIMITED_JSON', uris = ['gs://acme-logs/lua/logs/*'], hive_partition_uri_prefix = 'gs://acme-logs/lua/logs', require_hive_partition_filter = true ); ``` `require_hive_partition_filter` is worth leaving on: it turns a full-bucket scan into a query error instead of a bill. ```sql theme={null} SELECT JSON_VALUE(record, '$.attributes."gen_ai.tool.name"') AS tool, count(*) AS errors FROM lua_logs WHERE env = 'production' AND dt = '2026-09-22' AND JSON_VALUE(record, '$.eventName') = 'lua.skill.error' GROUP BY 1 ORDER BY 2 DESC; ``` ```sql theme={null} INSTALL httpfs; LOAD httpfs; SELECT record.eventName, count(*) FROM read_ndjson_auto( 's3://acme-compliance-logs/lua/logs/org=*/agent=*/env=production/dt=2026-09-22/hour=*/*.ndjson.gz', hive_partitioning = true ) GROUP BY 1 ORDER BY 2 DESC; ``` The glob deliberately does not match `_manifest.json`. ## What a read of the drain shows An `s3` drain reads back with its bucket configuration and **no credential**: ```json theme={null} { "id": "drn_7c1e0b44f9a2d5386e10ab93", "type": "s3", "format": "ndjson", "bucket": { "provider": "aws", "bucket": "acme-compliance-logs", "region": "eu-west-1", "prefix": "lua/logs", "credentials": { "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "secretLast4": "b31f" }, "challenge": { "tokenHash": "", "keyPattern": "lua/logs/_lua-verify-*", "issuedAt": "2026-09-22T14:11:07.244Z", "expiresAt": "2026-09-22T14:21:07.244Z" } }, "state": "pending_verification" } ``` * `credentials.secretAccessKey` is **write-only**. It is accepted on `POST` and on `PATCH`, and returned by nothing. * `credentials.accessKeyId` is shown, because SigV4 sends it in clear on every request anyway. * `credentials.secretLast4` is the last four characters of the stored secret, so you can tell one credential from another after a rotation without ever reading one. * `challenge.keyPattern` is **where to look, not what to send**. `tokenHash` is always an empty string on a read: a hash would be no use to you and is one thing less to leak. * `challenge.method` appears once the challenge settles: `readback` when Lua read the object back itself, `confirmed` when you echoed the token. A `PATCH` **replaces the whole `bucket` block**, never merges it. A half-patched destination — a new bucket with the old credentials — is a drain writing somewhere you did not ask for. Omit `bucket` entirely to keep the current one. ## If it isn't working Object-storage drains are not open on your deployment yet. The message names the types that are. Nothing about your bucket or your credentials is wrong. That is the write-only grant working as intended. List your prefix for `_lua-verify-*` and run `lua drains confirm --token `. If the object is not there either, the delivery row will say `s3:AccessDenied` — the grant is missing `s3:PutObject` on that prefix. The manifest is best effort and the objects are the record. Rebuild it by listing the partition. A grant without `s3:GetObject` cannot do read-modify-write, so an hour with several objects may list only the first. That is the window, not a fault. Object storage batches on 5 minutes or 50 MB. Watch the age of the backlog against the window rather than its depth. The bucket is in a different region from `--region`. An AWS endpoint is derived from the bucket and the region, so a wrong region is a wrong host. Recreate the drain with the right one. `dt=` is the hour Lua **wrote** the file, not the hour the events happened. Read one partition on either side and filter on `record.timestamp`. See the [partition caveat](#object-keys). ## Next steps Every field on the record inside each line. Why an arbitrary endpoint is refused, and what is re-checked on every send. A one-off NDJSON export, for a window that predates the drain. Every verb and flag, including `confirm`.