Object storage

An s3 drain writes your agents’ log records into a bucket you own, in your own region, as gzipped NDJSON under a Hive-style key layout. It is the destination a compliance owner asks for, and it is the only one where Lua is not the last hop: retention, encryption, object lock, and the query engine on top are all yours. All three providers are reached through the S3 API — Cloudflare R2 and Google Cloud Storage over their S3-compatible endpoints — so there is one set of rules for the three of them.

Object-storage drains are switched on for a deployment as a whole. Where the type is not open yet, a create naming it answers 422 DRAIN_TYPE_UNAVAILABLE, and the message names the types that are. It may not be enabled on your deployment yet — ask [email protected].

Before you begin

A bucket, and a prefix inside it that only Lua writes to.
Either a key pair or — better — a role Lua can assume. The grant is one action: write into that prefix.
Azure Blob Storage is out of scope: it has no S3-compatible surface.

What to enter

Field	Flag	Value
Name	`--name`	1–64 characters, unique in the organization
Type	`--type s3`
Provider	`--provider`	`aws` (the default), `r2`, or `gcs`
Bucket	`--bucket`	A plain DNS label with no dots
Region	`--region`	The bucket’s region; `auto` for R2
Prefix	`--prefix`	The key prefix everything is written under
Endpoint	`--endpoint`	R2 and GCS only. Refused for `aws`, whose URL is derived from the bucket and the region
Credentials	`--credentials-from-env <KEY_ID_VAR>=<SECRET_VAR>`	Two environment variable names. Neither value touches the command line
Role	`--role-arn` with `--external-id`	AWS only, instead of a key pair

A dotted bucket name is refused: a dot in a virtual-hosted URL breaks TLS certificate verification, and every provider recommends against it. --role-arn and --credentials-from-env are mutually exclusive — a drain either assumes a role at send time or holds a key pair, never both.

# AWS — the endpoint is derived from the bucket and the region
export AWS_KEY_ID=AKIA… AWS_SECRET=…
lua drains create --name archive --type s3 \
  --bucket acme-compliance-logs --region eu-west-1 --prefix lua/logs \
  --credentials-from-env AWS_KEY_ID=AWS_SECRET

# An assumable role instead — nothing long-lived is stored
lua drains create --name archive --type s3 \
  --bucket acme-compliance-logs --region eu-west-1 --prefix lua/logs \
  --role-arn arn:aws:iam::123456789012:role/LuaLogDrain --external-id org_4f2c9a1b-archive

# Cloudflare R2 — path style, region 'auto'
lua drains create --name archive --type s3 --provider r2 \
  --bucket acme-logs --region auto \
  --endpoint https://<32-hex-account-id>.r2.cloudflarestorage.com \
  --credentials-from-env R2_KEY_ID=R2_SECRET

# Google Cloud Storage over its S3 interoperability API (an HMAC key pair)
lua drains create --name archive --type s3 --provider gcs \
  --bucket acme-logs --region EU --endpoint https://storage.googleapis.com \
  --credentials-from-env GCS_KEY_ID=GCS_SECRET

The secret access key is never a flag. A shell history keeps one and so does a CI log. --credentials-from-env names two environment variables and reads them in the process that runs the command; neither value appears in argv. The access key id is stored and shown — SigV4 puts it in the Authorization header of every request in clear — while the secret goes to the encrypted credential store and appears in no response, no audit entry, and no error message.

Which hosts are allowed

The host is checked against the three providers’ own shapes before any DNS lookup happens, so an arbitrary S3-compatible endpoint is refused:

`--provider`	Host	Style
`aws`	`<bucket>.s3.<region>.amazonaws.com`, derived — `--endpoint` is refused	virtual-hosted
`r2`	`<32-hex-account-id>.r2.cloudflarestorage.com`	path
`gcs`	`storage.googleapis.com`	path

Anything else answers 422 DRAIN_ENDPOINT_NOT_ALLOWED. A deployment can widen the list for its own environment, but a widened host is still required to be https: and to resolve entirely to public addresses, re-checked on every send — the same posture as every other destination.

What lands in the bucket

Object keys

{prefix}/org={orgId}/agent={agentId}/env={environment}/dt=YYYY-MM-DD/hour=HH/{batchId}.ndjson.gz

Hive-style partitioning, because every query engine discovers key=value directories as partition columns with no DDL of its own. Each object is gzipped NDJSON: one line per record, and every line carries schemaUrl, batchId, the batch’s resource block, and one record. Each line’s record is the same shape the event schema describes and the same encoder the generic HTTPS drain uses in ndjson format, so the two cannot drift. A consumer that splits a file and drops lines still has the tenancy of every line it kept. format is always ndjson for a bucket drain; --format json is not honoured here. Dedup on record.id, never on batchId.

dt= and hour= are the observation hour — when Lua wrote the file — not the record’s own event time. A batch whose records straddle an hour boundary has to have one home, and a partition a query engine can prune has to be derivable from the key.Every record keeps its own timestamp and observedTimestamp, so nothing is lost. But a query that reads dt= as the event day will under-count the boundary: filter on the partition to prune, then filter on record.timestamp to be exact.

WHERE dt BETWEEN '2026-09-21' AND '2026-09-23'          -- prunes
  AND record.timestamp >= '2026-09-22T00:00:00Z'         -- is exact
  AND record.timestamp <  '2026-09-23T00:00:00Z'

Batching

Object storage batches on a window, not on a flush tick: 5 minutes or 50 MB, whichever comes first. The 500-record, 1 MiB ceilings that suit an HTTP intake would cut a five-minute window into twenty half-megabyte objects, and no query engine reads a partition of 43,000 tiny files at a sensible cost. A single flush writes at most 25,000 records, so a very busy drain gets a second object in the window rather than one enormous one. At roughly a kilobyte a record that sits well inside 50 MB, so most windows are one file. A drain holding its window looks like backlog, not failure: lua drains status shows a non-zero backlog for up to five minutes at a time, by design. A test or a verify never waits out the window.

The hourly manifest

Every hour partition gets a manifest beside its objects:

{prefix}/org=.../agent=.../env=.../dt=.../hour=HH/_manifest.json

{
  "manifestVersion": 1,
  "schemaUrl": "https://docs.heylua.ai/schemas/logs/1.0",
  "orgId": "org_4f2c9a1b",
  "agentId": "agent_1789214224176_2vta8rnyn",
  "environment": "production",
  "dt": "2026-09-22",
  "hour": "14",
  "objects": [
    {
      "object": "01K5XJ9R7Q2M4V8B0N3C6D1F5H.ndjson.gz",
      "batchId": "01K5XJ9R7Q2M4V8B0N3C6D1F5H",
      "records": 412,
      "bytes": 918273,
      "bytesCompressed": 71422,
      "sha256": "9f2c…",
      "firstRecordId": "1789217102008-a1c4ppzqx",
      "lastRecordId": "1789217994301-k0mzr7tly",
      "writtenAt": "2026-09-22T14:07:33.412Z"
    }
  ],
  "records": 412,
  "bytes": 918273,
  "updatedAt": "2026-09-22T14:07:33.412Z"
}

To reconcile an hour: list the partition, read _manifest.json, and for each row confirm the object exists, its length equals bytesCompressed, and the SHA-256 of the downloaded bytes equals sha256. The hash is over the stored, gzipped object, so you never have to decompress anything to verify it.

aws s3 cp s3://acme-compliance-logs/lua/logs/org=org_4f2c9a1b/agent=agt_7ac1/env=production/dt=2026-09-22/hour=14/_manifest.json - \
  | jq -r '.objects[] | "\(.sha256)  \(.object)"' > hour-14.sha256

aws s3 sync s3://acme-compliance-logs/lua/logs/org=org_4f2c9a1b/agent=agt_7ac1/env=production/dt=2026-09-22/hour=14/ ./hour-14/
(cd hour-14 && sha256sum -c hour-14.sha256)

Four properties worth knowing:

The merge is idempotent on the file name. A re-run that writes the same object replaces its row rather than appending a second one, and the totals are recomputed from the rows.
Entries are sorted by file name, which is a ULID, which is time order.
Past 5,000 entries a manifest keeps the oldest rows and sets "truncated": true. The objects are all still there; only the index is short.
The manifest is best effort. A delivery is never retried because a manifest write failed — a retry would mint a fresh batch id and write the same records to a second file. If you suspect a manifest is short, rebuild it by listing the partition: every column but sha256 is derivable from the objects themselves, and sha256 is derivable from their bytes.

Concurrent manifest writes use conditional puts. A provider that ignores them degrades to last-write-wins, which can lose a manifest row — never an object. Rebuild from the listing if a partition’s manifest and its objects disagree; the objects are the record.

Prove you own the bucket

Writing to a bucket proves somebody gave Lua a key. It does not prove the bucket is yours. So before an object-storage drain delivers anything, it writes a challenge object under your prefix and has to see the token come back.

Start the challenge

lua drains verify drn_2b7f10d9ac4e83615702ffab

Lua mints a one-time token and writes {prefix}/_lua-verify-<token> into the bucket, with the token as its content.

If the credentials can read it back, you are done

Verification finishes on its own and the drain goes to healthy. This is the path when the grant includes s3:GetObject.

Otherwise, echo the token yourself

With the write-only grant we recommend, Lua cannot read the object back. The drain stays in pending_verification, and the delivery row says token_not_echoed. List your prefix, open the object, and echo the token:

aws s3 ls s3://acme-compliance-logs/lua/logs/ | grep _lua-verify-
lua drains confirm drn_2b7f10d9ac4e83615702ffab --token <token>

Output

✅ Ownership confirmed for drn_2b7f10d9ac4e83615702ffab
   verified at 2026-09-22T14:11:07.244Z

Lua stores only a hash of the token and returns it from no route — not the read model, not the delivery row, not the audit trail. Finding it in the bucket is the proof: a control plane that could hand the token back would let anyone holding logs:manage confirm a bucket they cannot even read. The challenge expires with the verification token, after 10 minutes. Run lua drains verify again to mint a new one; verification is limited to 5 attempts per drain per hour. The read model carries a challenge.keyPattern telling you where to look, not what to send. There is no route that returns the token.

When verification fails

Delivery row	What it means	Fix
`s3:AccessDenied`	The grant does not allow `s3:PutObject` on this prefix	Apply the policy below
`s3:NoSuchBucket`	Wrong bucket name, or the right name in another region	Check `--bucket` and `--region`
`s3:InvalidAccessKeyId`	The key pair is wrong or deactivated	Re-enter it with `--credentials-from-env`
`s3:SignatureDoesNotMatch`	The same	The same
`s3:PermanentRedirect`	The bucket lives in a different region	Fix `--region`
`url:allowlist`	The endpoint is not one of the three providers’ hosts	See which hosts are allowed
`token_not_echoed`	The object landed; ownership is not proved yet	`lua drains confirm` — this is the normal path

Errors carry the provider’s own error code, never the SDK’s message. An SDK message quotes the signed request, and a signed request carries a credential.

The bucket policy

The grant is deliberately small: write into one prefix, and nothing else. s3:GetObject is optional — without it the ownership challenge takes the lua drains confirm path and the hourly manifest degrades from read-modify-write to write-only.

A key pair
An assumable role (preferred)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LuaLogDrainWrite",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<your-account-id>:user/lua-log-drain" },
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::acme-compliance-logs/lua/logs/*"
    },
    {
      "Sid": "LuaLogDrainReadBackOptional",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<your-account-id>:user/lua-log-drain" },
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::acme-compliance-logs/lua/logs/*"
    }
  ]
}

Nothing long-lived is stored. The trust policy on your role — sts:ExternalId is the confused-deputy guard, and it is required:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<lua-account-id>:root" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<the id you give Lua>" } }
    }
  ]
}

The permissions policy on the same role is the two statements from the other tab, as Action/Resource without the Principal block.

<lua-account-id> is the AWS account whose principal you trust. It is specific to the deployment your organization is on — ask [email protected] for the value, and pass the same --external-id you put in the condition.

Terraform

resource "aws_s3_bucket" "lua_logs" {
  bucket = "acme-compliance-logs"
}

resource "aws_s3_bucket_public_access_block" "lua_logs" {
  bucket                  = aws_s3_bucket.lua_logs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "lua_logs" {
  bucket = aws_s3_bucket.lua_logs.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_iam_role" "lua_log_drain" {
  name = "LuaLogDrain"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { AWS = "arn:aws:iam::${var.lua_account_id}:root" }
      Action    = "sts:AssumeRole"
      Condition = { StringEquals = { "sts:ExternalId" = var.lua_external_id } }
    }]
  })
}

resource "aws_iam_role_policy" "lua_log_drain" {
  role = aws_iam_role.lua_log_drain.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:PutObject", "s3:GetObject"]
      Resource = "${aws_s3_bucket.lua_logs.arn}/lua/logs/*"
    }]
  })
}

Object Lock is your setting, not Lua’s. Lua writes each data object exactly once and never deletes or overwrites one, so compliance mode is compatible with everything above except the manifest, which is rewritten as each hour fills. Either keep the manifest outside the locked prefix, or accept that a locked hour’s manifest reflects only its first object — the objects themselves are complete either way.

Query it

The Hive layout means a query engine discovers org, agent, env, dt, and hour as partition columns.

Athena
BigQuery
DuckDB

CREATE EXTERNAL TABLE lua_logs (
  schemaUrl string,
  batchId   string,
  resource  map<string,string>,
  record    struct<
    id:                string,
    timestamp:         string,
    observedTimestamp: string,
    eventName:         string,
    severityNumber:    int,
    severityText:      string,
    body:              string,
    attributes:        map<string,string>
  >
)
PARTITIONED BY (org string, agent string, env string, dt string, hour string)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://acme-compliance-logs/lua/logs/';

MSCK REPAIR TABLE lua_logs;

-- tool errors in one day, by tool
SELECT record.attributes['gen_ai.tool.name'] AS tool, count(*) AS errors
FROM lua_logs
WHERE env = 'production' AND dt = '2026-09-22'
  AND record.eventName = 'lua.skill.error'
GROUP BY 1 ORDER BY 2 DESC;

_manifest.json is not NDJSON, so exclude it — either keep manifests in a prefix of their own or add AND "$path" NOT LIKE '%_manifest.json' to the queries that would otherwise read one.A map<string,string> coerces every attribute value to a string, which is what you want for the long tail. Declare the handful you compare numerically (lua.duration_ms, say) as named struct fields instead.

CREATE EXTERNAL TABLE lua_logs
WITH PARTITION COLUMNS
OPTIONS (
  format = 'NEWLINE_DELIMITED_JSON',
  uris = ['gs://acme-logs/lua/logs/*'],
  hive_partition_uri_prefix = 'gs://acme-logs/lua/logs',
  require_hive_partition_filter = true
);

require_hive_partition_filter is worth leaving on: it turns a full-bucket scan into a query error instead of a bill.

SELECT JSON_VALUE(record, '$.attributes."gen_ai.tool.name"') AS tool, count(*) AS errors
FROM lua_logs
WHERE env = 'production' AND dt = '2026-09-22'
  AND JSON_VALUE(record, '$.eventName') = 'lua.skill.error'
GROUP BY 1 ORDER BY 2 DESC;

INSTALL httpfs; LOAD httpfs;

SELECT record.eventName, count(*)
FROM read_ndjson_auto(
  's3://acme-compliance-logs/lua/logs/org=*/agent=*/env=production/dt=2026-09-22/hour=*/*.ndjson.gz',
  hive_partitioning = true
)
GROUP BY 1 ORDER BY 2 DESC;

The glob deliberately does not match _manifest.json.

What a read of the drain shows

An s3 drain reads back with its bucket configuration and no credential:

{
  "id": "drn_7c1e0b44f9a2d5386e10ab93",
  "type": "s3",
  "format": "ndjson",
  "bucket": {
    "provider": "aws",
    "bucket": "acme-compliance-logs",
    "region": "eu-west-1",
    "prefix": "lua/logs",
    "credentials": {
      "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
      "secretLast4": "b31f"
    },
    "challenge": {
      "tokenHash": "",
      "keyPattern": "lua/logs/_lua-verify-*",
      "issuedAt": "2026-09-22T14:11:07.244Z",
      "expiresAt": "2026-09-22T14:21:07.244Z"
    }
  },
  "state": "pending_verification"
}

credentials.secretAccessKey is write-only. It is accepted on POST and on PATCH, and returned by nothing.
credentials.accessKeyId is shown, because SigV4 sends it in clear on every request anyway.
credentials.secretLast4 is the last four characters of the stored secret, so you can tell one credential from another after a rotation without ever reading one.
challenge.keyPattern is where to look, not what to send. tokenHash is always an empty string on a read: a hash would be no use to you and is one thing less to leak.
challenge.method appears once the challenge settles: readback when Lua read the object back itself, confirmed when you echoed the token.

A PATCH replaces the whole bucket block, never merges it. A half-patched destination — a new bucket with the old credentials — is a drain writing somewhere you did not ask for. Omit bucket entirely to keep the current one.

If it isn’t working

422 DRAIN_TYPE_UNAVAILABLE on create

Object-storage drains are not open on your deployment yet. The message names the types that are. Nothing about your bucket or your credentials is wrong.

The drain stays in pending_verification and nothing is wrong

That is the write-only grant working as intended. List your prefix for _lua-verify-* and run lua drains confirm <id> --token <token>. If the object is not there either, the delivery row will say s3:AccessDenied — the grant is missing s3:PutObject on that prefix.

Objects appear but the manifest is short or missing

The manifest is best effort and the objects are the record. Rebuild it by listing the partition. A grant without s3:GetObject cannot do read-modify-write, so an hour with several objects may list only the first.

The backlog sits at a few thousand and clears every five minutes

That is the window, not a fault. Object storage batches on 5 minutes or 50 MB. Watch the age of the backlog against the window rather than its depth.

s3:PermanentRedirect on every attempt

The bucket is in a different region from --region. An AWS endpoint is derived from the bucket and the region, so a wrong region is a wrong host. Recreate the drain with the right one.

Counts do not match at a day boundary

dt= is the hour Lua wrote the file, not the hour the events happened. Read one partition on either side and filter on record.timestamp. See the partition caveat.

Next steps

Event schema

Every field on the record inside each line.

Protecting your destination

Why an arbitrary endpoint is refused, and what is re-checked on every send.

Log export

A one-off NDJSON export, for a window that predates the drain.

lua drains

Every verb and flag, including confirm.

Get started

Concepts

Build

Channels

Integrations

Devices

Ship and operate

Ship logs to your stack

Marketplace

Build with AI

Examples

What to enter

Which hosts are allowed

What lands in the bucket

Object keys

Batching

The hourly manifest

Prove you own the bucket

When verification fails

The bucket policy

Terraform

Query it

What a read of the drain shows

If it isn’t working

Next steps

Event schema

Protecting your destination

Log export

lua drains

​What to enter

​Which hosts are allowed

​What lands in the bucket

​Object keys

​Batching

​The hourly manifest

​Prove you own the bucket

​When verification fails

​The bucket policy

​Terraform

​Query it

​What a read of the drain shows

​If it isn’t working

​Next steps

Event schema

Protecting your destination

Log export

lua drains

What to enter

Which hosts are allowed

What lands in the bucket

Object keys

Batching

The hourly manifest

Prove you own the bucket

When verification fails

The bucket policy

Terraform

Query it

What a read of the drain shows

If it isn’t working

Next steps