Grafana Loki

A loki drain pushes to Grafana Cloud Loki or a Loki you run yourself. Each record becomes one entry in a stream, labelled with five labels and carrying the whole Lua record as its JSON line — so | json reaches every field at query time without a label for it.

Vendor destinations are opened per deployment. Where loki is not open yet, a create answers 422 DRAIN_TYPE_UNAVAILABLE and the message names the types that are. It may not be enabled on your deployment yet.

Before you begin

Your Loki host. On Grafana Cloud it is the Loki data source’s URL, https://logs-prod-012.grafana.net.
Credentials: on Grafana Cloud, your instance ID and an access policy token, used as HTTP Basic. A self-hosted Loki behind a gateway may take a bearer token instead.
For a multi-tenant Loki: your tenant id, for X-Scope-OrgID.

What to enter

Field	Flag	Value
Name	`--name`	1–64 characters, unique in the organization
Type	`--type loki`
Endpoint	`--endpoint`	Your Loki host — `https://logs-prod-012.grafana.net`
Auth	`--header Authorization`	The complete header value: `Basic <base64(instanceId:token)>` or `Bearer <token>`
Tenant	`--header X-Scope-OrgID`	Multi-tenant Loki only

lua drains create \
  --name "Loki prod" \
  --type loki \
  --endpoint https://logs-prod-012.grafana.net \
  --header Authorization \
  --environments production \
  --min-severity info

In CI:

lua drains create --ci --json \
  --name "Loki prod" --type loki \
  --endpoint https://logs-prod-012.grafana.net \
  --header-from-env Authorization=LOKI_AUTH

Store the complete header value, scheme included — Basic dXNlcjpwYXNz… or Bearer glc_…. The stored value is sent verbatim; Lua never prefixes a scheme onto it. For Grafana Cloud, the Basic credential is <instance id>:<access policy token>, base64-encoded.

/loki/api/v1/push is composed onto whatever host you give, and composition is idempotent — paste the bare host or the full push URL, whichever your Grafana UI showed you. A non-default port is kept, which a self-hosted Loki usually needs; a query string or fragment is dropped. There is no host allow-list for this type: Grafana Cloud and a self-hosted stack are equally your host. It is still validated as https: and as resolving entirely to public addresses, re-checked on every send.

X-Scope-OrgID is a per-customer value, so it rides as one of the drain’s own headers rather than as something the preset sets. Add it with --header X-Scope-OrgID (the CLI prompts for the value) only if your Loki is multi-tenant. Grafana Cloud does not want it — the Basic credential already carries the tenant.

How ownership verification works

A vendor preset is reachability-checked, not ownership-verified. lua drains verify pushes one batch holding a single lua.drain.test record, and any 2xx from Loki is accepted. It proves the host answers and the credential works. It does not prove you own that host — no Loki push endpoint can echo a challenge token back. Only a generic HTTPS drain is ownership-verified by a token echo, and only an object-storage drain proves ownership by writing into a bucket.

lua drains verify drn_5a72c0e8194bd3f67210cc42

Loki answers a successful push with 204, which is a 2xx and verifies fine. A 401 means the Basic credential is wrong or is missing its scheme; a 403 on Grafana Cloud usually means the access policy lacks logs:write. Verification is limited to 5 attempts per drain per hour. Loki deliveries are not signed.

What arrives

{
  "streams": [
    {
      "stream": {
        "org": "org_4f2c9a1b",
        "agent": "agent_1789214224176_2vta8rnyn",
        "environment": "production",
        "source": "skill",
        "level": "error"
      },
      "values": [
        ["1789217997644000000", "{\"id\":\"1789217997644-jtoulxxxq\",\"eventName\":\"lua.skill.error\",\"severityText\":\"ERROR\",\"body\":\"Ticket lookup failed: upstream timeout\",\"attributes\":{\"lua.source\":\"skill\",\"gen_ai.tool.name\":\"lookup_tickets\"}}"]
      ]
    }
  ]
}

Records are grouped into streams by label set, and entries within a stream are sorted ascending by timestamp: an older Loki refuses an out-of-order push outright, and a modern one accepts it only inside its unordered_writes window.

Five labels, and why there are only five

Label	From
`org`	`resource["lua.org.id"]`
`agent`	`resource["lua.agent.id"]`
`environment`	`resource["deployment.environment.name"]`
`source`	`attributes["lua.source"]` — omitted when the record has none, never sent as `undefined`
`level`	`severityText`, lower-cased

Loki indexes label sets: every distinct combination is a stream with its own index entry and its own chunks. A label whose values are not a small closed enum is a cost bug in your Loki bill, not in ours — which is why eventName, tool names, and primitive names are deliberately not labels. They ride inside the JSON line, where | json reaches them at query time for free. Loki’s own ceiling is 15 labels. Five leaves room for whatever your own pipeline adds.

Useful queries

{org="org_4f2c9a1b", environment="production", level="error"}

{org="org_4f2c9a1b", source="skill"} | json | eventName = "lua.skill.error"

{org="org_4f2c9a1b"} | json | attributes_gen_ai_tool_name = "lookup_tickets"

sum by (agent) (rate({org="org_4f2c9a1b", level="error"}[5m]))

| json flattens nested keys with _, so attributes["gen_ai.tool.name"] is reachable as attributes_gen_ai_tool_name.

Caps

Cap	Value
Records per push	500
Uncompressed bytes per push	768 KiB (Loki’s default request limit is 1 MiB)
Bytes per line	256 KiB — Loki’s own `max_line_size`
Compression	`gzip`, always
Retried	`408`, `429`, and every `5xx`
Terminal	Every `3xx`, and every other `4xx`

A record over 256 KiB has its body cut and suffixed ... [truncated by lua] rather than being dropped.

Two terminal 400s worth recognising

Every non-429 4xx is terminal — the batch is dropped and counted under rejected, not retried, because the same bytes would be refused again.

line_too_long — an entry exceeded your Loki’s max_line_size. The 256 KiB per-record cap exists to make this unreachable, so if you see it, your Loki’s limit is set below the default. Raise max_line_size, or narrow the drain’s sources so the long records stop being produced.
entry too far behind / too_far_behind — the record is older than your reject_old_samples_max_age. Lua only retries a batch for 6 hours, so this normally only bites a drain that was paused for longer and then resumed with a deep backlog. Raise reject_old_samples_max_age, or accept the loss of the oldest range.

Test the credential with curl

LOKI='https://logs-prod-012.grafana.net'
LOKI_AUTH='Basic <base64 of instanceId:token>'
NOW_NS=$(( $(date +%s) * 1000000000 ))

curl -sS -i -X POST "$LOKI/loki/api/v1/push" \
  -H "Authorization: $LOKI_AUTH" \
  -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"org\":\"org_4f2c9a1b\",\"agent\":\"agt_probe\",\"environment\":\"production\",\"source\":\"drain\",\"level\":\"info\"},\"values\":[[\"$NOW_NS\",\"{\\\"eventName\\\":\\\"lua.drain.test\\\"}\"]]}]}"

204 with an empty body means accepted. Then query it back:

curl -sS -G "$LOKI/loki/api/v1/query_range" \
  -H "Authorization: $LOKI_AUTH" \
  --data-urlencode 'query={org="org_4f2c9a1b"}' | jq .

Then do the same through the real delivery path:

lua drains test drn_5a72c0e8194bd3f67210cc42

If it isn’t working

401 on every push

The stored Authorization value is incomplete. It must be the whole header value — Basic <base64> or Bearer <token> — with the scheme. Re-enter it with lua drains update <id> --header Authorization.

400 with no tenant provided

Your Loki is multi-tenant and wants X-Scope-OrgID. Add it: lua drains update <id> --header X-Scope-OrgID.

Queries return nothing although the drain is healthy

healthy means Loki accepted the push. Check the label selector first — every stream carries org, agent, and environment, so {org="…"} alone should match. Then check the time range: Loki files entries at the record’s own timestamp, not at ingest.

Our Loki bill jumped

Not from these labels — five is deliberately few. Check whether a pipeline of yours is promoting fields out of the JSON line into labels. eventName and tool names have high cardinality and belong in the line.

Next steps

Grafana dashboard pack

An importable dashboard built on these fields.

Event schema

Every field inside the JSON line.

Protecting your destination

What verification does and does not prove.

lua drains

Every verb and flag.

Get started

Concepts

Build

Channels

Integrations

Devices

Ship and operate

Ship logs to your stack

Marketplace

Build with AI

Examples

What to enter

How ownership verification works

What arrives

Five labels, and why there are only five

Useful queries

Caps

Two terminal 400s worth recognising

Test the credential with curl

If it isn’t working

Next steps

Grafana dashboard pack

Event schema

Protecting your destination

lua drains

​What to enter

​How ownership verification works

​What arrives

​Five labels, and why there are only five

​Useful queries

​Caps

​Two terminal 400s worth recognising

​Test the credential with curl

​If it isn’t working

​Next steps

Grafana dashboard pack

Event schema

Protecting your destination

lua drains

What to enter

How ownership verification works

What arrives

Five labels, and why there are only five

Useful queries

Caps

Two terminal 400s worth recognising

Test the credential with curl

If it isn’t working

Next steps