Datadog monitor pack

Four monitors that turn a working Datadog drain into pages. They are built on the field names in the event schema, so they keep working as new sources are added. Before you begin

A datadog drain in state healthy. Confirm with lua drains status.
A Datadog API key and an application key, if you import over the API.
Three log facets created once, from any record: @lua.eventName, @lua.attributes.lua.source, and @lua.attributes.gen_ai.tool.name. Open one Lua record in Log Explorer, expand lua, and choose Create facet on each. Grouping and alerting need them; plain search does not.

What you get

Monitor	Fires when	Why it matters
Log drain heartbeat missing	Fewer than 1 `lua.drain.heartbeat` in 15 minutes	A healthy drain heartbeats every 5 minutes. Without this monitor, a dead pipeline looks exactly like a quiet one, and every other monitor here silently stops firing
Agent error rate spiking	More than 5 records matching `lua.*.error` in 5 minutes, per agent and environment	The core signal. Warns at 3
Log drain dropped records	Any `lua.drain.dropped` in 15 minutes	Data was lost between Lua and Datadog. The record bounds the gap so you can reconcile
Execution output truncated	Any `lua.truncated.warn` in an hour, per agent and environment	The runtime dropped lines from an execution — usually a primitive logging in a loop. Those lines are gone from the platform’s copy too

Install the heartbeat monitor first. It is the one that makes the other three trustworthy: they all alert on the presence of something, so their silence is only meaningful while the pipeline is known to be up.

Import it

Datadog UI
API

One monitor at a time: Monitors → New Monitor → Import Monitor from JSON, paste a single object from the monitors array, then Save. Repeat for the other three.Add your notification handles to each monitor’s message before saving — the pack ships with none, so an imported monitor alerts in the UI and pages nobody.

Save the pack as lua-datadog-monitors.json, then:

DD_SITE=datadoghq.eu
DD_API_KEY='<your API key>'
DD_APP_KEY='<your application key>'

jq -c '.monitors[]' lua-datadog-monitors.json | while read -r monitor; do
  curl -sS -X POST "https://api.$DD_SITE/api/v1/monitor" \
    -H "DD-API-KEY: $DD_API_KEY" \
    -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
    -H 'Content-Type: application/json' \
    -d "$monitor" | jq -r '"created \(.id)\t\(.name)"'
done

Every monitor is tagged lua:pack, so you can find them again, update them, or remove the lot:

curl -sS "https://api.$DD_SITE/api/v1/monitor?monitor_tags=lua:pack" \
  -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  | jq -r '.[] | "\(.id)\t\(.name)"'

The pack

lua-datadog-monitors.json

{
  "monitors": [
    {
      "name": "[Lua] Log drain heartbeat missing",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.drain.heartbeat\").index(\"*\").rollup(\"count\").last(\"15m\") < 1",
      "message": "{{#is_alert}}\nNo `lua.drain.heartbeat` record has arrived for 15 minutes.\n\nA healthy Lua log drain heartbeats every 5 minutes, so three in a row are missing. Either the drain has left the `healthy` state, or delivery into Datadog has stopped. Every other Lua monitor is blind until this clears.\n\nCheck: `lua drains status` and `lua drains deliveries <id> --limit 20`.\nRunbook: https://docs.heylua.ai/drains/delivery-guarantees#heartbeat\n{{/is_alert}}\n\n{{#is_recovery}}\nHeartbeats are arriving again.\n{{/is_recovery}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 2,
      "options": {
        "thresholds": { "critical": 1 },
        "enable_logs_sample": false,
        "notify_no_data": true,
        "no_data_timeframe": 20,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 60,
        "new_group_delay": 60
      }
    },
    {
      "name": "[Lua] Agent error rate spiking",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.*.error\").index(\"*\").rollup(\"count\").by(\"service,env\").last(\"5m\") > 5",
      "message": "{{#is_alert}}\nMore than 5 error records in 5 minutes on {{service.name}} ({{env.name}}).\n\nEvery Lua record whose `eventName` ends in `.error` is an error: a thrown tool, a failing MCP call, a runtime error, or a failed execution. Open the event samples to see which source dominates.\n\nCheck: `lua logs --type agent_error --since 15m` and `lua logs --type skill --since 15m`.\nRunbook: https://docs.heylua.ai/ship/logs-and-debugging\n{{/is_alert}}\n\n{{#is_warning}}\nMore than 3 error records in 5 minutes on {{service.name}} ({{env.name}}).\n{{/is_warning}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 2,
      "options": {
        "thresholds": { "critical": 5, "warning": 3 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 60,
        "new_group_delay": 60,
        "group_retention_duration": "2d"
      }
    },
    {
      "name": "[Lua] Log drain dropped records",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.drain.dropped\").index(\"*\").rollup(\"count\").last(\"15m\") > 0",
      "message": "{{#is_alert}}\nA Lua log drain dropped records and has just recovered.\n\nThe drain emits one `lua.drain.dropped` record on its first success after a drop, carrying `@lua.attributes.lua.drain.dropped_count` and the `lua.drain.dropped_from` / `lua.drain.dropped_to` record ids that bound the gap. Anything inside that range never reached Datadog.\n\nThe platform's own copy is unaffected: reconcile the gap with `lua logs --since <from> --until <to>`.\n\nCheck: `lua drains status <id>` for the backlog, and raise `--min-severity` or narrow `--sources` if the drain cannot keep up.\nRunbook: https://docs.heylua.ai/drains/delivery-guarantees#backpressure-and-drops\n{{/is_alert}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 1,
      "options": {
        "thresholds": { "critical": 0 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 0,
        "new_group_delay": 60
      }
    },
    {
      "name": "[Lua] Execution output truncated",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.truncated.warn\").index(\"*\").rollup(\"count\").by(\"service,env\").last(\"1h\") > 0",
      "message": "{{#is_alert}}\nThe runtime dropped log lines from an execution on {{service.name}} ({{env.name}}).\n\n`lua.truncated.warn` carries `@lua.attributes.lua.truncated.lines` — how many lines were lost — and `@lua.attributes.lua.execution.id` to find the execution. The lines are gone from the platform's copy too, so they cannot be recovered by re-reading `lua logs`.\n\nThis is almost always a primitive logging inside a loop. Find it with `@lua.attributes.lua.execution.id` and cut the logging there.\nRunbook: https://docs.heylua.ai/drains/event-schema#eventname\n{{/is_alert}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 3,
      "options": {
        "thresholds": { "critical": 0 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 0,
        "new_group_delay": 60
      }
    }
  ]
}

Tune it

Notification handles. Append @slack-your-channel, @pagerduty-your-service, or @[email protected] to each message. Nothing in the pack notifies anyone as shipped.
Thresholds. “More than 5 errors in 5 minutes” is a starting point, not a measurement of your traffic. Run the query in Log Explorer over the last week and set the threshold above your normal peak.
Grouping. The error and truncation monitors group by("service,env"), where service is the agent’s name and env comes from the drain’s env: tag — so production and sandbox alert separately, per agent. Group by agent instead to key on the agent id.
Selecting errors. @lua.eventName:lua.*.error matches every source’s error records. @lua.severityText:ERROR selects exactly the same set and needs no wildcard, if you would rather not rely on one.
Multi-alert noise. new_group_delay: 60 stops a newly created agent alerting on its first minute. Raise it if you create agents in bursts.
The heartbeat monitor alerts twice on purpose. It carries both a < 1 threshold and notify_no_data, so it fires whether your index reports a zero count or reports nothing at all. For the one alert every other alert depends on, a duplicate page is better than a missed one. Drop notify_no_data once you have seen it fire correctly in your own organization.

Why these queries

The Datadog encoder puts the whole record under a lua key, so every schema field is addressable as an attribute: @lua.eventName, @lua.severityText, @lua.attributes.<name>. That makes eventName the natural selector here. The Grafana pack selects differently — on attributes such as lua_drain_backlog rather than on the event name — because Loki’s OTLP ingestion indexes attributes rather than the OTLP eventName field. The two packs watch the same four signals; only the way they address them differs.

Next steps

Datadog

Create the drain these monitors watch.

Delivery guarantees

What the heartbeat and the drop record mean.

Event schema

Every field these queries address.

Grafana dashboard

The same signals, over Loki.

Get started

Concepts

Build

Channels

Integrations

Devices

Ship and operate

Ship logs to your stack

Marketplace

Build with AI

Examples

Datadog monitor pack

What you get

Import it

The pack

Tune it

Why these queries

Next steps

Datadog

Delivery guarantees

Event schema

Grafana dashboard

​What you get

​Import it

​The pack

​Tune it

​Why these queries

​Next steps

Datadog

Delivery guarantees

Event schema

Grafana dashboard

What you get

Import it

The pack

Tune it

Why these queries

Next steps