> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heylua.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datadog monitor pack

> Four importable Datadog monitors over a Lua log drain: heartbeat absence, error-rate spikes, dropped records, and truncated executions

Four monitors that turn a working [Datadog drain](/drains/datadog) into pages. They are built on the field names in the [event schema](/drains/event-schema), so they keep working as new sources are added.

**Before you begin**

* A `datadog` drain in state `healthy`. Confirm with `lua drains status`.
* A Datadog API key **and** an application key, if you import over the API.
* Three log facets created once, from any record: `@lua.eventName`, `@lua.attributes.lua.source`, and `@lua.attributes.gen_ai.tool.name`. Open one Lua record in Log Explorer, expand `lua`, and choose *Create facet* on each. Grouping and alerting need them; plain search does not.

## What you get

| Monitor                         | Fires when                                                                         | Why it matters                                                                                                                                                       |
| ------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Log drain heartbeat missing** | Fewer than 1 `lua.drain.heartbeat` in 15 minutes                                   | A healthy drain heartbeats every 5 minutes. Without this monitor, a dead pipeline looks exactly like a quiet one, and every other monitor here silently stops firing |
| **Agent error rate spiking**    | More than 5 records matching `lua.*.error` in 5 minutes, per agent and environment | The core signal. Warns at 3                                                                                                                                          |
| **Log drain dropped records**   | Any `lua.drain.dropped` in 15 minutes                                              | Data was lost between Lua and Datadog. The record bounds the gap so you can reconcile                                                                                |
| **Execution output truncated**  | Any `lua.truncated.warn` in an hour, per agent and environment                     | The runtime dropped lines from an execution — usually a primitive logging in a loop. Those lines are gone from the platform's copy too                               |

Install the heartbeat monitor first. It is the one that makes the other three trustworthy: they all alert on the *presence* of something, so their silence is only meaningful while the pipeline is known to be up.

## Import it

<Tabs>
  <Tab title="Datadog UI">
    One monitor at a time: **Monitors → New Monitor → Import Monitor from JSON**, paste a single object from the `monitors` array, then **Save**. Repeat for the other three.

    Add your notification handles to each monitor's message before saving — the pack ships with none, so an imported monitor alerts in the UI and pages nobody.
  </Tab>

  <Tab title="API">
    Save the pack as `lua-datadog-monitors.json`, then:

    ```bash theme={null}
    DD_SITE=datadoghq.eu
    DD_API_KEY='<your API key>'
    DD_APP_KEY='<your application key>'

    jq -c '.monitors[]' lua-datadog-monitors.json | while read -r monitor; do
      curl -sS -X POST "https://api.$DD_SITE/api/v1/monitor" \
        -H "DD-API-KEY: $DD_API_KEY" \
        -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
        -H 'Content-Type: application/json' \
        -d "$monitor" | jq -r '"created \(.id)\t\(.name)"'
    done
    ```

    Every monitor is tagged `lua:pack`, so you can find them again, update them, or remove the lot:

    ```bash theme={null}
    curl -sS "https://api.$DD_SITE/api/v1/monitor?monitor_tags=lua:pack" \
      -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
      | jq -r '.[] | "\(.id)\t\(.name)"'
    ```
  </Tab>
</Tabs>

## The pack

```json lua-datadog-monitors.json theme={null}
{
  "monitors": [
    {
      "name": "[Lua] Log drain heartbeat missing",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.drain.heartbeat\").index(\"*\").rollup(\"count\").last(\"15m\") < 1",
      "message": "{{#is_alert}}\nNo `lua.drain.heartbeat` record has arrived for 15 minutes.\n\nA healthy Lua log drain heartbeats every 5 minutes, so three in a row are missing. Either the drain has left the `healthy` state, or delivery into Datadog has stopped. Every other Lua monitor is blind until this clears.\n\nCheck: `lua drains status` and `lua drains deliveries <id> --limit 20`.\nRunbook: https://docs.heylua.ai/drains/delivery-guarantees#heartbeat\n{{/is_alert}}\n\n{{#is_recovery}}\nHeartbeats are arriving again.\n{{/is_recovery}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 2,
      "options": {
        "thresholds": { "critical": 1 },
        "enable_logs_sample": false,
        "notify_no_data": true,
        "no_data_timeframe": 20,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 60,
        "new_group_delay": 60
      }
    },
    {
      "name": "[Lua] Agent error rate spiking",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.*.error\").index(\"*\").rollup(\"count\").by(\"service,env\").last(\"5m\") > 5",
      "message": "{{#is_alert}}\nMore than 5 error records in 5 minutes on {{service.name}} ({{env.name}}).\n\nEvery Lua record whose `eventName` ends in `.error` is an error: a thrown tool, a failing MCP call, a runtime error, or a failed execution. Open the event samples to see which source dominates.\n\nCheck: `lua logs --type agent_error --since 15m` and `lua logs --type skill --since 15m`.\nRunbook: https://docs.heylua.ai/ship/logs-and-debugging\n{{/is_alert}}\n\n{{#is_warning}}\nMore than 3 error records in 5 minutes on {{service.name}} ({{env.name}}).\n{{/is_warning}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 2,
      "options": {
        "thresholds": { "critical": 5, "warning": 3 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 60,
        "new_group_delay": 60,
        "group_retention_duration": "2d"
      }
    },
    {
      "name": "[Lua] Log drain dropped records",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.drain.dropped\").index(\"*\").rollup(\"count\").last(\"15m\") > 0",
      "message": "{{#is_alert}}\nA Lua log drain dropped records and has just recovered.\n\nThe drain emits one `lua.drain.dropped` record on its first success after a drop, carrying `@lua.attributes.lua.drain.dropped_count` and the `lua.drain.dropped_from` / `lua.drain.dropped_to` record ids that bound the gap. Anything inside that range never reached Datadog.\n\nThe platform's own copy is unaffected: reconcile the gap with `lua logs --since <from> --until <to>`.\n\nCheck: `lua drains status <id>` for the backlog, and raise `--min-severity` or narrow `--sources` if the drain cannot keep up.\nRunbook: https://docs.heylua.ai/drains/delivery-guarantees#backpressure-and-drops\n{{/is_alert}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 1,
      "options": {
        "thresholds": { "critical": 0 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 0,
        "new_group_delay": 60
      }
    },
    {
      "name": "[Lua] Execution output truncated",
      "type": "log alert",
      "query": "logs(\"source:lua @lua.eventName:lua.truncated.warn\").index(\"*\").rollup(\"count\").by(\"service,env\").last(\"1h\") > 0",
      "message": "{{#is_alert}}\nThe runtime dropped log lines from an execution on {{service.name}} ({{env.name}}).\n\n`lua.truncated.warn` carries `@lua.attributes.lua.truncated.lines` — how many lines were lost — and `@lua.attributes.lua.execution.id` to find the execution. The lines are gone from the platform's copy too, so they cannot be recovered by re-reading `lua logs`.\n\nThis is almost always a primitive logging inside a loop. Find it with `@lua.attributes.lua.execution.id` and cut the logging there.\nRunbook: https://docs.heylua.ai/drains/event-schema#eventname\n{{/is_alert}}",
      "tags": ["source:lua", "lua:log-drain", "lua:pack"],
      "priority": 3,
      "options": {
        "thresholds": { "critical": 0 },
        "enable_logs_sample": true,
        "notify_no_data": false,
        "notify_audit": false,
        "include_tags": true,
        "renotify_interval": 0,
        "new_group_delay": 60
      }
    }
  ]
}
```

## Tune it

* **Notification handles.** Append `@slack-your-channel`, `@pagerduty-your-service`, or `@your-team@example.com` to each `message`. Nothing in the pack notifies anyone as shipped.
* **Thresholds.** "More than 5 errors in 5 minutes" is a starting point, not a measurement of your traffic. Run the query in Log Explorer over the last week and set the threshold above your normal peak.
* **Grouping.** The error and truncation monitors group `by("service,env")`, where `service` is the agent's name and `env` comes from the drain's `env:` tag — so production and sandbox alert separately, per agent. Group by `agent` instead to key on the agent id.
* **Selecting errors.** `@lua.eventName:lua.*.error` matches every source's error records. `@lua.severityText:ERROR` selects exactly the same set and needs no wildcard, if you would rather not rely on one.
* **Multi-alert noise.** `new_group_delay: 60` stops a newly created agent alerting on its first minute. Raise it if you create agents in bursts.
* **The heartbeat monitor alerts twice on purpose.** It carries both a `< 1` threshold and `notify_no_data`, so it fires whether your index reports a zero count or reports nothing at all. For the one alert every other alert depends on, a duplicate page is better than a missed one. Drop `notify_no_data` once you have seen it fire correctly in your own organization.

## Why these queries

The Datadog encoder puts the **whole record** under a `lua` key, so every schema field is addressable as an attribute: `@lua.eventName`, `@lua.severityText`, `@lua.attributes.<name>`. That makes `eventName` the natural selector here.

The [Grafana pack](/drains/packs/grafana-dashboard) selects differently — on attributes such as `lua_drain_backlog` rather than on the event name — because Loki's OTLP ingestion indexes attributes rather than the OTLP `eventName` field. The two packs watch the same four signals; only the way they address them differs.

## Next steps

<Columns cols={2}>
  <Card title="Datadog" href="/drains/datadog">Create the drain these monitors watch.</Card>
  <Card title="Delivery guarantees" href="/drains/delivery-guarantees">What the heartbeat and the drop record mean.</Card>
  <Card title="Event schema" href="/drains/event-schema">Every field these queries address.</Card>
  <Card title="Grafana dashboard" href="/drains/packs/grafana-dashboard">The same signals, over Loki.</Card>
</Columns>