> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heylua.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Grafana dashboard
> An importable Grafana dashboard over a Loki fed by the OpenTelemetry Collector: events by severity, errors by agent, tool-call errors, and the drain heartbeat
A four-panel Grafana dashboard over the logs an [OTLP drain](/drains/opentelemetry) delivers, through the [collector config](/drains/packs/otel-collector), into Loki.
**Before you begin**
* An `otlp` drain in state `healthy`, delivering into a collector.
* A Loki datasource in Grafana, receiving from that collector.
* Grafana 10 or later.
The queries assume Loki 3.x with OTLP ingestion, where `service.name` becomes the `service_name` stream label and the remaining resource and record attributes become structured metadata with every non-alphanumeric character replaced by `_` — so `severity_text`, `gen_ai_tool_name`, `lua_drain_backlog`. If your pipeline promotes a different set of attributes to labels, change the stream selector in each panel; the filter stages after it stay as they are.
## What you get
| Panel | Query shape | What it shows |
| ------------------------------ | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Events by severity** | `sum by (severity_text)` | Stacked volume by `DEBUG` / `INFO` / `WARN` / `ERROR`. The shape of the day, and where a spike sits |
| **Errors by agent** | `sum by (service_name)`, `severity_text = "ERROR"` | Which agent is failing. `service_name` is the agent's name |
| **Tool-call errors** | `sum by (gen_ai_tool_name)`, `severity_text = "ERROR"` | Which tool is failing, across skills and MCP servers together |
| **Drain heartbeat (last 15m)** | `sum(count_over_time(… lua_drain_backlog …))` | Heartbeats received. A healthy drain sends one every 5 minutes, so 3 is normal and `0` reads `DOWN` in red |
An `$agent` template variable over `service_name` scopes all four panels to one agent or to all of them.
## Import it
Save the JSON below as `lua-grafana-dashboard.json`.
In Grafana, **Dashboards → New → Import**, upload the file, pick your Loki datasource for the `DS_LOKI` input, and **Import**.
The dashboard's UID is `lua-log-drain`, so re-importing updates it in place rather than creating a second copy.
It should read `3` within fifteen minutes of a healthy drain. `DOWN` in red means either no heartbeat reached Loki or the attribute name differs in your pipeline — run the panel's query in Explore and confirm `lua_drain_backlog` is the structured-metadata name your Loki produced.
## The dashboard
```json lua-grafana-dashboard.json theme={null}
{
"__inputs": [
{
"name": "DS_LOKI",
"label": "Loki",
"description": "The Loki datasource fed by the OpenTelemetry Collector",
"type": "datasource",
"pluginId": "loki",
"pluginName": "Loki"
}
],
"__requires": [
{ "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
{ "type": "datasource", "id": "loki", "name": "Loki", "version": "1.0.0" },
{ "type": "panel", "id": "timeseries", "name": "Time series", "version": "" },
{ "type": "panel", "id": "stat", "name": "Stat", "version": "" }
],
"annotations": { "list": [] },
"description": "Agent logs shipped from Lua by an OTLP log drain, through an OpenTelemetry Collector into Loki.",
"editable": true,
"graphTooltip": 1,
"id": null,
"links": [
{
"title": "Ship logs to your stack",
"type": "link",
"url": "https://docs.heylua.ai/drains/overview",
"targetBlank": true,
"icon": "external link",
"tags": []
}
],
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "Events by severity",
"description": "Every record the drain delivered, split by severityText.",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
"fieldConfig": {
"defaults": {
"custom": { "drawStyle": "bars", "fillOpacity": 60, "stacking": { "mode": "normal" }, "lineWidth": 0 },
"unit": "short",
"min": 0
},
"overrides": [
{ "matcher": { "id": "byName", "options": "ERROR" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }] },
{ "matcher": { "id": "byName", "options": "WARN" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } }] },
{ "matcher": { "id": "byName", "options": "INFO" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "blue" } }] },
{ "matcher": { "id": "byName", "options": "DEBUG" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "text" } }] }
]
},
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"queryType": "range",
"editorMode": "code",
"expr": "sum by (severity_text) (count_over_time({service_name=~\"$agent\"} | severity_text != `` [$__auto]))",
"legendFormat": "{{severity_text}}"
}
]
},
{
"id": 2,
"type": "timeseries",
"title": "Errors by agent",
"description": "ERROR records per agent. service_name is the Lua agent's name.",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 0 },
"fieldConfig": {
"defaults": {
"custom": { "drawStyle": "bars", "fillOpacity": 70, "lineWidth": 0 },
"unit": "short",
"min": 0
},
"overrides": []
},
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"queryType": "range",
"editorMode": "code",
"expr": "sum by (service_name) (count_over_time({service_name=~\"$agent\"} | severity_text = `ERROR` [$__auto]))",
"legendFormat": "{{service_name}}"
}
]
},
{
"id": 3,
"type": "timeseries",
"title": "Tool-call errors",
"description": "Failing tool calls, by the tool the model called. Covers both skills and MCP servers.",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"gridPos": { "h": 9, "w": 16, "x": 0, "y": 9 },
"fieldConfig": {
"defaults": {
"custom": { "drawStyle": "bars", "fillOpacity": 70, "stacking": { "mode": "normal" }, "lineWidth": 0 },
"unit": "short",
"min": 0
},
"overrides": []
},
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"queryType": "range",
"editorMode": "code",
"expr": "sum by (gen_ai_tool_name) (count_over_time({service_name=~\"$agent\"} | severity_text = `ERROR` | gen_ai_tool_name != `` [$__auto]))",
"legendFormat": "{{gen_ai_tool_name}}"
}
]
},
{
"id": 4,
"type": "stat",
"title": "Drain heartbeat (last 15m)",
"description": "Heartbeats received in the last 15 minutes. A healthy drain sends one every 5 minutes, so 3 is normal and 0 means the pipeline is down.",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"gridPos": { "h": 9, "w": 8, "x": 16, "y": 9 },
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"mappings": [{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red", "index": 0 } } }],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "orange", "value": 1 },
{ "color": "green", "value": 2 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "background",
"graphMode": "area",
"textMode": "auto",
"justifyMode": "auto"
},
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"queryType": "instant",
"editorMode": "code",
"expr": "sum(count_over_time({service_name=~\"$agent\"} | lua_drain_backlog != `` [15m]))",
"legendFormat": "heartbeats"
}
]
}
],
"refresh": "1m",
"schemaVersion": 39,
"tags": ["lua", "log-drain", "agents"],
"templating": {
"list": [
{
"name": "agent",
"label": "Agent",
"type": "query",
"datasource": { "type": "loki", "uid": "${DS_LOKI}" },
"query": { "label": "service_name", "refId": "LokiVariableQueryEditor-VariableQuery", "type": 1 },
"refresh": 2,
"includeAll": true,
"allValue": ".+",
"multi": true,
"current": { "selected": true, "text": ["All"], "value": ["$__all"] },
"sort": 1
}
]
},
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Lua agents — logs",
"uid": "lua-log-drain",
"version": 1,
"weekStart": ""
}
```
## Why these selectors
Each panel keys on a field that is unique to what it measures, which avoids depending on the OTLP `eventName` field reaching Loki as a queryable label:
* `severity_text` distinguishes the four severities directly.
* `gen_ai_tool_name` is present only on tool calls, so `severity_text = "ERROR"` plus that attribute is exactly "a tool call failed", across skills and MCP servers.
* `lua_drain_backlog` is present only on the heartbeat record. Counting records that carry it *is* counting heartbeats.
The same logic gives you the other two synthetic records if you want panels for them: `lua_drain_dropped_count` appears only on a drop record, and `lua_truncated_lines` only on a truncation record.
```logql theme={null}
# Records dropped between Lua and your backend
sum(count_over_time({service_name=~"$agent"} | lua_drain_dropped_count != `` [$__auto]))
# Executions whose output the runtime truncated
sum by (service_name) (count_over_time({service_name=~"$agent"} | lua_truncated_lines != `` [$__auto]))
```
The [Datadog pack](/drains/packs/datadog-monitors) selects on `@lua.eventName` instead, because the Datadog encoder ships the whole record as a nested object and the event name is directly addressable there. The two packs watch the same four signals from opposite ends of the schema.
## Alert on it
Grafana alert rules over the same queries, if Grafana rather than the backend owns your alerting:
```logql Drain heartbeat missing — alert when IS BELOW 1 for 5m theme={null}
sum(count_over_time({service_name=~".+"} | lua_drain_backlog != "" [15m]))
```
```logql Errors spiking — alert when IS ABOVE 5 for 5m theme={null}
sum by (service_name) (count_over_time({service_name=~".+"} | severity_text = "ERROR" [5m]))
```
```logql Records dropped — alert when IS ABOVE 0 theme={null}
sum(count_over_time({service_name=~".+"} | lua_drain_dropped_count != "" [15m]))
```
Set the heartbeat rule's **No data** handling to `Alerting`: with no heartbeats there is no series at all, and the default of `No Data` would leave it silent exactly when it matters.
## Next steps
The pipeline that feeds this dashboard.
Create the drain.
Where these attribute names come from.
The same signals, in Datadog.