Grafana dashboard

A four-panel Grafana dashboard over the logs an OTLP drain delivers, through the collector config, into Loki. Before you begin

An otlp drain in state healthy, delivering into a collector.
A Loki datasource in Grafana, receiving from that collector.
Grafana 10 or later.

The queries assume Loki 3.x with OTLP ingestion, where service.name becomes the service_name stream label and the remaining resource and record attributes become structured metadata with every non-alphanumeric character replaced by _ — so severity_text, gen_ai_tool_name, lua_drain_backlog. If your pipeline promotes a different set of attributes to labels, change the stream selector in each panel; the filter stages after it stay as they are.

What you get

Panel	Query shape	What it shows
Events by severity	`sum by (severity_text)`	Stacked volume by `DEBUG` / `INFO` / `WARN` / `ERROR`. The shape of the day, and where a spike sits
Errors by agent	`sum by (service_name)`, `severity_text = "ERROR"`	Which agent is failing. `service_name` is the agent’s name
Tool-call errors	`sum by (gen_ai_tool_name)`, `severity_text = "ERROR"`	Which tool is failing, across skills and MCP servers together
Drain heartbeat (last 15m)	`sum(count_over_time(… lua_drain_backlog …))`	Heartbeats received. A healthy drain sends one every 5 minutes, so 3 is normal and `0` reads `DOWN` in red

An $agent template variable over service_name scopes all four panels to one agent or to all of them.

Import it

Save the dashboard

Save the JSON below as lua-grafana-dashboard.json.

Import

In Grafana, Dashboards → New → Import, upload the file, pick your Loki datasource for the DS_LOKI input, and Import.The dashboard’s UID is lua-log-drain, so re-importing updates it in place rather than creating a second copy.

Check the heartbeat panel

It should read 3 within fifteen minutes of a healthy drain. DOWN in red means either no heartbeat reached Loki or the attribute name differs in your pipeline — run the panel’s query in Explore and confirm lua_drain_backlog is the structured-metadata name your Loki produced.

The dashboard

lua-grafana-dashboard.json

{
  "__inputs": [
    {
      "name": "DS_LOKI",
      "label": "Loki",
      "description": "The Loki datasource fed by the OpenTelemetry Collector",
      "type": "datasource",
      "pluginId": "loki",
      "pluginName": "Loki"
    }
  ],
  "__requires": [
    { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
    { "type": "datasource", "id": "loki", "name": "Loki", "version": "1.0.0" },
    { "type": "panel", "id": "timeseries", "name": "Time series", "version": "" },
    { "type": "panel", "id": "stat", "name": "Stat", "version": "" }
  ],
  "annotations": { "list": [] },
  "description": "Agent logs shipped from Lua by an OTLP log drain, through an OpenTelemetry Collector into Loki.",
  "editable": true,
  "graphTooltip": 1,
  "id": null,
  "links": [
    {
      "title": "Ship logs to your stack",
      "type": "link",
      "url": "https://docs.heylua.ai/drains/overview",
      "targetBlank": true,
      "icon": "external link",
      "tags": []
    }
  ],
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Events by severity",
      "description": "Every record the drain delivered, split by severityText.",
      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
      "fieldConfig": {
        "defaults": {
          "custom": { "drawStyle": "bars", "fillOpacity": 60, "stacking": { "mode": "normal" }, "lineWidth": 0 },
          "unit": "short",
          "min": 0
        },
        "overrides": [
          { "matcher": { "id": "byName", "options": "ERROR" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }] },
          { "matcher": { "id": "byName", "options": "WARN" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } }] },
          { "matcher": { "id": "byName", "options": "INFO" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "blue" } }] },
          { "matcher": { "id": "byName", "options": "DEBUG" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "text" } }] }
        ]
      },
      "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
      "targets": [
        {
          "refId": "A",
          "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
          "queryType": "range",
          "editorMode": "code",
          "expr": "sum by (severity_text) (count_over_time({service_name=~\"$agent\"} | severity_text != `` [$__auto]))",
          "legendFormat": "{{severity_text}}"
        }
      ]
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "Errors by agent",
      "description": "ERROR records per agent. service_name is the Lua agent's name.",
      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 0 },
      "fieldConfig": {
        "defaults": {
          "custom": { "drawStyle": "bars", "fillOpacity": 70, "lineWidth": 0 },
          "unit": "short",
          "min": 0
        },
        "overrides": []
      },
      "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
      "targets": [
        {
          "refId": "A",
          "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
          "queryType": "range",
          "editorMode": "code",
          "expr": "sum by (service_name) (count_over_time({service_name=~\"$agent\"} | severity_text = `ERROR` [$__auto]))",
          "legendFormat": "{{service_name}}"
        }
      ]
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Tool-call errors",
      "description": "Failing tool calls, by the tool the model called. Covers both skills and MCP servers.",
      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
      "gridPos": { "h": 9, "w": 16, "x": 0, "y": 9 },
      "fieldConfig": {
        "defaults": {
          "custom": { "drawStyle": "bars", "fillOpacity": 70, "stacking": { "mode": "normal" }, "lineWidth": 0 },
          "unit": "short",
          "min": 0
        },
        "overrides": []
      },
      "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, "tooltip": { "mode": "multi", "sort": "desc" } },
      "targets": [
        {
          "refId": "A",
          "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
          "queryType": "range",
          "editorMode": "code",
          "expr": "sum by (gen_ai_tool_name) (count_over_time({service_name=~\"$agent\"} | severity_text = `ERROR` | gen_ai_tool_name != `` [$__auto]))",
          "legendFormat": "{{gen_ai_tool_name}}"
        }
      ]
    },
    {
      "id": 4,
      "type": "stat",
      "title": "Drain heartbeat (last 15m)",
      "description": "Heartbeats received in the last 15 minutes. A healthy drain sends one every 5 minutes, so 3 is normal and 0 means the pipeline is down.",
      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
      "gridPos": { "h": 9, "w": 8, "x": 16, "y": 9 },
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "min": 0,
          "mappings": [{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red", "index": 0 } } }],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "red", "value": null },
              { "color": "orange", "value": 1 },
              { "color": "green", "value": 2 }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
        "colorMode": "background",
        "graphMode": "area",
        "textMode": "auto",
        "justifyMode": "auto"
      },
      "targets": [
        {
          "refId": "A",
          "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
          "queryType": "instant",
          "editorMode": "code",
          "expr": "sum(count_over_time({service_name=~\"$agent\"} | lua_drain_backlog != `` [15m]))",
          "legendFormat": "heartbeats"
        }
      ]
    }
  ],
  "refresh": "1m",
  "schemaVersion": 39,
  "tags": ["lua", "log-drain", "agents"],
  "templating": {
    "list": [
      {
        "name": "agent",
        "label": "Agent",
        "type": "query",
        "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
        "query": { "label": "service_name", "refId": "LokiVariableQueryEditor-VariableQuery", "type": 1 },
        "refresh": 2,
        "includeAll": true,
        "allValue": ".+",
        "multi": true,
        "current": { "selected": true, "text": ["All"], "value": ["$__all"] },
        "sort": 1
      }
    ]
  },
  "time": { "from": "now-6h", "to": "now" },
  "timepicker": {},
  "timezone": "",
  "title": "Lua agents — logs",
  "uid": "lua-log-drain",
  "version": 1,
  "weekStart": ""
}

Why these selectors

Each panel keys on a field that is unique to what it measures, which avoids depending on the OTLP eventName field reaching Loki as a queryable label:

severity_text distinguishes the four severities directly.
gen_ai_tool_name is present only on tool calls, so severity_text = "ERROR" plus that attribute is exactly “a tool call failed”, across skills and MCP servers.
lua_drain_backlog is present only on the heartbeat record. Counting records that carry it is counting heartbeats.

The same logic gives you the other two synthetic records if you want panels for them: lua_drain_dropped_count appears only on a drop record, and lua_truncated_lines only on a truncation record.

# Records dropped between Lua and your backend
sum(count_over_time({service_name=~"$agent"} | lua_drain_dropped_count != `` [$__auto]))

# Executions whose output the runtime truncated
sum by (service_name) (count_over_time({service_name=~"$agent"} | lua_truncated_lines != `` [$__auto]))

The Datadog pack selects on @lua.eventName instead, because the Datadog encoder ships the whole record as a nested object and the event name is directly addressable there. The two packs watch the same four signals from opposite ends of the schema.

Alert on it

Grafana alert rules over the same queries, if Grafana rather than the backend owns your alerting:

Drain heartbeat missing — alert when IS BELOW 1 for 5m

sum(count_over_time({service_name=~".+"} | lua_drain_backlog != "" [15m]))

Errors spiking — alert when IS ABOVE 5 for 5m

sum by (service_name) (count_over_time({service_name=~".+"} | severity_text = "ERROR" [5m]))

Records dropped — alert when IS ABOVE 0

sum(count_over_time({service_name=~".+"} | lua_drain_dropped_count != "" [15m]))

Set the heartbeat rule’s No data handling to Alerting: with no heartbeats there is no series at all, and the default of No Data would leave it silent exactly when it matters.

Next steps

OpenTelemetry Collector config

The pipeline that feeds this dashboard.

OpenTelemetry

Create the drain.

Event schema

Where these attribute names come from.

Datadog monitor pack

The same signals, in Datadog.

Get started

Concepts

Build

Channels

Integrations

Devices

Ship and operate

Ship logs to your stack

Marketplace

Build with AI

Examples

Grafana dashboard

What you get

Import it

The dashboard

Why these selectors

Alert on it

Next steps

OpenTelemetry Collector config

OpenTelemetry

Event schema

Datadog monitor pack

​What you get

​Import it

​The dashboard

​Why these selectors

​Alert on it

​Next steps

OpenTelemetry Collector config

OpenTelemetry

Event schema

Datadog monitor pack

What you get

Import it

The dashboard

Why these selectors

Alert on it

Next steps