Skip to main content

Overview

LuaVoice is the class-based primitive you define in code to declare a voice-enabled agent — its speech-to-text engine, text-to-speech engine, LLM, turn detection, and any voice-specific tools.
For testing voice agents live or running automated voice tests, see the Voice Command. For the direct plugin route (when string descriptors aren’t enough), see Plugin and Realtime Engines below.
Persona is configured on the parent LuaAgent, not on LuaVoice. Use the channel-aware persona shape { base, voice, text } on the agent to give a voice its own prompt — see Channel-Aware Personas.

llm, stt, and tts all accept a provider-prefixed string descriptor. This is the canonical form — it routes through Lua’s inference layer so you don’t manage provider credentials yourself.
The model and voice catalogs below are a living list — your descriptor is forwarded straight to Lua’s inference layer, so newer provider models may work before they’re listed here and retired ones may drop off. Treat these tables as a starting point, not an exhaustive allowlist.

LLM options

Provider-prefixed model id. Grouped by tier — pick a tier based on the latency/cost/quality trade-off you need. Fast tier — lowest latency, lowest cost: Balanced tier — good default for most voice agents: Quality tier — best capability, higher latency/cost:
Anthropic / Claude is intentionally absent — Lua’s inference layer does not carry Anthropic models for voice as of this writing. Use OpenAI, Google, xAI, DeepSeek, or Kimi for voice LLMs.

STT options

Deepgram is the recommended STT provider, and deepgram/nova-3 is the standard choice. stt is required for cascaded LLMs — omit it only when the llm is a realtime speech-to-speech model (which handles audio directly).
Combine with sttLanguage to pin the spoken language:
  • BCP-47 code ('en', 'es', 'pt-BR', etc.) — pins recognition to that language.
  • 'multi' — multilingual transcription. Applies to both the Inference route and the direct Deepgram plugin.
Want non-default Deepgram options (smart formatting, filler-word filtering, custom keywords)? Use the plugin class form: stt: new deepgram.STT({ model: 'nova-3', smartFormat: true }). See Plugin and Realtime Engines for the full plugin route.

ElevenLabs Scribe

ElevenLabs has an STT model called Scribe, available via the Inference route:
Useful when you want STT and TTS from the same provider, or when Scribe’s behavior on a specific language outperforms Deepgram in your testing.

TTS options

ElevenLabs is the canonical TTS provider. The descriptor format is elevenlabs/<model>:<voiceId>.
Models: Curated voice IDs: Lua maintains a curated list with metadata (gender, accent, style) the raw ElevenLabs API doesn’t expose: You can also use any ElevenLabs voice ID from your own ElevenLabs account — these are just the curated defaults. Alternative: object form If you’d rather not concatenate model and voice with a colon, the object form works too:

Deepgram Aura

Deepgram offers TTS via the Aura family. The voice id is encoded inside the model id as aura-2-<name>-<lang>:
Common Aura 2 voices (English): Spanish voices are also available: aura-2-celeste-es, aura-2-estrella-es.

Other TTS providers (via Inference)

Lua’s inference layer also exposes Cartesia, Inworld, Rime, and xAI TTS. The descriptors follow the same provider/model shape:

Plugin and Realtime Engines

For most voice agents the string-descriptor form above is all you need. Reach for the plugin/class forms here in two cases: (1) you need provider-specific options the descriptor route doesn’t expose, or (2) you’re using a realtime (speech-to-speech) model in the llm slot. lua-cli/voice re-exports the LiveKit plugin namespaces that LuaVoice accepts as class instances — importing through it means you don’t add the underlying plugin packages as direct dependencies:

What’s allowed where

The compiler enforces two separate allowlists:
new openai.LLM(...), new google.LLM(...), new xai.LLM(...) and similar class forms fail compile-time validation. These providers are not on the plugin allowlist. Use string descriptors ('openai/gpt-5') or — for speech-to-speech — the realtime form (new openai.realtime.RealtimeModel({...})).

Plugin route: Deepgram + ElevenLabs

The two allowlisted plugin providers. Use these class forms when you need provider-specific options not exposed by the string-descriptor route.

Deepgram STT (plugin form)

Deepgram exposes two STT classes:
  • new deepgram.STT({...}) — Deepgram’s v1 WebSocket endpoint. Use this for nova-3, nova-2, etc.
  • new deepgram.STTv2({...}) — Deepgram’s v2 endpoint. Required for Flux models that use semantic endpointing (eotThreshold, eagerEotThreshold, eotTimeoutMs).
The compiler routes each to the correct underlying plugin based on which class you used.

ElevenLabs TTS (plugin form)

The plugin route lets you pass advanced ElevenLabs options (stability, similarity boost, style, speaker boost, etc.) that the descriptor route doesn’t surface.

Inference route (typed shortcut)

inference.LLM, inference.STT, inference.TTS are typed wrappers for the string-descriptor route. The compiler normalizes both forms to the same wire shape; the class form just gives you better TypeScript autocomplete on the options.
The model option is required — it’s the same provider-prefixed string you’d pass directly. For TTS, pass voice separately. This is the only way to use class syntax for providers that aren’t on the plugin allowlist (OpenAI, Google, xAI, Cartesia, etc.).

Realtime route (speech-to-speech)

The realtime route puts a speech-to-speech model in the llm slot, replacing the cascaded STT → LLM → TTS pipeline. The class-construction path differs by provider:
  • OpenAI: new openai.realtime.RealtimeModel({...})
  • Google (Gemini): new google.beta.realtime.RealtimeModel({...}) — note the .beta. prefix (matches Google’s Node SDK shape)

Available realtime models

xai is reserved in the realtime allowlist but no xAI realtime models are currently published.

Half-cascade mode

You can keep a separate tts with a realtime LLM — the worker injects modalities: ['text'] so the realtime model emits text and tts handles synthesis. Useful when you want realtime’s low-latency reasoning but ElevenLabs’ voice quality:
You cannot combine a realtime llm with a custom stt — the compiler rejects it. Realtime models handle audio input directly.

Credentials

Plugin class instances rely on credentials provisioned by the Lua platform — you do not need to set DEEPGRAM_API_KEY, ELEVENLABS_API_KEY, etc. in your project’s .env. Lua manages the provider credentials for you; your code just references the class form and the platform constructs the actual engine at runtime.

When to use which form


Configuration Reference

Required fields

string
required
Unique name for this voice. Used to address the voice in lua voice --voice <name> and as the server-side identifier. Allowed characters: a-zA-Z0-9_-, 1–64 chars.
string | LLMConfig
required
The LLM that drives the conversation. String descriptor (e.g. 'openai/gpt-5.1-chat-latest') is the canonical form. See LLM options above for the catalog.
string | STTConfig
required
Speech-to-text engine. String descriptor (e.g. 'deepgram/nova-3') is canonical. Required for cascaded LLMs; omit only when using a realtime speech-to-speech model in the llm slot.
string | TTSConfig
required
Text-to-speech engine. String descriptor with colon-separated voice id (e.g. 'elevenlabs/eleven_turbo_v2_5:<voiceId>'), or object form { model, voice }. Required for cascaded LLMs.

Optional fields

string
Human-readable description. Surfaced in the compiled manifest and admin listings.
string
Opening line spoken at session start. Empty string means no greeting. Generated through the LLM at session connect, so it can be dynamic if onEnter sets up context first.
string
BCP-47 language code (e.g. 'en', 'es', 'pt-BR') or 'multi' for multilingual transcription. Applies to both Inference STT and the Deepgram plugin.
'multilingual' | 'english' | 'vad' | 'stt' | 'manual'
How the agent decides when the user has finished speaking. 'vad' is the safest choice for most setups. 'multilingual' and 'english' use LiveKit’s turn-detector model; 'manual' defers to your own logic.
string
default:"silero"
Voice activity detection engine. 'silero' is the only currently-supported value.
object
Silero VAD tuning. Useful when the default endpointing clips quiet callers or fires too eagerly mid-thought.
  • minSpeechDuration (ms, 0–5000) — speech required before a turn starts. Default: 50.
  • minSilenceDuration (ms, 0–5000) — silence required to end a turn. Default: 550.
  • prefixPaddingDuration (ms, 0–2000) — audio captured before detected speech start, forwarded into STT. Default: 500.
  • activationThreshold (0–1) — lower = more sensitive to speech onset.
boolean
default:"false"
Krisp BVC background noise cancellation. Recommended for inbound phone calls — it removes background chatter, traffic, and other ambient noise. Billed separately, so opt-in.
number
Maximum sequential tool calls per turn (1–20). Higher values let the agent chain more tools before responding.
number
Seconds of silence before the agent considers the user “away” and ends the session. Useful for cleanly handling abandoned calls.
boolean
Generate the assistant’s response speculatively as the user is still speaking. Reduces perceived latency for predictable turns but can be wasted on highly interruptive callers.
InterruptionOptions
How the agent handles being interrupted mid-response.
  • enabled — whether interruption is allowed.
  • mode'adaptive' (recommended) or 'vad'.
  • falseInterruptionTimeout (seconds) — how long to wait before treating a brief noise as a false interruption.
  • resumeFalseInterruption (boolean) — resume the cut-off response after a false interruption.
  • minDelay / maxDelay (seconds) — bounds on the interruption response window.
Record<string, string>
Word-boundary text replacements applied before TTS synthesis. Keys are matched case-insensitively as whole words. Use for acronyms and proper nouns the TTS mispronounces.
Cascaded path only. Setting pronunciations on a full-realtime voice (realtime llm with no tts) is rejected at compile time — pair with a half-cascade tts, or drop the field.
{ ambient?, thinking? }
Background audio layered onto the agent’s output. Pass a built-in clip name, a { source, volume, probability } config, or an array (probabilistic mix).Built-in clips: 'office-ambience', 'keyboard-typing', 'keyboard-typing-2'.
number
Output speech volume, 0–100. Applied as a per-frame multiplier. Omit to pass the TTS provider’s native level through unchanged.
boolean
default:"false"
When true, the worker writes session.history to Data.set('call:<sessionId>') after the call ends. Read it back from a job or webhook with Data.get('call:<sessionId>') for post-call analytics, follow-ups, or QA.
string
Short line spoken to the caller when a tool call fails (throws, times out, or returns an unsupported result) — fills the 2–3s gap before the LLM’s own recovery response. Spoken once per failed call, then the error is surfaced to the LLM. Keep it short and on-brand (e.g. 'Sorry, let me try that another way.'); omit for no spoken fallback.
Array<LuaTool | LuaVoiceTool>
Voice-specific tools in addition to skills attached to the owning agent. See Defining Voice Tools.

Lifecycle Hooks

Three hooks let you wire up per-session state, RAG injection, and post-call work.
(ctx: LuaVoiceHookContext) => Promise<void>
Fires after the session connects to the room and before the greeting. Use it to hydrate session.userdata from User, Data, etc., or to set up any per-call state.
(turnCtx, message) => Promise<void>
Fires after the user finishes a turn, before the LLM is invoked. This is the canonical RAG-injection point — turnCtx.addMessage(...) adds context messages the LLM sees on this turn.
(ctx: LuaVoiceHookContext) => Promise<void>
Fires when the session is closing. Use for transcript persistence, outcome reporting, CRM updates, etc.

Defining Voice Tools

Voice tools run during a voice conversation. LuaVoiceTool is a concrete class — instantiate it with a config object:

Config fields

string
required
Tool name. Used by the LLM to identify and call the tool.
string
required
What the tool does. Action-oriented description the LLM reads when deciding to invoke.
ZodType
required
Zod schema for the tool’s input. Validated before execute is called.
(input, ctx?: LuaVoiceToolCtx) => Promise<any>
required
Tool body. Receives the validated input and an optional voice-specific context.
() => Promise<boolean>
Optional gate. When provided, the tool is only exposed to the LLM if condition() returns true. Use for feature flags or runtime availability checks.
ToolFlag[]
Voice-specific tool flags (e.g. controlling barge-in behavior).

ctx — LuaVoiceToolCtx

string
Identifier for this specific tool invocation.
(text: string) => Promise<void>
Speak text to the caller via the active LiveKit session. Useful for status updates during long-running tool work (“Looking that up — one moment.”).
(msisdn, opts?) => Promise<void>
Transfer the live caller to a human at msisdn. Two mechanisms:
  • mode: 'refer' (default) — SIP REFER on the inbound leg. Cheap (one billed leg) but depends on the inbound carrier accepting REFER end-to-end.
  • mode: 'bridge' — dial the human as a second SIP participant into the same room. Two billed legs but works regardless of carrier REFER support. Use for high-stakes transfers.
announce is spoken before the transfer fires.
You can also share regular LuaTool instances between chat skills and voice tools — just pass them in the same tools array. The tools field accepts both LuaTool and LuaVoiceTool instances.

Function-style: defineVoice

Equivalent to new LuaVoice(config) if you prefer a function call:

Wiring Up to an Agent

The agent’s persona.voice branch is what gives supportLine its voice-specific prompt.
Voice persona tips:
  • Keep replies short (1–2 sentences). Voice users can’t skim.
  • No markdown — TTS reads it literally.
  • Spell out numbers and prices (“nine o’clock”, “twenty dollars”) — TTS reads digits robotically otherwise.

Connect a phone number

Attaching the voice in code makes it available; to make the agent answer phone calls, bind a number to it. Push your voice first, then run the channels flow and choose “☎️ Manage phone numbers”:
From there you can search available numbers, purchase one, and bind it to this agent. During bind you pick which LuaVoice answers inbound calls on that number:
Binding requires a code-defined LuaVoice that has been pushed. Without one, inbound calls fall through to platform-default STT/LLM/TTS — no greeting, lifecycle hooks, or voice-only tools. Author the voice, lua push, then bind.
When purchasing, answering “Allow customers to text this number too?” with yes provisions a voice + SMS number; no provisions a voice-only number with lower-latency inbound. See Channels Command for the full phone-number flow (list, unbind, release).