Windows Controller

Overview

Use case: Personal computer automation via WhatsApp or web chat Turn your Windows PC into an AI-controllable device. Ask your agent to take screenshots, open apps, search files, manage processes, control volume, and more — all from a chat message. Commands:

take_screenshot — Capture the screen and return the image
send_notification — Show a toast notification
open_url — Open a URL in the default browser
open_app — Launch an application
search_files — Find files by name
get_active_window — Get the current foreground window title
system_info — Hostname, OS, CPU, RAM, uptime, battery
get_clipboard — Read clipboard contents
set_clipboard — Write text to the clipboard
lock_screen — Lock the PC
set_volume — Set the system volume (0—100)
list_processes — List running processes with CPU usage
kill_process — Kill a process by name

Prerequisites:

Node.js 18+ (for the Node.js client) or Python 3.10+ (for the Python client)
PowerShell 5.1+ (included with Windows 10/11)
Optional: BurntToast PowerShell module for richer toast notifications (Install-Module -Name BurntToast)
Optional: nircmd for volume control (alternative to PowerShell)

Device Client

import { DeviceClient } from '@lua-ai-global/device-client';
import { execSync } from 'child_process';
import { readFileSync, unlinkSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';

// Helper to run PowerShell commands
function ps(command: string): string {
  return execSync(`powershell -NoProfile -Command "${command.replace(/"/g, '\\"')}"`)
    .toString()
    .trim();
}

const device = new DeviceClient({
  agentId: process.env.LUA_AGENT_ID!,
  apiKey: process.env.LUA_API_KEY!,
  deviceName: 'my-windows-pc',
  group: 'personal-devices',
  commands: [
    {
      name: 'take_screenshot',
      description: 'Capture a screenshot of the entire screen and return the image URL.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'send_notification',
      description: 'Show a Windows toast notification.',
      inputSchema: {
        type: 'object',
        properties: {
          title: { type: 'string', description: 'Notification title' },
          message: { type: 'string', description: 'Notification body text' },
        },
        required: ['title', 'message'],
      },
    },
    {
      name: 'open_url',
      description: 'Open a URL in the default browser.',
      inputSchema: {
        type: 'object',
        properties: {
          url: { type: 'string', description: 'The URL to open' },
        },
        required: ['url'],
      },
    },
    {
      name: 'open_app',
      description: 'Launch an application by name or path (e.g., "notepad", "calc", "C:\\Program Files\\App\\app.exe").',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Application name or path' },
        },
        required: ['name'],
      },
    },
    {
      name: 'search_files',
      description: 'Search for files by name. Searches common locations (Desktop, Documents, Downloads).',
      inputSchema: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Filename or wildcard pattern (e.g., "invoice.pdf", "*.xlsx")' },
          path: { type: 'string', description: 'Directory to search (default: user home)' },
          limit: { type: 'number', description: 'Max results (default 10)' },
        },
        required: ['query'],
      },
    },
    {
      name: 'get_active_window',
      description: 'Get the title of the currently focused window.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'system_info',
      description: 'Get system information: hostname, OS version, CPU, RAM, uptime, and battery level.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'get_clipboard',
      description: 'Read the current clipboard text contents.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_clipboard',
      description: 'Set the clipboard text contents.',
      inputSchema: {
        type: 'object',
        properties: {
          text: { type: 'string', description: 'Text to copy to clipboard' },
        },
        required: ['text'],
      },
    },
    {
      name: 'lock_screen',
      description: 'Lock the Windows PC immediately.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_volume',
      description: 'Set the system output volume.',
      inputSchema: {
        type: 'object',
        properties: {
          level: { type: 'number', minimum: 0, maximum: 100, description: 'Volume level 0-100' },
        },
        required: ['level'],
      },
    },
    {
      name: 'list_processes',
      description: 'List running processes sorted by CPU usage.',
      inputSchema: {
        type: 'object',
        properties: {
          limit: { type: 'number', description: 'Max number of processes to return (default 10)' },
        },
      },
    },
    {
      name: 'kill_process',
      description: 'Kill a running process by name.',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Process name (e.g., "notepad", "chrome")' },
        },
        required: ['name'],
      },
    },
  ],
});

device.onCommand('take_screenshot', async () => {
  const filepath = join(tmpdir(), `screenshot-${Date.now()}.png`);
  ps(`
    Add-Type -AssemblyName System.Windows.Forms;
    Add-Type -AssemblyName System.Drawing;
    $screen = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds;
    $bitmap = New-Object System.Drawing.Bitmap($screen.Width, $screen.Height);
    $graphics = [System.Drawing.Graphics]::FromImage($bitmap);
    $graphics.CopyFromScreen($screen.Location, [System.Drawing.Point]::Empty, $screen.Size);
    $bitmap.Save('${filepath.replace(/\\/g, '\\\\')}');
    $graphics.Dispose();
    $bitmap.Dispose()
  `);
  const buffer = readFileSync(filepath);
  const url = await device.uploadFile(buffer, `screenshot-${Date.now()}.png`, 'image/png');
  unlinkSync(filepath);
  return { imageUrl: url, timestamp: new Date().toISOString() };
});

device.onCommand('send_notification', async (payload) => {
  const title = payload.title.replace(/'/g, "''");
  const message = payload.message.replace(/'/g, "''");
  try {
    ps(`New-BurntToastNotification -Text '${title}', '${message}'`);
  } catch {
    // Fallback if BurntToast is not installed
    ps(`
      [Windows.UI.Notifications.ToastNotificationManager, Windows.UI.Notifications, ContentType = WindowsRuntime] | Out-Null;
      $xml = '<toast><visual><binding template="ToastText02"><text id="1">${title}</text><text id="2">${message}</text></binding></visual></toast>';
      $toast = [Windows.UI.Notifications.ToastNotification]::new([Windows.Data.Xml.Dom.XmlDocument]::new());
      $toast.Content.LoadXml($xml);
      [Windows.UI.Notifications.ToastNotificationManager]::CreateToastNotifier('Lua Agent').Show($toast)
    `);
  }
  return { sent: true, title: payload.title, message: payload.message };
});

device.onCommand('open_url', async (payload) => {
  execSync(`start "" "${payload.url}"`);
  return { opened: true, url: payload.url };
});

device.onCommand('open_app', async (payload) => {
  ps(`Start-Process '${payload.name}'`);
  return { opened: true, app: payload.name };
});

device.onCommand('search_files', async (payload) => {
  const limit = payload.limit || 10;
  const searchPath = payload.path || '$env:USERPROFILE';
  const raw = ps(`
    Get-ChildItem -Path ${searchPath} -Recurse -Filter '${payload.query}' -ErrorAction SilentlyContinue |
    Select-Object -First ${limit} -ExpandProperty FullName
  `);
  const files = raw ? raw.split('\r\n').filter(Boolean) : [];
  return { query: payload.query, results: files, count: files.length };
});

device.onCommand('get_active_window', async () => {
  const title = ps(`
    Add-Type -Name Win32 -Namespace Native -MemberDefinition '[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow(); [DllImport("user32.dll", CharSet = CharSet.Auto)] public static extern int GetWindowText(IntPtr hWnd, System.Text.StringBuilder lpString, int nMaxCount);';
    $sb = New-Object System.Text.StringBuilder(256);
    [Native.Win32]::GetWindowText([Native.Win32]::GetForegroundWindow(), $sb, 256) | Out-Null;
    $sb.ToString()
  `);
  return { activeWindow: title };
});

device.onCommand('system_info', async () => {
  const info = ps(`
    $os = Get-CimInstance Win32_OperatingSystem;
    $cpu = (Get-CimInstance Win32_Processor).Name;
    $ramGB = [math]::Round($os.TotalVisibleMemorySize / 1MB, 1);
    $uptime = (Get-Date) - $os.LastBootUpTime;
    $battery = (Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue).EstimatedChargeRemaining;
    @{
      hostname = $env:COMPUTERNAME;
      os = $os.Caption;
      cpu = $cpu;
      memoryGB = $ramGB;
      uptime = '{0}d {1}h {2}m' -f $uptime.Days, $uptime.Hours, $uptime.Minutes;
      battery = if ($battery) { "$battery%" } else { 'N/A (desktop)' }
    } | ConvertTo-Json
  `);
  return JSON.parse(info);
});

device.onCommand('get_clipboard', async () => {
  const text = ps('Get-Clipboard');
  return { clipboard: text };
});

device.onCommand('set_clipboard', async (payload) => {
  const escaped = payload.text.replace(/'/g, "''");
  ps(`Set-Clipboard -Value '${escaped}'`);
  return { set: true, text: payload.text };
});

device.onCommand('lock_screen', async () => {
  execSync('rundll32.exe user32.dll,LockWorkStation');
  return { locked: true, timestamp: new Date().toISOString() };
});

device.onCommand('set_volume', async (payload) => {
  ps(`
    $wsh = New-Object -ComObject WScript.Shell;
    1..50 | ForEach-Object { $wsh.SendKeys([char]174) };
    $steps = [math]::Round(${payload.level} / 2);
    1..$steps | ForEach-Object { $wsh.SendKeys([char]175) }
  `);
  return { volume: payload.level };
});

device.onCommand('list_processes', async (payload) => {
  const limit = payload?.limit || 10;
  const raw = ps(`
    Get-Process | Sort-Object CPU -Descending |
    Select-Object -First ${limit} Name, Id, @{N='CPU_Seconds';E={[math]::Round($_.CPU,1)}}, @{N='Memory_MB';E={[math]::Round($_.WorkingSet64/1MB,1)}} |
    ConvertTo-Json
  `);
  return { processes: JSON.parse(raw) };
});

device.onCommand('kill_process', async (payload) => {
  const name = payload.name.replace(/'/g, "''");
  ps(`Stop-Process -Name '${name}' -Force -ErrorAction SilentlyContinue`);
  return { killed: true, process: payload.name };
});

async function main() {
  await device.connect();
  console.log('Windows controller online');
}

main().catch(console.error);

import asyncio
import json
import os
import subprocess
import tempfile
from datetime import datetime, timezone
from lua_device import DeviceClient, DeviceCommandDefinition


def ps(command: str) -> str:
    """Run a PowerShell command and return its output."""
    result = subprocess.run(
        ["powershell", "-NoProfile", "-Command", command],
        capture_output=True, text=True,
    )
    return result.stdout.strip()


client = DeviceClient(
    agent_id=os.environ["LUA_AGENT_ID"],
    api_key=os.environ["LUA_API_KEY"],
    device_name="my-windows-pc",
    group="personal-devices",
    commands=[
        DeviceCommandDefinition(
            name="take_screenshot",
            description="Capture a screenshot of the entire screen and return the image URL.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="send_notification",
            description="Show a Windows toast notification.",
            input_schema={
                "type": "object",
                "properties": {
                    "title": {"type": "string", "description": "Notification title"},
                    "message": {"type": "string", "description": "Notification body text"},
                },
                "required": ["title", "message"],
            },
        ),
        DeviceCommandDefinition(
            name="open_url",
            description="Open a URL in the default browser.",
            input_schema={
                "type": "object",
                "properties": {
                    "url": {"type": "string", "description": "The URL to open"},
                },
                "required": ["url"],
            },
        ),
        DeviceCommandDefinition(
            name="open_app",
            description="Launch an application by name or path (e.g., 'notepad', 'calc').",
            input_schema={
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Application name or path"},
                },
                "required": ["name"],
            },
        ),
        DeviceCommandDefinition(
            name="search_files",
            description="Search for files by name. Searches common locations (Desktop, Documents, Downloads).",
            input_schema={
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Filename or wildcard pattern (e.g., 'invoice.pdf', '*.xlsx')"},
                    "path": {"type": "string", "description": "Directory to search (default: user home)"},
                    "limit": {"type": "number", "description": "Max results (default 10)"},
                },
                "required": ["query"],
            },
        ),
        DeviceCommandDefinition(
            name="get_active_window",
            description="Get the title of the currently focused window.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="system_info",
            description="Get system information: hostname, OS version, CPU, RAM, uptime, and battery level.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="get_clipboard",
            description="Read the current clipboard text contents.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="set_clipboard",
            description="Set the clipboard text contents.",
            input_schema={
                "type": "object",
                "properties": {
                    "text": {"type": "string", "description": "Text to copy to clipboard"},
                },
                "required": ["text"],
            },
        ),
        DeviceCommandDefinition(
            name="lock_screen",
            description="Lock the Windows PC immediately.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="set_volume",
            description="Set the system output volume.",
            input_schema={
                "type": "object",
                "properties": {
                    "level": {"type": "number", "minimum": 0, "maximum": 100, "description": "Volume level 0-100"},
                },
                "required": ["level"],
            },
        ),
        DeviceCommandDefinition(
            name="list_processes",
            description="List running processes sorted by CPU usage.",
            input_schema={
                "type": "object",
                "properties": {
                    "limit": {"type": "number", "description": "Max number of processes to return (default 10)"},
                },
            },
        ),
        DeviceCommandDefinition(
            name="kill_process",
            description="Kill a running process by name.",
            input_schema={
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Process name (e.g., 'notepad', 'chrome')"},
                },
                "required": ["name"],
            },
        ),
    ],
)


@client.on_command("take_screenshot")
async def handle_take_screenshot(payload):
    filepath = os.path.join(tempfile.gettempdir(), f"screenshot-{int(datetime.now().timestamp())}.png")
    ps(f"""
        Add-Type -AssemblyName System.Windows.Forms;
        Add-Type -AssemblyName System.Drawing;
        $screen = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds;
        $bitmap = New-Object System.Drawing.Bitmap($screen.Width, $screen.Height);
        $graphics = [System.Drawing.Graphics]::FromImage($bitmap);
        $graphics.CopyFromScreen($screen.Location, [System.Drawing.Point]::Empty, $screen.Size);
        $bitmap.Save('{filepath}');
        $graphics.Dispose();
        $bitmap.Dispose()
    """)
    with open(filepath, "rb") as f:
        buffer = f.read()
    url = await client.upload_file(buffer, f"screenshot-{int(datetime.now().timestamp())}.png", "image/png")
    os.unlink(filepath)
    return {"imageUrl": url, "timestamp": datetime.now(timezone.utc).isoformat()}


@client.on_command("send_notification")
async def handle_send_notification(payload):
    title = payload["title"].replace("'", "''")
    message = payload["message"].replace("'", "''")
    try:
        ps(f"New-BurntToastNotification -Text '{title}', '{message}'")
    except Exception:
        ps(f"""
            [Windows.UI.Notifications.ToastNotificationManager, Windows.UI.Notifications, ContentType = WindowsRuntime] | Out-Null;
            $xml = '<toast><visual><binding template="ToastText02"><text id="1">{title}</text><text id="2">{message}</text></binding></visual></toast>';
            $toast = [Windows.UI.Notifications.ToastNotification]::new([Windows.Data.Xml.Dom.XmlDocument]::new());
            $toast.Content.LoadXml($xml);
            [Windows.UI.Notifications.ToastNotificationManager]::CreateToastNotifier('Lua Agent').Show($toast)
        """)
    return {"sent": True, "title": payload["title"], "message": payload["message"]}


@client.on_command("open_url")
async def handle_open_url(payload):
    subprocess.run(["start", "", payload["url"]], shell=True, check=True)
    return {"opened": True, "url": payload["url"]}


@client.on_command("open_app")
async def handle_open_app(payload):
    ps(f"Start-Process '{payload['name']}'")
    return {"opened": True, "app": payload["name"]}


@client.on_command("search_files")
async def handle_search_files(payload):
    limit = payload.get("limit", 10)
    search_path = payload.get("path", "$env:USERPROFILE")
    query = payload["query"].replace("'", "''")
    raw = ps(f"""
        Get-ChildItem -Path {search_path} -Recurse -Filter '{query}' -ErrorAction SilentlyContinue |
        Select-Object -First {limit} -ExpandProperty FullName
    """)
    files = [f for f in raw.split("\r\n") if f] if raw else []
    return {"query": payload["query"], "results": files, "count": len(files)}


@client.on_command("get_active_window")
async def handle_get_active_window(payload):
    title = ps("""
        Add-Type -Name Win32 -Namespace Native -MemberDefinition '[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow(); [DllImport("user32.dll", CharSet = CharSet.Auto)] public static extern int GetWindowText(IntPtr hWnd, System.Text.StringBuilder lpString, int nMaxCount);';
        $sb = New-Object System.Text.StringBuilder(256);
        [Native.Win32]::GetWindowText([Native.Win32]::GetForegroundWindow(), $sb, 256) | Out-Null;
        $sb.ToString()
    """)
    return {"activeWindow": title}


@client.on_command("system_info")
async def handle_system_info(payload):
    raw = ps("""
        $os = Get-CimInstance Win32_OperatingSystem;
        $cpu = (Get-CimInstance Win32_Processor).Name;
        $ramGB = [math]::Round($os.TotalVisibleMemorySize / 1MB, 1);
        $uptime = (Get-Date) - $os.LastBootUpTime;
        $battery = (Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue).EstimatedChargeRemaining;
        @{
            hostname = $env:COMPUTERNAME;
            os = $os.Caption;
            cpu = $cpu;
            memoryGB = $ramGB;
            uptime = '{0}d {1}h {2}m' -f $uptime.Days, $uptime.Hours, $uptime.Minutes;
            battery = if ($battery) { "$battery%" } else { 'N/A (desktop)' }
        } | ConvertTo-Json
    """)
    return json.loads(raw)


@client.on_command("get_clipboard")
async def handle_get_clipboard(payload):
    text = ps("Get-Clipboard")
    return {"clipboard": text}


@client.on_command("set_clipboard")
async def handle_set_clipboard(payload):
    escaped = payload["text"].replace("'", "''")
    ps(f"Set-Clipboard -Value '{escaped}'")
    return {"set": True, "text": payload["text"]}


@client.on_command("lock_screen")
async def handle_lock_screen(payload):
    subprocess.run(["rundll32.exe", "user32.dll,LockWorkStation"], check=True)
    return {"locked": True, "timestamp": datetime.now(timezone.utc).isoformat()}


@client.on_command("set_volume")
async def handle_set_volume(payload):
    ps(f"""
        $wsh = New-Object -ComObject WScript.Shell;
        1..50 | ForEach-Object {{ $wsh.SendKeys([char]174) }};
        $steps = [math]::Round({payload['level']} / 2);
        1..$steps | ForEach-Object {{ $wsh.SendKeys([char]175) }}
    """)
    return {"volume": payload["level"]}


@client.on_command("list_processes")
async def handle_list_processes(payload):
    limit = payload.get("limit", 10) if payload else 10
    raw = ps(f"""
        Get-Process | Sort-Object CPU -Descending |
        Select-Object -First {limit} Name, Id, @{{N='CPU_Seconds';E={{[math]::Round($_.CPU,1)}}}}, @{{N='Memory_MB';E={{[math]::Round($_.WorkingSet64/1MB,1)}}}} |
        ConvertTo-Json
    """)
    return {"processes": json.loads(raw)}


@client.on_command("kill_process")
async def handle_kill_process(payload):
    name = payload["name"].replace("'", "''")
    ps(f"Stop-Process -Name '{name}' -Force -ErrorAction SilentlyContinue")
    return {"killed": True, "process": payload["name"]}


async def main():
    await client.connect()
    print("Windows controller online")

asyncio.run(main())

Agent Configuration

// src/index.ts
import { LuaAgent, LuaSkill } from 'lua-cli';

const windowsControlSkill = new LuaSkill({
  name: 'windows-controller',
  description: 'Control a Windows PC remotely',
  context: `
    You are a personal assistant that controls a Windows PC.
    You can take screenshots, open apps, search files, manage processes,
    control volume, and more.

    Device tools:
    - take_screenshot: Captures the screen and returns an image URL.
    - send_notification: Shows a Windows toast notification with a title and message.
    - open_url: Opens a URL in the default browser.
    - open_app: Launches an application by name or path.
    - search_files: Searches for files by name across user directories.
    - get_active_window: Reports the title of the currently focused window.
    - system_info: Returns hostname, OS, CPU, RAM, uptime, and battery status.
    - get_clipboard: Reads the current clipboard text.
    - set_clipboard: Sets the clipboard to the provided text.
    - lock_screen: Locks the PC immediately.
    - set_volume: Sets the system volume from 0 to 100.
    - list_processes: Lists running processes sorted by CPU usage.
    - kill_process: Kills a process by name. Use with caution.

    Guidelines:
    - When asked for a screenshot, take it and share the image URL
    - For file searches, show the full paths in the results
    - Confirm destructive actions (locking the screen, killing processes) before executing
    - Be conversational and helpful
  `,
  tools: [],
});

export const agent = new LuaAgent({
  name: 'windows-assistant',
  persona: `You are a personal assistant that controls a Windows PC. You can take
    screenshots, open apps, search files, manage processes, control volume,
    and automate tasks. Be helpful, concise, and confirm before taking
    potentially disruptive actions like locking the screen or killing processes.`,
  skills: [windowsControlSkill],
});

What You Can Ask

Here are real conversational examples you can send from WhatsApp or web chat:

Screenshots

“Take a screenshot and send it to me”

Open Apps

“Open Notepad” / “Launch Excel”

File Search

“Search for files called invoice.pdf”

Active Window

“What window do I have open right now?”

Lock Screen

“Lock my PC”

Volume

“Set the volume to 30%”

Processes

“What processes are using the most CPU?”

Kill Process

“Kill the Notepad process”

Clipboard

“Copy this text to my clipboard: Meeting at 3pm”

Notifications

“Send me a notification that says Stand up and stretch”

Browser

“Open github.com in my browser”

System Info

“How much RAM do I have? What’s my uptime?”

Security note: This device client gives your AI agent direct control over your computer. Only run it on machines you trust, and consider limiting which commands are registered based on your comfort level.

Next Steps

Mac Controller

Control a Mac the same way

Smart Office

Office automation with sensors and displays

CDN Uploads

How screenshot uploads work

Self-Describing Commands

How to write effective command definitions

Getting Started

Device Client

Agent Integration

Examples

Advanced

CLI Commands

Windows Controller

Overview

Device Client

Agent Configuration

What You Can Ask

Screenshots

Open Apps

File Search

Active Window

Lock Screen

Volume

Processes

Kill Process

Clipboard

Notifications

Browser

System Info

Next Steps

Mac Controller

Smart Office

CDN Uploads

Self-Describing Commands

​Overview

​Device Client

​Agent Configuration

​What You Can Ask

Screenshots

Open Apps

File Search

Active Window

Lock Screen

Volume

Processes

Kill Process

Clipboard

Notifications

Browser

System Info

​Next Steps

Mac Controller

Smart Office

CDN Uploads

Self-Describing Commands

Overview

Device Client

Agent Configuration

What You Can Ask

Next Steps