Skip to main content

Overview

Use case: Personal computer automation via WhatsApp or web chat Turn your Windows PC into an AI-controllable device. Ask your agent to take screenshots, open apps, search files, manage processes, control volume, and more — all from a chat message. Commands:
  • take_screenshot — Capture the screen and return the image
  • send_notification — Show a toast notification
  • open_url — Open a URL in the default browser
  • open_app — Launch an application
  • search_files — Find files by name
  • get_active_window — Get the current foreground window title
  • system_info — Hostname, OS, CPU, RAM, uptime, battery
  • get_clipboard — Read clipboard contents
  • set_clipboard — Write text to the clipboard
  • lock_screen — Lock the PC
  • set_volume — Set the system volume (0—100)
  • list_processes — List running processes with CPU usage
  • kill_process — Kill a process by name
Prerequisites:
  • Node.js 18+ (for the Node.js client) or Python 3.10+ (for the Python client)
  • PowerShell 5.1+ (included with Windows 10/11)
  • Optional: BurntToast PowerShell module for richer toast notifications (Install-Module -Name BurntToast)
  • Optional: nircmd for volume control (alternative to PowerShell)

Device Client

import { DeviceClient } from '@lua-ai-global/device-client';
import { execSync } from 'child_process';
import { readFileSync, unlinkSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';

// Helper to run PowerShell commands
function ps(command: string): string {
  return execSync(`powershell -NoProfile -Command "${command.replace(/"/g, '\\"')}"`)
    .toString()
    .trim();
}

const device = new DeviceClient({
  agentId: process.env.LUA_AGENT_ID!,
  apiKey: process.env.LUA_API_KEY!,
  deviceName: 'my-windows-pc',
  group: 'personal-devices',
  commands: [
    {
      name: 'take_screenshot',
      description: 'Capture a screenshot of the entire screen and return the image URL.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'send_notification',
      description: 'Show a Windows toast notification.',
      inputSchema: {
        type: 'object',
        properties: {
          title: { type: 'string', description: 'Notification title' },
          message: { type: 'string', description: 'Notification body text' },
        },
        required: ['title', 'message'],
      },
    },
    {
      name: 'open_url',
      description: 'Open a URL in the default browser.',
      inputSchema: {
        type: 'object',
        properties: {
          url: { type: 'string', description: 'The URL to open' },
        },
        required: ['url'],
      },
    },
    {
      name: 'open_app',
      description: 'Launch an application by name or path (e.g., "notepad", "calc", "C:\\Program Files\\App\\app.exe").',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Application name or path' },
        },
        required: ['name'],
      },
    },
    {
      name: 'search_files',
      description: 'Search for files by name. Searches common locations (Desktop, Documents, Downloads).',
      inputSchema: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Filename or wildcard pattern (e.g., "invoice.pdf", "*.xlsx")' },
          path: { type: 'string', description: 'Directory to search (default: user home)' },
          limit: { type: 'number', description: 'Max results (default 10)' },
        },
        required: ['query'],
      },
    },
    {
      name: 'get_active_window',
      description: 'Get the title of the currently focused window.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'system_info',
      description: 'Get system information: hostname, OS version, CPU, RAM, uptime, and battery level.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'get_clipboard',
      description: 'Read the current clipboard text contents.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_clipboard',
      description: 'Set the clipboard text contents.',
      inputSchema: {
        type: 'object',
        properties: {
          text: { type: 'string', description: 'Text to copy to clipboard' },
        },
        required: ['text'],
      },
    },
    {
      name: 'lock_screen',
      description: 'Lock the Windows PC immediately.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_volume',
      description: 'Set the system output volume.',
      inputSchema: {
        type: 'object',
        properties: {
          level: { type: 'number', minimum: 0, maximum: 100, description: 'Volume level 0-100' },
        },
        required: ['level'],
      },
    },
    {
      name: 'list_processes',
      description: 'List running processes sorted by CPU usage.',
      inputSchema: {
        type: 'object',
        properties: {
          limit: { type: 'number', description: 'Max number of processes to return (default 10)' },
        },
      },
    },
    {
      name: 'kill_process',
      description: 'Kill a running process by name.',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Process name (e.g., "notepad", "chrome")' },
        },
        required: ['name'],
      },
    },
  ],
});

device.onCommand('take_screenshot', async () => {
  const filepath = join(tmpdir(), `screenshot-${Date.now()}.png`);
  ps(`
    Add-Type -AssemblyName System.Windows.Forms;
    Add-Type -AssemblyName System.Drawing;
    $screen = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds;
    $bitmap = New-Object System.Drawing.Bitmap($screen.Width, $screen.Height);
    $graphics = [System.Drawing.Graphics]::FromImage($bitmap);
    $graphics.CopyFromScreen($screen.Location, [System.Drawing.Point]::Empty, $screen.Size);
    $bitmap.Save('${filepath.replace(/\\/g, '\\\\')}');
    $graphics.Dispose();
    $bitmap.Dispose()
  `);
  const buffer = readFileSync(filepath);
  const url = await device.uploadFile(buffer, `screenshot-${Date.now()}.png`, 'image/png');
  unlinkSync(filepath);
  return { imageUrl: url, timestamp: new Date().toISOString() };
});

device.onCommand('send_notification', async (payload) => {
  const title = payload.title.replace(/'/g, "''");
  const message = payload.message.replace(/'/g, "''");
  try {
    ps(`New-BurntToastNotification -Text '${title}', '${message}'`);
  } catch {
    // Fallback if BurntToast is not installed
    ps(`
      [Windows.UI.Notifications.ToastNotificationManager, Windows.UI.Notifications, ContentType = WindowsRuntime] | Out-Null;
      $xml = '<toast><visual><binding template="ToastText02"><text id="1">${title}</text><text id="2">${message}</text></binding></visual></toast>';
      $toast = [Windows.UI.Notifications.ToastNotification]::new([Windows.Data.Xml.Dom.XmlDocument]::new());
      $toast.Content.LoadXml($xml);
      [Windows.UI.Notifications.ToastNotificationManager]::CreateToastNotifier('Lua Agent').Show($toast)
    `);
  }
  return { sent: true, title: payload.title, message: payload.message };
});

device.onCommand('open_url', async (payload) => {
  execSync(`start "" "${payload.url}"`);
  return { opened: true, url: payload.url };
});

device.onCommand('open_app', async (payload) => {
  ps(`Start-Process '${payload.name}'`);
  return { opened: true, app: payload.name };
});

device.onCommand('search_files', async (payload) => {
  const limit = payload.limit || 10;
  const searchPath = payload.path || '$env:USERPROFILE';
  const raw = ps(`
    Get-ChildItem -Path ${searchPath} -Recurse -Filter '${payload.query}' -ErrorAction SilentlyContinue |
    Select-Object -First ${limit} -ExpandProperty FullName
  `);
  const files = raw ? raw.split('\r\n').filter(Boolean) : [];
  return { query: payload.query, results: files, count: files.length };
});

device.onCommand('get_active_window', async () => {
  const title = ps(`
    Add-Type -Name Win32 -Namespace Native -MemberDefinition '[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow(); [DllImport("user32.dll", CharSet = CharSet.Auto)] public static extern int GetWindowText(IntPtr hWnd, System.Text.StringBuilder lpString, int nMaxCount);';
    $sb = New-Object System.Text.StringBuilder(256);
    [Native.Win32]::GetWindowText([Native.Win32]::GetForegroundWindow(), $sb, 256) | Out-Null;
    $sb.ToString()
  `);
  return { activeWindow: title };
});

device.onCommand('system_info', async () => {
  const info = ps(`
    $os = Get-CimInstance Win32_OperatingSystem;
    $cpu = (Get-CimInstance Win32_Processor).Name;
    $ramGB = [math]::Round($os.TotalVisibleMemorySize / 1MB, 1);
    $uptime = (Get-Date) - $os.LastBootUpTime;
    $battery = (Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue).EstimatedChargeRemaining;
    @{
      hostname = $env:COMPUTERNAME;
      os = $os.Caption;
      cpu = $cpu;
      memoryGB = $ramGB;
      uptime = '{0}d {1}h {2}m' -f $uptime.Days, $uptime.Hours, $uptime.Minutes;
      battery = if ($battery) { "$battery%" } else { 'N/A (desktop)' }
    } | ConvertTo-Json
  `);
  return JSON.parse(info);
});

device.onCommand('get_clipboard', async () => {
  const text = ps('Get-Clipboard');
  return { clipboard: text };
});

device.onCommand('set_clipboard', async (payload) => {
  const escaped = payload.text.replace(/'/g, "''");
  ps(`Set-Clipboard -Value '${escaped}'`);
  return { set: true, text: payload.text };
});

device.onCommand('lock_screen', async () => {
  execSync('rundll32.exe user32.dll,LockWorkStation');
  return { locked: true, timestamp: new Date().toISOString() };
});

device.onCommand('set_volume', async (payload) => {
  ps(`
    $wsh = New-Object -ComObject WScript.Shell;
    1..50 | ForEach-Object { $wsh.SendKeys([char]174) };
    $steps = [math]::Round(${payload.level} / 2);
    1..$steps | ForEach-Object { $wsh.SendKeys([char]175) }
  `);
  return { volume: payload.level };
});

device.onCommand('list_processes', async (payload) => {
  const limit = payload?.limit || 10;
  const raw = ps(`
    Get-Process | Sort-Object CPU -Descending |
    Select-Object -First ${limit} Name, Id, @{N='CPU_Seconds';E={[math]::Round($_.CPU,1)}}, @{N='Memory_MB';E={[math]::Round($_.WorkingSet64/1MB,1)}} |
    ConvertTo-Json
  `);
  return { processes: JSON.parse(raw) };
});

device.onCommand('kill_process', async (payload) => {
  const name = payload.name.replace(/'/g, "''");
  ps(`Stop-Process -Name '${name}' -Force -ErrorAction SilentlyContinue`);
  return { killed: true, process: payload.name };
});

async function main() {
  await device.connect();
  console.log('Windows controller online');
}

main().catch(console.error);

Agent Configuration

// src/index.ts
import { LuaAgent, LuaSkill } from 'lua-cli';

const windowsControlSkill = new LuaSkill({
  name: 'windows-controller',
  description: 'Control a Windows PC remotely',
  context: `
    You are a personal assistant that controls a Windows PC.
    You can take screenshots, open apps, search files, manage processes,
    control volume, and more.

    Device tools:
    - take_screenshot: Captures the screen and returns an image URL.
    - send_notification: Shows a Windows toast notification with a title and message.
    - open_url: Opens a URL in the default browser.
    - open_app: Launches an application by name or path.
    - search_files: Searches for files by name across user directories.
    - get_active_window: Reports the title of the currently focused window.
    - system_info: Returns hostname, OS, CPU, RAM, uptime, and battery status.
    - get_clipboard: Reads the current clipboard text.
    - set_clipboard: Sets the clipboard to the provided text.
    - lock_screen: Locks the PC immediately.
    - set_volume: Sets the system volume from 0 to 100.
    - list_processes: Lists running processes sorted by CPU usage.
    - kill_process: Kills a process by name. Use with caution.

    Guidelines:
    - When asked for a screenshot, take it and share the image URL
    - For file searches, show the full paths in the results
    - Confirm destructive actions (locking the screen, killing processes) before executing
    - Be conversational and helpful
  `,
  tools: [],
});

export const agent = new LuaAgent({
  name: 'windows-assistant',
  persona: `You are a personal assistant that controls a Windows PC. You can take
    screenshots, open apps, search files, manage processes, control volume,
    and automate tasks. Be helpful, concise, and confirm before taking
    potentially disruptive actions like locking the screen or killing processes.`,
  skills: [windowsControlSkill],
});

What You Can Ask

Here are real conversational examples you can send from WhatsApp or web chat:

Screenshots

“Take a screenshot and send it to me”

Open Apps

“Open Notepad” / “Launch Excel”

File Search

“Search for files called invoice.pdf”

Active Window

“What window do I have open right now?”

Lock Screen

“Lock my PC”

Volume

“Set the volume to 30%”

Processes

“What processes are using the most CPU?”

Kill Process

“Kill the Notepad process”

Clipboard

“Copy this text to my clipboard: Meeting at 3pm”

Notifications

“Send me a notification that says Stand up and stretch”

Browser

“Open github.com in my browser”

System Info

“How much RAM do I have? What’s my uptime?”
Security note: This device client gives your AI agent direct control over your computer. Only run it on machines you trust, and consider limiting which commands are registered based on your comfort level.

Next Steps

Mac Controller

Control a Mac the same way

Smart Office

Office automation with sensors and displays

CDN Uploads

How screenshot uploads work

Self-Describing Commands

How to write effective command definitions