Skip to main content

Overview

Use case: Personal computer automation via WhatsApp or web chat Turn your Mac into an AI-controllable device. Ask your agent to take screenshots, open apps, search files, control music, manage your clipboard, and more — all from a chat message. Commands:
  • take_screenshot — Capture the screen and return the image
  • send_notification — Show a desktop notification
  • open_url — Open a URL in the default browser
  • open_app — Launch an application by name
  • search_files — Find files using Spotlight
  • get_active_app — Get the currently focused application
  • system_info — Hostname, CPU, memory, uptime, battery
  • get_clipboard — Read clipboard contents
  • set_clipboard — Write text to the clipboard
  • lock_screen — Lock the Mac
  • play_music — Play, pause, or skip tracks in Apple Music or Spotify
  • set_volume — Set the system volume (0—100)
  • run_shortcut — Run a Shortcuts.app shortcut by name
Prerequisites:
  • Node.js 18+ (for the Node.js client) or Python 3.10+ (for the Python client)
  • terminal-notifier for desktop notifications — install with brew install terminal-notifier
  • Screen recording permission granted to your terminal app (for screenshots)

Device Client

import { DeviceClient } from '@lua-ai-global/device-client';
import { execSync } from 'child_process';
import { readFileSync, unlinkSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';

const device = new DeviceClient({
  agentId: process.env.LUA_AGENT_ID!,
  apiKey: process.env.LUA_API_KEY!,
  deviceName: 'my-macbook',
  group: 'personal-devices',
  commands: [
    {
      name: 'take_screenshot',
      description: 'Capture a screenshot of the entire screen and return the image URL.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'send_notification',
      description: 'Show a macOS desktop notification.',
      inputSchema: {
        type: 'object',
        properties: {
          title: { type: 'string', description: 'Notification title' },
          message: { type: 'string', description: 'Notification body text' },
        },
        required: ['title', 'message'],
      },
    },
    {
      name: 'open_url',
      description: 'Open a URL in the default browser.',
      inputSchema: {
        type: 'object',
        properties: {
          url: { type: 'string', description: 'The URL to open' },
        },
        required: ['url'],
      },
    },
    {
      name: 'open_app',
      description: 'Launch an application by name (e.g., "Slack", "Safari", "Terminal").',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Application name' },
        },
        required: ['name'],
      },
    },
    {
      name: 'search_files',
      description: 'Search for files by name using Spotlight (mdfind).',
      inputSchema: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Filename or search term' },
          limit: { type: 'number', description: 'Max results (default 10)' },
        },
        required: ['query'],
      },
    },
    {
      name: 'get_active_app',
      description: 'Get the name of the currently focused application.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'system_info',
      description: 'Get system information: hostname, CPU, memory, uptime, and battery level.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'get_clipboard',
      description: 'Read the current clipboard text contents.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_clipboard',
      description: 'Set the clipboard text contents.',
      inputSchema: {
        type: 'object',
        properties: {
          text: { type: 'string', description: 'Text to copy to clipboard' },
        },
        required: ['text'],
      },
    },
    {
      name: 'lock_screen',
      description: 'Lock the Mac screen immediately.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'play_music',
      description: 'Control music playback: play, pause, or skip to next/previous track.',
      inputSchema: {
        type: 'object',
        properties: {
          action: {
            type: 'string',
            enum: ['play', 'pause', 'next', 'previous'],
            description: 'Playback action',
          },
          app: {
            type: 'string',
            enum: ['Music', 'Spotify'],
            description: 'Music app to control (default: Music)',
          },
        },
        required: ['action'],
      },
    },
    {
      name: 'set_volume',
      description: 'Set the system output volume.',
      inputSchema: {
        type: 'object',
        properties: {
          level: { type: 'number', minimum: 0, maximum: 100, description: 'Volume level 0-100' },
        },
        required: ['level'],
      },
    },
    {
      name: 'run_shortcut',
      description: 'Run a macOS Shortcuts.app shortcut by name.',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Shortcut name' },
        },
        required: ['name'],
      },
    },
  ],
});

device.onCommand('take_screenshot', async () => {
  const filepath = join(tmpdir(), `screenshot-${Date.now()}.png`);
  execSync(`screencapture -x ${filepath}`);
  const buffer = readFileSync(filepath);
  const url = await device.uploadFile(buffer, `screenshot-${Date.now()}.png`, 'image/png');
  unlinkSync(filepath);
  return { imageUrl: url, timestamp: new Date().toISOString() };
});

device.onCommand('send_notification', async (payload) => {
  const title = payload.title.replace(/"/g, '\\"');
  const message = payload.message.replace(/"/g, '\\"');
  execSync(`terminal-notifier -title "${title}" -message "${message}"`);
  return { sent: true, title: payload.title, message: payload.message };
});

device.onCommand('open_url', async (payload) => {
  execSync(`open "${payload.url}"`);
  return { opened: true, url: payload.url };
});

device.onCommand('open_app', async (payload) => {
  execSync(`open -a "${payload.name}"`);
  return { opened: true, app: payload.name };
});

device.onCommand('search_files', async (payload) => {
  const limit = payload.limit || 10;
  const raw = execSync(`mdfind -name "${payload.query}" | head -${limit}`).toString().trim();
  const files = raw ? raw.split('\n') : [];
  return { query: payload.query, results: files, count: files.length };
});

device.onCommand('get_active_app', async () => {
  const script = 'tell application "System Events" to get name of first application process whose frontmost is true';
  const name = execSync(`osascript -e '${script}'`).toString().trim();
  return { activeApp: name };
});

device.onCommand('system_info', async () => {
  const hostname = execSync('hostname').toString().trim();
  const cpu = execSync('sysctl -n machdep.cpu.brand_string').toString().trim();
  const memBytes = parseInt(execSync('sysctl -n hw.memsize').toString().trim());
  const memGB = Math.round(memBytes / 1073741824);
  const uptime = execSync('uptime').toString().trim();
  let battery = 'N/A';
  try {
    battery = execSync('pmset -g batt | grep -o "[0-9]*%"').toString().trim();
  } catch {}
  return { hostname, cpu, memoryGB: memGB, uptime, battery };
});

device.onCommand('get_clipboard', async () => {
  const text = execSync('osascript -e "the clipboard"').toString().trim();
  return { clipboard: text };
});

device.onCommand('set_clipboard', async (payload) => {
  const escaped = payload.text.replace(/"/g, '\\"');
  execSync(`osascript -e 'set the clipboard to "${escaped}"'`);
  return { set: true, text: payload.text };
});

device.onCommand('lock_screen', async () => {
  execSync('pmset displaysleepnow');
  return { locked: true, timestamp: new Date().toISOString() };
});

device.onCommand('play_music', async (payload) => {
  const app = payload.app || 'Music';
  const actionMap: Record<string, string> = {
    play: 'play', pause: 'pause', next: 'next track', previous: 'previous track',
  };
  const command = actionMap[payload.action];
  execSync(`osascript -e 'tell application "${app}" to ${command}'`);
  return { action: payload.action, app };
});

device.onCommand('set_volume', async (payload) => {
  const vol = Math.round((payload.level / 100) * 7);
  execSync(`osascript -e 'set volume output volume ${payload.level}'`);
  return { volume: payload.level };
});

device.onCommand('run_shortcut', async (payload) => {
  execSync(`shortcuts run "${payload.name}"`);
  return { ran: true, shortcut: payload.name };
});

async function main() {
  await device.connect();
  console.log('Mac controller online');
}

main().catch(console.error);

Agent Configuration

// src/index.ts
import { LuaAgent, LuaSkill } from 'lua-cli';

const macControlSkill = new LuaSkill({
  name: 'mac-controller',
  description: 'Control a MacBook remotely',
  context: `
    You are a personal assistant that controls a MacBook.
    You can take screenshots, open apps, search files, control music,
    manage clipboard, and more.

    Device tools:
    - take_screenshot: Captures the screen and returns an image URL.
    - send_notification: Shows a desktop notification with a title and message.
    - open_url: Opens a URL in the default browser.
    - open_app: Launches a Mac application by name.
    - search_files: Searches for files using Spotlight. Great for finding documents.
    - get_active_app: Reports which application is currently in the foreground.
    - system_info: Returns hostname, CPU, memory, uptime, and battery status.
    - get_clipboard: Reads the current clipboard text.
    - set_clipboard: Sets the clipboard to the provided text.
    - lock_screen: Locks the Mac immediately.
    - play_music: Controls Apple Music or Spotify (play, pause, next, previous).
    - set_volume: Sets the system volume from 0 to 100.
    - run_shortcut: Runs a Shortcuts.app shortcut by name.

    Guidelines:
    - When asked for a screenshot, take it and share the image URL
    - For file searches, show the full paths in the results
    - Confirm destructive actions (like locking the screen) before executing
    - Be conversational and helpful
  `,
  tools: [],
});

export const agent = new LuaAgent({
  name: 'mac-assistant',
  persona: `You are a personal assistant that controls a MacBook. You can take
    screenshots, open apps, search files, control music, manage the clipboard,
    and automate tasks. Be helpful, concise, and confirm before taking
    potentially disruptive actions like locking the screen.`,
  skills: [macControlSkill],
});

What You Can Ask

Here are real conversational examples you can send from WhatsApp or web chat:

Screenshots

“Take a screenshot and send it to me”

Open Apps

“Open Slack” / “Launch Safari”

File Search

“Search for files called invoice.pdf”

Active App

“What app am I currently using?”

Lock Screen

“Lock my Mac”

Volume

“Set the volume to 30%”

Music

“Play the next song” / “Pause the music”

Clipboard

“Copy this text to my clipboard: Meeting at 3pm”

System Info

“How much battery do I have left?”

Notifications

“Send me a notification that says Stand up and stretch”

Browser

“Open github.com in my browser”

Shortcuts

“Run my Focus Mode shortcut”
Security note: This device client gives your AI agent direct control over your computer. Only run it on machines you trust, and consider limiting which commands are registered based on your comfort level.

Next Steps

Windows Controller

Control a Windows PC the same way

Smart Office

Office automation with sensors and displays

CDN Uploads

How screenshot uploads work

Self-Describing Commands

How to write effective command definitions