Mac Controller

Overview

Use case: Personal computer automation via WhatsApp or web chat Turn your Mac into an AI-controllable device. Ask your agent to take screenshots, open apps, search files, control music, manage your clipboard, and more — all from a chat message. Commands:

take_screenshot — Capture the screen and return the image
send_notification — Show a desktop notification
open_url — Open a URL in the default browser
open_app — Launch an application by name
search_files — Find files using Spotlight
get_active_app — Get the currently focused application
system_info — Hostname, CPU, memory, uptime, battery
get_clipboard — Read clipboard contents
set_clipboard — Write text to the clipboard
lock_screen — Lock the Mac
play_music — Play, pause, or skip tracks in Apple Music or Spotify
set_volume — Set the system volume (0—100)
run_shortcut — Run a Shortcuts.app shortcut by name

Prerequisites:

Node.js 18+ (for the Node.js client) or Python 3.10+ (for the Python client)
terminal-notifier for desktop notifications — install with brew install terminal-notifier
Screen recording permission granted to your terminal app (for screenshots)

Device Client

import { DeviceClient } from '@lua-ai-global/device-client';
import { execSync } from 'child_process';
import { readFileSync, unlinkSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';

const device = new DeviceClient({
  agentId: process.env.LUA_AGENT_ID!,
  apiKey: process.env.LUA_API_KEY!,
  deviceName: 'my-macbook',
  group: 'personal-devices',
  commands: [
    {
      name: 'take_screenshot',
      description: 'Capture a screenshot of the entire screen and return the image URL.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'send_notification',
      description: 'Show a macOS desktop notification.',
      inputSchema: {
        type: 'object',
        properties: {
          title: { type: 'string', description: 'Notification title' },
          message: { type: 'string', description: 'Notification body text' },
        },
        required: ['title', 'message'],
      },
    },
    {
      name: 'open_url',
      description: 'Open a URL in the default browser.',
      inputSchema: {
        type: 'object',
        properties: {
          url: { type: 'string', description: 'The URL to open' },
        },
        required: ['url'],
      },
    },
    {
      name: 'open_app',
      description: 'Launch an application by name (e.g., "Slack", "Safari", "Terminal").',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Application name' },
        },
        required: ['name'],
      },
    },
    {
      name: 'search_files',
      description: 'Search for files by name using Spotlight (mdfind).',
      inputSchema: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Filename or search term' },
          limit: { type: 'number', description: 'Max results (default 10)' },
        },
        required: ['query'],
      },
    },
    {
      name: 'get_active_app',
      description: 'Get the name of the currently focused application.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'system_info',
      description: 'Get system information: hostname, CPU, memory, uptime, and battery level.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'get_clipboard',
      description: 'Read the current clipboard text contents.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'set_clipboard',
      description: 'Set the clipboard text contents.',
      inputSchema: {
        type: 'object',
        properties: {
          text: { type: 'string', description: 'Text to copy to clipboard' },
        },
        required: ['text'],
      },
    },
    {
      name: 'lock_screen',
      description: 'Lock the Mac screen immediately.',
      inputSchema: { type: 'object', properties: {} },
    },
    {
      name: 'play_music',
      description: 'Control music playback: play, pause, or skip to next/previous track.',
      inputSchema: {
        type: 'object',
        properties: {
          action: {
            type: 'string',
            enum: ['play', 'pause', 'next', 'previous'],
            description: 'Playback action',
          },
          app: {
            type: 'string',
            enum: ['Music', 'Spotify'],
            description: 'Music app to control (default: Music)',
          },
        },
        required: ['action'],
      },
    },
    {
      name: 'set_volume',
      description: 'Set the system output volume.',
      inputSchema: {
        type: 'object',
        properties: {
          level: { type: 'number', minimum: 0, maximum: 100, description: 'Volume level 0-100' },
        },
        required: ['level'],
      },
    },
    {
      name: 'run_shortcut',
      description: 'Run a macOS Shortcuts.app shortcut by name.',
      inputSchema: {
        type: 'object',
        properties: {
          name: { type: 'string', description: 'Shortcut name' },
        },
        required: ['name'],
      },
    },
  ],
});

device.onCommand('take_screenshot', async () => {
  const filepath = join(tmpdir(), `screenshot-${Date.now()}.png`);
  execSync(`screencapture -x ${filepath}`);
  const buffer = readFileSync(filepath);
  const url = await device.uploadFile(buffer, `screenshot-${Date.now()}.png`, 'image/png');
  unlinkSync(filepath);
  return { imageUrl: url, timestamp: new Date().toISOString() };
});

device.onCommand('send_notification', async (payload) => {
  const title = payload.title.replace(/"/g, '\\"');
  const message = payload.message.replace(/"/g, '\\"');
  execSync(`terminal-notifier -title "${title}" -message "${message}"`);
  return { sent: true, title: payload.title, message: payload.message };
});

device.onCommand('open_url', async (payload) => {
  execSync(`open "${payload.url}"`);
  return { opened: true, url: payload.url };
});

device.onCommand('open_app', async (payload) => {
  execSync(`open -a "${payload.name}"`);
  return { opened: true, app: payload.name };
});

device.onCommand('search_files', async (payload) => {
  const limit = payload.limit || 10;
  const raw = execSync(`mdfind -name "${payload.query}" | head -${limit}`).toString().trim();
  const files = raw ? raw.split('\n') : [];
  return { query: payload.query, results: files, count: files.length };
});

device.onCommand('get_active_app', async () => {
  const script = 'tell application "System Events" to get name of first application process whose frontmost is true';
  const name = execSync(`osascript -e '${script}'`).toString().trim();
  return { activeApp: name };
});

device.onCommand('system_info', async () => {
  const hostname = execSync('hostname').toString().trim();
  const cpu = execSync('sysctl -n machdep.cpu.brand_string').toString().trim();
  const memBytes = parseInt(execSync('sysctl -n hw.memsize').toString().trim());
  const memGB = Math.round(memBytes / 1073741824);
  const uptime = execSync('uptime').toString().trim();
  let battery = 'N/A';
  try {
    battery = execSync('pmset -g batt | grep -o "[0-9]*%"').toString().trim();
  } catch {}
  return { hostname, cpu, memoryGB: memGB, uptime, battery };
});

device.onCommand('get_clipboard', async () => {
  const text = execSync('osascript -e "the clipboard"').toString().trim();
  return { clipboard: text };
});

device.onCommand('set_clipboard', async (payload) => {
  const escaped = payload.text.replace(/"/g, '\\"');
  execSync(`osascript -e 'set the clipboard to "${escaped}"'`);
  return { set: true, text: payload.text };
});

device.onCommand('lock_screen', async () => {
  execSync('pmset displaysleepnow');
  return { locked: true, timestamp: new Date().toISOString() };
});

device.onCommand('play_music', async (payload) => {
  const app = payload.app || 'Music';
  const actionMap: Record<string, string> = {
    play: 'play', pause: 'pause', next: 'next track', previous: 'previous track',
  };
  const command = actionMap[payload.action];
  execSync(`osascript -e 'tell application "${app}" to ${command}'`);
  return { action: payload.action, app };
});

device.onCommand('set_volume', async (payload) => {
  const vol = Math.round((payload.level / 100) * 7);
  execSync(`osascript -e 'set volume output volume ${payload.level}'`);
  return { volume: payload.level };
});

device.onCommand('run_shortcut', async (payload) => {
  execSync(`shortcuts run "${payload.name}"`);
  return { ran: true, shortcut: payload.name };
});

async function main() {
  await device.connect();
  console.log('Mac controller online');
}

main().catch(console.error);

import asyncio
import os
import subprocess
import tempfile
from datetime import datetime, timezone
from lua_device import DeviceClient, DeviceCommandDefinition

client = DeviceClient(
    agent_id=os.environ["LUA_AGENT_ID"],
    api_key=os.environ["LUA_API_KEY"],
    device_name="my-macbook",
    group="personal-devices",
    commands=[
        DeviceCommandDefinition(
            name="take_screenshot",
            description="Capture a screenshot of the entire screen and return the image URL.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="send_notification",
            description="Show a macOS desktop notification.",
            input_schema={
                "type": "object",
                "properties": {
                    "title": {"type": "string", "description": "Notification title"},
                    "message": {"type": "string", "description": "Notification body text"},
                },
                "required": ["title", "message"],
            },
        ),
        DeviceCommandDefinition(
            name="open_url",
            description="Open a URL in the default browser.",
            input_schema={
                "type": "object",
                "properties": {
                    "url": {"type": "string", "description": "The URL to open"},
                },
                "required": ["url"],
            },
        ),
        DeviceCommandDefinition(
            name="open_app",
            description="Launch an application by name (e.g., 'Slack', 'Safari', 'Terminal').",
            input_schema={
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Application name"},
                },
                "required": ["name"],
            },
        ),
        DeviceCommandDefinition(
            name="search_files",
            description="Search for files by name using Spotlight (mdfind).",
            input_schema={
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Filename or search term"},
                    "limit": {"type": "number", "description": "Max results (default 10)"},
                },
                "required": ["query"],
            },
        ),
        DeviceCommandDefinition(
            name="get_active_app",
            description="Get the name of the currently focused application.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="system_info",
            description="Get system information: hostname, CPU, memory, uptime, and battery level.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="get_clipboard",
            description="Read the current clipboard text contents.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="set_clipboard",
            description="Set the clipboard text contents.",
            input_schema={
                "type": "object",
                "properties": {
                    "text": {"type": "string", "description": "Text to copy to clipboard"},
                },
                "required": ["text"],
            },
        ),
        DeviceCommandDefinition(
            name="lock_screen",
            description="Lock the Mac screen immediately.",
            input_schema={"type": "object", "properties": {}},
        ),
        DeviceCommandDefinition(
            name="play_music",
            description="Control music playback: play, pause, or skip to next/previous track.",
            input_schema={
                "type": "object",
                "properties": {
                    "action": {
                        "type": "string",
                        "enum": ["play", "pause", "next", "previous"],
                        "description": "Playback action",
                    },
                    "app": {
                        "type": "string",
                        "enum": ["Music", "Spotify"],
                        "description": "Music app to control (default: Music)",
                    },
                },
                "required": ["action"],
            },
        ),
        DeviceCommandDefinition(
            name="set_volume",
            description="Set the system output volume.",
            input_schema={
                "type": "object",
                "properties": {
                    "level": {"type": "number", "minimum": 0, "maximum": 100, "description": "Volume level 0-100"},
                },
                "required": ["level"],
            },
        ),
        DeviceCommandDefinition(
            name="run_shortcut",
            description="Run a macOS Shortcuts.app shortcut by name.",
            input_schema={
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Shortcut name"},
                },
                "required": ["name"],
            },
        ),
    ],
)


@client.on_command("take_screenshot")
async def handle_take_screenshot(payload):
    filepath = os.path.join(tempfile.gettempdir(), f"screenshot-{int(datetime.now().timestamp())}.png")
    subprocess.run(["screencapture", "-x", filepath], check=True)
    with open(filepath, "rb") as f:
        buffer = f.read()
    url = await client.upload_file(buffer, f"screenshot-{int(datetime.now().timestamp())}.png", "image/png")
    os.unlink(filepath)
    return {"imageUrl": url, "timestamp": datetime.now(timezone.utc).isoformat()}


@client.on_command("send_notification")
async def handle_send_notification(payload):
    subprocess.run(
        ["terminal-notifier", "-title", payload["title"], "-message", payload["message"]],
        check=True,
    )
    return {"sent": True, "title": payload["title"], "message": payload["message"]}


@client.on_command("open_url")
async def handle_open_url(payload):
    subprocess.run(["open", payload["url"]], check=True)
    return {"opened": True, "url": payload["url"]}


@client.on_command("open_app")
async def handle_open_app(payload):
    subprocess.run(["open", "-a", payload["name"]], check=True)
    return {"opened": True, "app": payload["name"]}


@client.on_command("search_files")
async def handle_search_files(payload):
    limit = payload.get("limit", 10)
    result = subprocess.run(
        ["mdfind", "-name", payload["query"]],
        capture_output=True, text=True,
    )
    files = [f for f in result.stdout.strip().split("\n") if f][:limit]
    return {"query": payload["query"], "results": files, "count": len(files)}


@client.on_command("get_active_app")
async def handle_get_active_app(payload):
    script = 'tell application "System Events" to get name of first application process whose frontmost is true'
    result = subprocess.run(["osascript", "-e", script], capture_output=True, text=True)
    return {"activeApp": result.stdout.strip()}


@client.on_command("system_info")
async def handle_system_info(payload):
    hostname = subprocess.run(["hostname"], capture_output=True, text=True).stdout.strip()
    cpu = subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True).stdout.strip()
    mem_bytes = int(subprocess.run(["sysctl", "-n", "hw.memsize"], capture_output=True, text=True).stdout.strip())
    mem_gb = round(mem_bytes / 1073741824)
    uptime = subprocess.run(["uptime"], capture_output=True, text=True).stdout.strip()
    try:
        batt = subprocess.run("pmset -g batt | grep -o '[0-9]*%'", shell=True, capture_output=True, text=True).stdout.strip()
    except Exception:
        batt = "N/A"
    return {"hostname": hostname, "cpu": cpu, "memoryGB": mem_gb, "uptime": uptime, "battery": batt or "N/A"}


@client.on_command("get_clipboard")
async def handle_get_clipboard(payload):
    result = subprocess.run(["osascript", "-e", "the clipboard"], capture_output=True, text=True)
    return {"clipboard": result.stdout.strip()}


@client.on_command("set_clipboard")
async def handle_set_clipboard(payload):
    escaped = payload["text"].replace('"', '\\"')
    subprocess.run(["osascript", "-e", f'set the clipboard to "{escaped}"'], check=True)
    return {"set": True, "text": payload["text"]}


@client.on_command("lock_screen")
async def handle_lock_screen(payload):
    subprocess.run(["pmset", "displaysleepnow"], check=True)
    return {"locked": True, "timestamp": datetime.now(timezone.utc).isoformat()}


@client.on_command("play_music")
async def handle_play_music(payload):
    app = payload.get("app", "Music")
    action_map = {"play": "play", "pause": "pause", "next": "next track", "previous": "previous track"}
    command = action_map[payload["action"]]
    subprocess.run(["osascript", "-e", f'tell application "{app}" to {command}'], check=True)
    return {"action": payload["action"], "app": app}


@client.on_command("set_volume")
async def handle_set_volume(payload):
    subprocess.run(["osascript", "-e", f'set volume output volume {payload["level"]}'], check=True)
    return {"volume": payload["level"]}


@client.on_command("run_shortcut")
async def handle_run_shortcut(payload):
    subprocess.run(["shortcuts", "run", payload["name"]], check=True)
    return {"ran": True, "shortcut": payload["name"]}


async def main():
    await client.connect()
    print("Mac controller online")

asyncio.run(main())

Agent Configuration

// src/index.ts
import { LuaAgent, LuaSkill } from 'lua-cli';

const macControlSkill = new LuaSkill({
  name: 'mac-controller',
  description: 'Control a MacBook remotely',
  context: `
    You are a personal assistant that controls a MacBook.
    You can take screenshots, open apps, search files, control music,
    manage clipboard, and more.

    Device tools:
    - take_screenshot: Captures the screen and returns an image URL.
    - send_notification: Shows a desktop notification with a title and message.
    - open_url: Opens a URL in the default browser.
    - open_app: Launches a Mac application by name.
    - search_files: Searches for files using Spotlight. Great for finding documents.
    - get_active_app: Reports which application is currently in the foreground.
    - system_info: Returns hostname, CPU, memory, uptime, and battery status.
    - get_clipboard: Reads the current clipboard text.
    - set_clipboard: Sets the clipboard to the provided text.
    - lock_screen: Locks the Mac immediately.
    - play_music: Controls Apple Music or Spotify (play, pause, next, previous).
    - set_volume: Sets the system volume from 0 to 100.
    - run_shortcut: Runs a Shortcuts.app shortcut by name.

    Guidelines:
    - When asked for a screenshot, take it and share the image URL
    - For file searches, show the full paths in the results
    - Confirm destructive actions (like locking the screen) before executing
    - Be conversational and helpful
  `,
  tools: [],
});

export const agent = new LuaAgent({
  name: 'mac-assistant',
  persona: `You are a personal assistant that controls a MacBook. You can take
    screenshots, open apps, search files, control music, manage the clipboard,
    and automate tasks. Be helpful, concise, and confirm before taking
    potentially disruptive actions like locking the screen.`,
  skills: [macControlSkill],
});

What You Can Ask

Here are real conversational examples you can send from WhatsApp or web chat:

Screenshots

“Take a screenshot and send it to me”

Open Apps

“Open Slack” / “Launch Safari”

File Search

“Search for files called invoice.pdf”

Active App

“What app am I currently using?”

Lock Screen

“Lock my Mac”

Volume

“Set the volume to 30%”

Music

“Play the next song” / “Pause the music”

Clipboard

“Copy this text to my clipboard: Meeting at 3pm”

System Info

“How much battery do I have left?”

Notifications

“Send me a notification that says Stand up and stretch”

Browser

“Open github.com in my browser”

Shortcuts

“Run my Focus Mode shortcut”

Security note: This device client gives your AI agent direct control over your computer. Only run it on machines you trust, and consider limiting which commands are registered based on your comfort level.

Next Steps

Windows Controller

Control a Windows PC the same way

Smart Office

Office automation with sensors and displays

CDN Uploads

How screenshot uploads work

Self-Describing Commands

How to write effective command definitions

Getting Started

Device Client

Agent Integration

Examples

Advanced

CLI Commands

Overview

Device Client

Agent Configuration

What You Can Ask

Screenshots

Open Apps

File Search

Active App

Lock Screen

Volume

Music

Clipboard

System Info

Notifications

Browser

Shortcuts

Next Steps

Windows Controller

Smart Office

CDN Uploads

Self-Describing Commands

​Overview

​Device Client

​Agent Configuration

​What You Can Ask

Screenshots

Open Apps

File Search

Active App

Lock Screen

Volume

Music

Clipboard

System Info

Notifications

Browser

Shortcuts

​Next Steps

Windows Controller

Smart Office

CDN Uploads

Self-Describing Commands

Overview

Device Client

Agent Configuration

What You Can Ask

Next Steps