Drive LightBites from code.

Overview

LightBites listens on a port on your own Mac when you ask it to. You send one line of JSON, you get one line back. Three kinds of request cover everything a person can do in the app.

The controls layer reads the app’s own accessibility information, the same names VoiceOver reads. It works inside the app, so it sends no clicks or keystrokes to the system and needs no permission. The better a control is named for VoiceOver, the better it can be found here.

Quick start

Start the app with the port switched on, then talk to it.

1. Start LightBites with a port

LIGHTCRAFT_DEMO=1 LIGHTCRAFT_CONTROL_PORT=18731 \
  /Applications/LightBites.app/Contents/MacOS/LightBites

2. Send a request

echo '{"id":1,"method":"host.state"}' | nc -w 2 127.0.0.1 18731

The reply comes back as one line.

{"id":1,"ok":true,"result":{"active":21,"selectionCount":1,"visibleCount":24,"undo":null,"redo":null,…}}

3. Or from Python

import json, socket

def call(method, params=None, port=18731):
    with socket.create_connection(("127.0.0.1", port)) as s:
        s.sendall((json.dumps({"id": 1, "method": method, "params": params or {}}) + "\n").encode())
        return json.loads(s.makefile().readline())

print(call("ui.press", {"label": "4 stars"}))

Protocol

Plain text over a local TCP connection. One JSON object per line, in both directions.

{"id": 7, "method": "ui.press", "params": {"label": "4 stars"}}

{"id": 7, "ok": true, "result": {"label": "4 stars", "path": "0.0.2.3", "role": "AXButton"}}

• id is yours. It comes back unchanged, so you can match replies to requests. It can be a number or a string.
• method is the name of the request; params is an object, and may be left out.
• A connection can send many requests, one after another. Each is answered in turn.
• A request that takes time (an export, say) is answered when it is done, up to two minutes.

Errors

A failed request has "ok": false, a short code, and an error in plain words.

{"id": 8, "ok": false, "code": "ui", "error": "no element named “Nothing here”"}

Commands report their own codes, such as unknownCommand, disabled (the command can't run now) and badParams. Requests about controls use ui.

Commands

Every action on your photos is a named command, the same one the menu and the shortcuts run. This is the most exact way in: the parameters are plain values, and nothing depends on what is on screen.

{"method": "engine.execute", "params": {
  "command": "develop.set",
  "params": {"control": "light.exposure", "value": 0.8}
}}

Ask engine.commands what exists. Some you will use often: develop.set, develop.adjust and develop.get for the sliders, library.select for the selection, and edit.undo.

Controls

Anything you can click in a window can be found by name and used. This is for what commands don't cover: choosing a photo, an album or a tab, flipping a switch, filling in a field, or choosing a menu item.

{"method": "ui.tree", "params": {"filter": "stars"}}

→ {"count": 5, "unnamed": 0, "window": "All Photos", "controls": [
    {"label": "1 stars", "role": "AXButton", "path": "0.0.2.0", "enabled": true, "frame": [272, 969, 20, 24]},
    {"label": "2 stars", "role": "AXButton", "path": "0.0.2.1", "enabled": true, "frame": [293, 969, 20, 24]},
    …
  ]}

Screens and state

Move the app around, and check what it shows.

AI assistants (MCP)

An assistant that speaks the Model Context Protocol can use LightBites through lightcraft-cli, the command-line tool of the open-source LightCraft engine. It connects to the port you opened and turns its requests into tools.

Claude Code

claude mcp add lightbites -- lightcraft-cli mcp --connect 127.0.0.1:18731

Any MCP client

{
  "mcpServers": {
    "lightbites": {
      "command": "lightcraft-cli",
      "args": ["mcp", "--connect", "127.0.0.1:18731"]
    }
  }
}

The assistant gets tools to take a screenshot, list the controls on screen (list_widgets, with all and window), press one (click, by id or by label, with an optional action), set a slider or a field (set_value), choose a menu bar item (menu_item), press keys, type text, move between screens (set_ui), and run any command (run_command, get_develop, set_develop, query_photos, export and more). Start LightBites first; the connection is made on the first request, and again if the app restarts.

Scripts

To run a fixed list of requests without a connection, put them in a file, one per line, and point the app at it.

{"method": "ui.set", "params": {"screen": "grid"}}
{"method": "ui.press", "params": {"label": "Portfolio"}}
{"sleep": 500}
{"method": "ui.press", "params": {"label": "3 stars"}}
{"method": "ui.screenshot", "params": {"path": "portfolio.png"}}

LIGHTCRAFT_DEMO=1 LIGHTCRAFT_SCRIPT=/tmp/tour.jsonl \
  /Applications/LightBites.app/Contents/MacOS/LightBites
# replies are appended to /tmp/tour.out, one line per request

{"sleep": 500} waits half a second. A relative path in a request, for a screenshot, is next to the script.

Recipes

Short requests that do one thing. Each was run against the app.

{"method": "ui.press", "params": {"label": "4 stars"}}

{"method": "ui.press", "params": {"label": "Lagoon", "action": "confirm"}}

{"method": "ui.press", "params": {"label": "Portfolio"}}

{"method": "engine.execute", "params": {"command": "develop.set",
 "params": {"control": "light.exposure", "value": 0.8}}}

{"method": "ui.press", "params": {"label": "Exposure", "action": "increment"}}

{"method": "ui.screenshot", "params": {"path": "/tmp/lightbites.png"}}

Safety

• Off unless you ask. Nothing listens until LIGHTCRAFT_CONTROL_PORT is set.
• This computer only. The port is open on 127.0.0.1, not on your network. Other devices can't reach it.
• Web pages can't use it. A request that doesn't start with a JSON object closes the connection before anything runs, so a web page can't send commands to it.
• Any program you run can use it. While the port is open, a program on this Mac can do what you can do in the app. Open it only while you need it.
• No synthetic input. It never moves your pointer or presses keys at the system level, so it can't act on another app, or on whatever window is in front.

Limits

What it can't do yet.

• Mac only. The commands work on every platform; the controls layer is built for the Mac app. iPhone and iPad need a different approach.
• Pop-up menus. The items of a menu button, once it is open, aren't listed. Run what the item does as a command, or press Escape with ui.key. The menu bar itself works with ui.menuItem.
• Develop sliders. They appear with all: true, as plain elements, and show no current value. Read values with develop.get and set them with develop.set or develop.adjust.
• Icon-only buttons. A few still carry the name of their symbol (“Gear Shape”) until they get a proper label. Use ui.tree to see what they are called today.
• Paths change. A path from ui.tree is good while the screen holds still. Prefer label in anything you keep.