2026
LLMs & Agents, Apps & Tools
MochiOn-Device AI Desktop Agent for macOS
A private AI assistant that sees and operates a Mac the way a person does. It clicks, types, browses and talks back, using a small local model kept accurate by engineering rather than scale.
In numbers
- 1/7 → 13/17
- real tasks completed, checked against ground truth
- 1.5–3.5×
- faster tasks after profiling every model turn
- 48 → 76 tok/s
- generation speed on an Apple M4 (Q4 + flash attention)
Screenshots
click any figure to enlarge ↗
What I did
- 1.
Built an AI agent for macOS in Swift that runs Liquid AI's LFM3-1.7B locally on llama.cpp (Metal) and operates real apps through 38 tools: the macOS Accessibility API, real mouse and keyboard input, screen capture with on-device OCR, and the Chrome DevTools Protocol for the web.
- 2.
Paired the text model with a small on-device vision model (LFM3-VL-2B), using numbered on-screen markers and grid zoom instead of asking it for pixel coordinates, and added code-level checks that fix the small model's tool mistakes before they run. Together these raised real-task success from 1 of 7 to 13 of 17 on a ground-truth benchmark.
- 3.
Made tasks 1.5–3.5× faster by profiling each model turn: the model now only "thinks" on its first move, after an error and for the final answer; switching to Q4_K_M with flash attention took generation from 48 to 76 tokens/s on an Apple M4; and table-aware page reading with line filtering cut ~10k-token web pages down to a few hundred tokens.
- 4.
Designed a tiered permission system (look / act / change / risky / blocked) with app-specific rules for trading platforms, downloads, document edits and macOS permission prompts, so the agent never deletes, sends, buys or edits anything without consent. Every action is logged for audit.
- 5.
Shipped a hands-free voice interface with on-device speech: a "Hey Mochi" wake word, interrupting it mid-answer (with echo rejection), and natural dismissals and follow-ups. I also added a cloud planner that breaks multi-step requests into checkable steps and re-plans when one fails.
- 6.
Backed it with 30 unit tests and an unattended benchmark harness that scores each run against ground truth (the real page, the app's state), rather than trusting the model's own "done!".
Notes
Mochi started as "use a small local model to build my own desktop agent with a cute avatar" and grew one feature at a time, each tested on the real Mac. The interesting engineering is in making a 1.7B model reliable enough to drive a computer.
What kept a small model on track
- –Thinking only where it pays off: on the first move, after an error and before answering. Routine actions skip it (about 2 s each instead of 13 s).
- –Fixing slips in code, not prompts: browser clicks aimed at Mac apps are rerouted, shortened element ids are repaired, and repeated actions trigger a "try something else" note.
- –Context pruning keeps the prompt near 55% of the 32k window; empty answers are retried without thinking.
- –Pages are read one table row per line with a find filter, about 600 characters instead of a 10,000-token page.
Lessons learned
- –Check against ground truth, not the model's reply. Several "done!" answers were wrong until window titles and the real page were checked.
- –Measure the GPU, not just the CPU. A transparent full-screen overlay looked free but cut generation from 45 to 3.4 tokens/s.
- –Read before you look. OCR reads a ticker perfectly where a 2B vision model guesses.
You can always see who's driving: a pink pointer, a thin glowing border, a status pill with Stop, and a companion that narrates each step. Touch the mouse and Mochi stops.
Built with
Swift · SwiftUI / AppKit · llama.cpp · LFM3-1.7B · LFM3-VL-2B · Accessibility API · Chrome DevTools Protocol · OpenRouter
Next project
AskDocs →