What happens when you try to run an AI agent - a model that doesn't just answer questions but takes actions, browses the web, and chains multiple steps together - entirely on your phone or laptop?
It falls apart, and the reasons are more fundamental than "we just need better chips."
The core constraint is memory, not processing speed. Agentic AI (where the model plans, executes, and adjusts across multiple steps) requires keeping a large context window active. That means holding the model's weights plus a growing conversation history plus tool outputs all in RAM simultaneously. A capable agent model needs 8-16GB of memory just for its own parameters. Add the working memory for a multi-step task, and you've consumed everything a typical laptop has, with nothing left for the actual operating system or apps the agent is trying to control.
Cloud-based agents like ChatGPT, Claude, and Gemini dodge this entirely. They run on clusters with hundreds of gigabytes of RAM per instance and can scale context windows without worrying about your local hardware.
There's also a latency problem that compounds with each step. An agent that needs 5 seconds per reasoning step on-device turns a 10-step task into a 50-second wait. The same task through a cloud API with optimized inference hardware might finish in 8 seconds total.
None of this means on-device AI is pointless. For single-turn tasks like transcription, text summarization, or quick lookups, local models work fine and offer genuine privacy benefits. But the promise of a fully local AI assistant that manages your calendar, drafts emails, and books flights - all without sending data to the cloud - is still years away from being practical on consumer hardware. The gap between what on-device models can do in a demo and what they can sustain in real daily use remains wide.