The Future of AI Is On-Device

Most of the AI conversation still revolves around the cloud: which company has the best model, who has the most GPUs, and how much compute the next generation of models will require.

But a parallel shift is happening that may matter just as much for application developers: AI inference is moving onto the devices we already own.

Phones. Laptops. Desktop computers. Smart speakers. Cars. Wearables. Eventually, almost anything with enough compute and memory.

This doesn't mean frontier models or cloud AI are going away. It means we probably won't need to send every AI task to them.

Models are getting small enough to live everywhere

The capabilities available in relatively small models have improved enormously.

Google's Gemma family is explicitly being designed for mobile and edge hardware. Gemma 3n, for example, can process text, images, video and audio while operating with memory requirements comparable to a 2B or 4B parameter model. Google describes it as being designed specifically for phones, tablets and laptops.

Apple now gives developers direct access to the foundation model running on Apple Intelligence devices. Its on-device models support things like summarization, extraction, image understanding, structured output and tool calling without requiring every request to leave the device.

Microsoft has been embedding small models such as Phi-4-mini directly into Edge, while Meta has released 1B and 3B Llama models specifically designed to fit on mobile and edge devices.

And the list is growing quickly.

Some representative examples:

• Gemma 3n / Gemma models: multimodal understanding, transcription, translation, reasoning and general application intelligence on phones, tablets and laptops.

• Llama 3.2 1B/3B: summarization, rewriting, instruction following and other lightweight language tasks on phones and edge hardware.

• Qwen3 0.6B and other small Qwen variants: very compact language models capable of instruction following, multilingual processing, reasoning and tool-oriented workflows; small enough to make local deployment practical across increasingly modest hardware.

• Whisper Tiny/Base/Small: speech recognition that can already run completely offline on iPhones, Android devices, Raspberry Pis, Macs and PCs.

• Task-specific models: VAD, noise suppression, wake-word detection, speaker identification, embeddings, translation and other narrowly focused models can run on considerably more constrained devices, including embedded systems and wearables.

That last category is especially important.

On-device AI doesn't necessarily mean putting one general-purpose LLM everywhere. Often the better architecture is several small models, each doing one job extremely well.

Why send everything to the cloud?

If an application can determine whether someone is speaking locally, there isn't much reason to stream silence to a server.

If it can transcribe speech locally, perhaps only the resulting text needs to reach a larger model.

If a small local model can classify a request, summarize some text, understand a command or call a function, perhaps the cloud model doesn't need to be involved at all.

The advantages compound.

Latency improves because there is no network round trip.

Costs fall because fewer tokens, audio streams and API requests need to be processed in the cloud.

Privacy improves because sensitive data can stay on the user's device.

Reliability improves because some features continue working with a bad connection—or no connection at all.

And scaling gets easier. Ten million devices performing inference themselves is very different infrastructure from ten million devices continuously asking your servers to do it for them.

The interesting architecture is hybrid

None of this means every application should download the largest model that can physically fit on a phone.

The cloud remains incredibly useful. Larger models will continue to be better at difficult reasoning, broad knowledge, large contexts and tasks where additional compute actually improves the result.

The more interesting architecture is therefore:
local first, cloud when necessary.

A wearable might handle wake words, audio preprocessing and simple intents locally, then hand a complex request to a phone.

The phone might run speech recognition and a small language model locally, but route difficult reasoning to a frontier model.

A laptop or desktop can push that boundary considerably further, running much larger models entirely locally.

A smart speaker might locally handle the majority of common commands while calling the cloud for the long tail.

Instead of choosing one model, developers increasingly get to choose where each piece of intelligence belongs.

This is where we've been positioning Switchboard

This shift is also a big part of how we've been building our Switchboard Voice AI product line.

Voice is particularly well suited to hybrid architectures because there are so many pieces in the pipeline: voice activity detection, echo cancellation, noise suppression, speech recognition, turn detection, speaker processing, language models and text-to-speech.

Some of those pieces should almost always happen locally. Others increasingly can happen locally. And some still make sense in the cloud.

Our goal is to make those decisions modular.

Today, our tooling already supports architectures where audio processing, VAD, STT and other components run on-device, while cloud models are used only where necessary. We're extending that approach toward complete local and hybrid voice-agent pipelines across mobile, desktop, embedded devices, smart speakers and wearables.

We've been betting for some time that AI applications won't ultimately be built around a single giant API call.

They'll be built from a combination of models running in different places, selected according to the task, available hardware, latency requirements, privacy requirements and cost.

And as small models keep improving, the portion running on the device is only going to get bigger.