Voice AI for Mobile Apps: Companions, Tutors, Fitness Coaches and More

For developers, the interesting question is becoming less "Can I add voice AI?" and more "Where would voice actually make my app better?"

Here are a few places where we think the answer is pretty obvious.

Voice AI for fitness apps

Fitness is a great fit for voice because looking at and touching your phone is often exactly what you don't want to be doing.

Imagine talking to a coach while running or working out:

"What's my next exercise?"

"Give me another 30 seconds."

"That was too heavy. Drop the next set by ten pounds."

The AI can answer while also calling functions inside the app to change timers, log workouts, modify exercises, or pull up information.

That makes the app feel less like something you operate between sets, and more like a coach that's actually there with you.

It's also a good example of where hybrid Voice AI gets interesting. Something like "start my timer" probably doesn't need a frontier model running in the cloud. That can potentially happen locally, while more complicated questions get routed to a cloud model.

Voice AI for language-learning apps

Language learning might be an even more obvious use case. If you're trying to learn a language, you need to actually speak it.

Instead of recording a sentence, submitting it, and waiting for feedback, you can just have a conversation with an AI tutor.

That also means dealing properly with the messy parts of human conversation:

Tutor: What did you do this weekend?
User: Je suis allé au... uh...

The tutor shouldn't immediately assume you're finished because you paused for half a second. Especially when you're learning a language, pauses, corrections, and half-finished thoughts are part of the experience.

And because the AI can call into the rest of the app, the conversation doesn't have to live in a vacuum. It can save vocabulary, show corrections, adjust the difficulty, or move you into another lesson.

At that point, voice isn't really an add-on anymore. It's part of the interface.

Voice AI for companions and characters

Companion apps, virtual characters, and conversational games are a little different because the conversation itself is often the product.

Here, latency and turn-taking matter a lot.

If the AI is halfway through a long response, you should be able to say:

"Wait, that's not what I meant."

and have it stop and listen.

That sounds simple, but getting interruptions, echo cancellation, turn detection, and mobile audio working well together is a surprisingly large part of making a voice agent actually feel natural.

That's one of the reasons we built our open-source OpenAI Realtime Toolkit for React Native. It handles a lot of that plumbing around OpenAI's Realtime API so developers can spend more time on the actual experience.

The same problems come up with AI NPCs, kids' characters, interactive stories, meditation coaches, and plenty of other conversational apps.

Voice as a hands-free interface

There's also a much bigger category that doesn't necessarily need an AI personality at all.

Cooking apps. Navigation. Field-service software. Travel apps. Automotive apps. Accessibility tools. Productivity apps.

Basically, anything you might want to use while your hands or eyes are busy.

Instead of tapping through menus, you can just say:

"What's next?"
"Read that again."
"Skip this step."
"Show me the photo."
"Add that to my list."

The LLM handles the language, and tool calls connect that conversation to what the app can actually do.

It's a pretty natural extension of the interfaces these apps already have.

Not everything needs to go to the cloud

This is probably the part we're most interested in.

Developers increasingly don't have to choose between cloud AI and on-device AI. You can mix the two.

Phones are now capable of running speech recognition, voice activity detection, text-to-speech, and increasingly capable small language models locally.

So you can keep fast, frequent, or predictable interactions on the device, and use bigger cloud models when they're actually useful.

A fitness app might handle:

"Start my timer."

entirely on-device, while sending:

"Can you change today's workout because my legs are still sore from Monday?"

to a cloud LLM.

A language tutor might do speech recognition and text-to-speech locally while using a cloud model for the conversation itself.

A companion might use local processing to figure out when you've started and stopped talking, while a realtime cloud model handles the character.

That's what we mean by hybrid Voice AI. It's not really about replacing the cloud. It's about not sending everything to the cloud just because that's where the LLM happens to live.

That can mean lower latency, lower inference costs, better offline behavior, and more control over what data leaves the device.

A few tools for building this

We've open-sourced a couple of the pieces we've been using to make these kinds of apps easier to build.

EdgeSpeech

Gives React Native developers a simple interface for running speech processing directly on the device.

OpenAI Realtime Toolkit

Handles the mobile audio and conversational plumbing around OpenAI Realtime, including things like echo cancellation, turn detection, interruptions, and tool calling.

Switchboard

Is the broader framework we've built for connecting all of this together: real-time audio graphs where on-device components and cloud AI services can be mixed and matched.

Fitness coaches, tutors, companions, and characters are some of the obvious applications.

But the more interesting question is probably: what apps become possible when talking to software starts to feel as natural as tapping it?