Innovation Heroes: On-device AI is going to improve your work life this year

Imagine a world where your device understands you so well that it prioritizes your work, streamlines your day, and operates with unparalleled efficiency—all without relying on the cloud. That world is no longer a futuristic dream; it’s the promise of on-device AI. This transformative technology is reshaping how businesses and consumers interact with the digital realm, offering unprecedented speed, security, and sustainability.

On this episode of Innovation Heroes, host Ed McNamara talks to Mike Belcher, Director of North American Commercial Channels at Qualcomm. Mike shares insights on this game-changing innovation and its potential to revolutionize industries.

The shift back to on-device AI

On-device AI is making a strong comeback as businesses and developers look for ways to overcome cloud computing’s inefficiencies. Historically, AI relied on on-device systems before the rise of cloud-based solutions. However, as the cost of running AI in the cloud soars, the industry is reevaluating its approach.

“One of the reasons why we have to move from cloud to endpoint devices is the cost of what it takes to drive AI up in the cloud,” explains Mike Belcher.

Belcher also warns that continuing the current trajectory of cloud-based AI could lead to economic and environmental challenges. “By 2033, we would actually hit a point where it would cost more than the GDP of the world to run the internet at the current growth rates,” he notes. These insights highlight the urgency of adopting more efficient, localized AI solutions.

Personalized experiences with on-device AI

One of the most exciting benefits of on-device AI is its ability to deliver deeply personalized experiences. Unlike cloud-based systems, on-device AI can continuously learn and adapt to individual user behaviors without requiring constant internet connectivity.

“The AI is actually running all the time and it’s learning you—how you write, how you communicate, all of those things,” says Belcher. “It’s designed to operate not being connected to the internet at all.”

This personalization has practical applications, such as with Microsoft Copilot+ PCs. These machines leverage on-device AI to streamline productivity tasks. For example, Belcher uses it to organize his email more effectively: “When I come back from a long trip, I ask it to find all the emails from the last 10 days that require a response and prioritize them. In seconds, I get a list of what I need to address immediately.”

Sustainability and efficiency: A winning combination

As concerns about energy consumption in data centers grow, on-device AI offers a sustainable alternative. Thanks to innovations in neural processing units (NPUs), devices can now efficiently perform high-powered AI computations.

“We can run 45 trillion operations per second on an NPU and do it all day running less than 300 milliwatts of power,” explains Belcher. “It’s about 150 times more efficient than running AI in the cloud.”

This efficiency not only reduces power consumption but also alleviates the strain on energy grids. Belcher adds, “If you have a very large data center in your neighborhood, you’ll notice they’re sucking power out of the grid to run and cool them. On-device AI helps address this challenge by keeping computations localized.”

Bridging the AI digital divide

As on-device AI becomes more mainstream, a new divide is emerging between businesses that adopt the technology and those that don’t. Belcher emphasizes the need for organizations to prepare for this shift or risk falling behind.

“If your business is trying to compete with others who are utilizing AI, you’re going to be at a disadvantage,” he says. “Start to dive into this now—understand the tools you use and ensure they will leverage AI.”

With predictions that 60% of all devices sold by 2026 will include advanced AI capabilities, the gap between early adopters and laggards will only widen. As Belcher puts it, “This will create a digital divide—but it will be an AI digital divide.”

On-device AI is not just a technological evolution; it’s a paradigm shift. From powering personalized user experiences to addressing sustainability concerns, this technology is poised to shape the future. As Belcher succinctly puts it, “On-device and endpoint AI is where we’re going to have to migrate the majority of the work that we do every day.”

The future of computing is here—and it’s on your device.

Listen to the full conversation here. You can also find episodes on SHI’s Resource Hub, Spotify, and other major podcast platforms, as well as on YouTube in video format.

Video + audio

Audio only