-
In data centers, ML researchers’ vote for Nvidia is what made Nvidia a runaway success in AI training. On the embedded market, who holds the key for edge AI?
-
What’s at stake: Edge AI – or AIOT (Artificial Intelligence of Things) – on the embedded market has been a hot notion among MCU vendors for more than a few years. So, how’s it going? Will AI skate on every edge?
It’s coming, say edge AI proponents, but very slowly.
One reality that must be acknowledged is that edge AI, despite its hype, was never previously a fait accompli, nor is it today.
The reasons for edge AI’s prolonged gestation are many.
Most obvious is the severely constrained hardware currently deployed in embedded systems. Among the issues are processing power and memory size. Add to these the energy consumption of MCUs sitting in low-cost embedded systems, trying to run rapidly advancing AI algorithms. If this isn’t a classic case of square peg, round hole, what is?
Another issue is that the growth of the IoT market has been dogged by that fragmented hardware and software. Edge AI — or AIOT — poses no exception.
The foremost reason why edge AI has been so difficult is “engineering skill sets,” or the lack thereof, according to Tom Hackenberg, principal analyst, computing and software at Yole Intelligence.
The hardware challenges are big, but the bigger dilemma for the embedded community is a critical shortage of embedded system engineers trained in AI.
“At the embedded level, you have a huge intellectual property base, which is reused in the next iteration of a design,” said Hackenberg. “You have engineers used to working in either C++ or R or some platform they are skilled in. But if you tell them their next design is going to be in tinyML, that means that they will have to go back and relearn a whole new way of doing things.”
Hyperscalers have long recognized this. Engineering skills on their staffs are “pretty well balanced by now,” observed Hackenberg. But for embedded system companies, the journey has barely begun.
Ali Ors, global director, AI and Machine Learning strategies and technology at NXP Semiconductors, concurred. “A lot of companies – users of our portfolio of edge devices and microcontrollers – don’t really have a data science team. Many companies have a very strong background in embedded development, but they haven’t ramped up fully the [AI] skill set of their teams.”
Data centers vs embedded
Whether in training or inference, AI has emerged in two different worlds, in data centers and at the edge. Each world leverages the different hardware and software available in their markets.
Today, AI processors in data centers are a winner-takes-all market dominated by Nvidia.
Edge AI on the embedded market, however, is a “free for all,” Yole’s Hackenberg told us. In fact, while leading MCU vendors are offering hardware to address certain AI aspects, no single MCU leads the AI race in embedded.
The tipping point for Nvidia for AI training in data centers was its GPU hardware, because GPUs specialize in the matrix calculations that AI generally requires.
But the secret of Nvidia’s unassailable dominance in the data center market was ML researchers’ broad confidence in the Nvidia platform. They found Nvidia’s hardware easier to use and its software stack more mature, loaded with examples and documentation thanks to the proliferation of CUDA, Nvidia’s programming model.
Who has AI purchasing power?
As Peter Warden, CEO of Useful Sensors, wrote in his blog, the real purchasing power in choosing AI processors for data centers falls to ML researchers. Warden wrote, “They need to be kept happy, and one of the things they demand is to use the Nvidia platform. It’s what they know, they’re productive with it, picking up an alternative would take time and not result in skills the job market values.”
In short, ML researchers are expensive to hire and retain. So, their preferences get priority when purchasing hardware, he explained.
In contrast, who gets to choose which MCU or MPU to use on the edge AI market?
Answer: embedded system OEMs.
Thus far, however, embedded system OEMs are still on the fence. Yole’s Hackenberg explained that they have yet to decide about whether to use AI at the edge. If they do, they aren’t quite sure whether there’s any profit in AI.
OEM hesitation also stems from the possibility that an MPU that might exceed the price of an embedded device, and from a lack of engineering skills to implement AI. Other factors include the learning curve and the training cost of creating AI models.
Tools are the battleground
Given these factors, MCU vendors recognize that the edge AI battleground lies in AI and ML software development tools. At stake is how easily and efficiently OEMs can use them.
Because the implementation of edge AI in embedded isn’t solidified either in a single MCU or programming model like CUDA, leading MCU vendors are developing their own tools. “Without the tools, MCU vendors are selling a bill of goods that their customers can get neither the performance enhancements nor the efficiency they need,” noted Hackenberg. For now, expect “a lot of reinventing of the wheel” among MCU suppliers on edge AI, he added.
Against this backdrop, NXP Semiconductors recently unveiled two new software tools, the eIQ Time Series Studio and the GenAI Flow.
The eIQ Time Series addresses AI “born at the edge level” handling “sensor signals.” GenAI Flow provides building blocks for Large Language Models (LLMs) that power generative AI solutions, explained Ali Ors, director of AI and Machine Learning Strategies and Technology at NXP.
The eIQ Time Series Studio’s function is to streamline development and deployment of ML models across NXP’s MCU-class devices. Where applicable, GenAI Flow addresses NXP’s i.MX applications processors, noted Ors.
How do they stack up?
With edge AI, are there any big differences among MCU vendors’ hardware and software tools? Hackenberg told us, “Some are doing it better than others, simply because they’ve probably been investing in [AI] longer, and have had better visibility as to what they were going to need.”
(Image: Yole Intelligence)
In his opinion, “ST does a very good job.” Meanwhile, “This recent release from NXP looks to be quite extensive and looks to be geared towards the industrial equipment needs.”
Hackenberg added, “Silicon Labs has been doing edge AI for a little while now.” Renesas recently released its M85 based microcontroller. Hackenberg described it as “an AI accelerated microcontroller, which is the first true microcontroller with AI acceleration.” Renesas has tools enabling customers to work on AI, he added.
Referring to other leading MCU suppliers, NXP’s Ors acknowledged, “We’re all trying to achieve the same thing here … namely helping users deploy AI.”
While there are tools from other chip companies trying to cover the same ground, Ors emphasized that differences in maturity sometimes lead to differences in quality or ease of use of tools.
Ors claimed his team also looked at tools designed by third-party tool companies. “There too, we felt that we had something to offer the market.”
But a key reason NXP has “gone in house” with AI software development is that it is also building an AI accelerator, a neural processing unit dubbed “eIQ Neutron NPU” across the range of its MCUs and MPUs. Neutron NPU is billed as a highly scalable accelerator core architecture providing machine learning (ML) acceleration.
Scalable AI and Machine Learning Solutions (Image: NXP Semiconductors)
NXP’s AI software development tools are designed to work both for CPU cores — Cortex M cores used everywhere on the embedded MCU market — and for the unified hardware architecture that combines MCU and NPU.
NXP explained that eIQ Neutron NPUs support a range of neural network types such as CNN, RNN, TCN and Transformer networks. ML application development with the eIQ Neutron NPU is fully supported by an eIQ machine learning software development environment, NXP added.
Having a scalable accelerator core architecture like eIQ Neutron NPU that can work with processors ranging from application processors and crossover MCUs to traditional MCUs is particularly important, explained Ors. It allows NXP to support “a super dynamic [AI] space where types of frameworks, models and use cases constantly change.”
He said, “By owning both the hardware and the software, we can react better and for longer, because our products are going into automotive, industrial, medical, things that need to be available for a very long time, and they still need to be able to accept software changes and use case changes over that long period of time.”
Yole’s Hackenberg said the advantages of NXP’s AI offerings indeed lie in their ability to offer a wide frame of references, hardware and software, in a kind of a one-stop shopping experience. But he cautioned that “NXP isn’t exclusive.” ST and Renesas can do this, too. “And Infineon has kind of a unique approach in that it uses their PSoC microcontrollers, where you can configure, like an FPGA, part of the microcontroller to be either more I/O, or it can be an AI accelerator, or it could be a variety of things.”
In short, among leading MCU suppliers, nobody has a lock on edge AI.
Whenever embedded system OEMs take the plunge into a new AI platform, they will prefer edge AI solutions compatible with much of their corporate IP in their embedded systems. And they don’t want to retrain staff, Hackenberg noted. In short, the path of least resistance. “It is much more attractive for them to work with a partner they may have been working with for a long time,” said Hackenberg, instead of a nouvelle hardware architecture and tool set. They will seek partners who can offer a little hand-holding and training, “and a lot of support to say here we have both the hardware and the tools to make this an easy transition for you.”
Bottom line
While the world celebrates remarkable advancements in AI and Nvidia rules the roost on the data-center side, edge AI poses a whole different market dynamic and technical challenges for the embedded community.
Junko Yoshida
Former Editor in Chief, The Ojo-Yoshida Report
This article was published by the The Ojo-Yoshida Report. For more in-depth analysis, register today and get a free two-month all-access subscription.
