This market research report was originally published at the Yole Group’s website. It is reprinted here with the permission of the Yole Group.
Power and processor-hungry AI set to drive next stage of chip development – Generative artificial intelligence (AI) is redefining the value of work, and its exponential need for performance is driving demand for power consumption and processing capacity in ways that it is not yet clear how this will be met.
Tools based on generative AI have moved beyond niche technical applications to consumer use, changing the way images, video, music, and other content are produced. The impact is expected to reach almost every professional role, including lawyers, architects, doctors and more.
The AI server market is expected to expand at a compound annual growth rate of 36% from 2023 to 2029, outpacing overall data center growth.
Jérôme Azémar, Sales Force Effectiveness Director at Yole Group, recently attended the International Semiconductor Executive Summit (ISES) Taiwan and among the key themes that emerged are the evolution in semiconductor technologies to meet demand.
This article is based on the following Yole Group’s products: Generative AI 2024 – Impact on Processors, Memory, Advanced Packaging, and Substrates – Next-Generation DRAM 2024 – Focus on HBM and 3D DRAM…
Increasing processor capacity driving power demand
The massive growth that the data center GPU and AI ASIC market experienced in 2023 — climbing by 167% year on year, is expected to continue in 2024, before stabilizing in 2025.
Yole Group expects this stabilization as the number of companies able to buy GPUs and AI ASICs on a large scale is limited, and as the lifecycle of these components is also growing on average. The total market is expected to reach more than $230 billion by 2029 from more than $150 billion in 2025, representing a CAGR of 29%.
The explosion in the total number of FLOPs used to train AI models over the past decade has driven up the cost into the hundreds of millions of dollars and power consumption from a few kilowatts into megawatts.
Companies are now focusing on increasing power efficiency, as there is currently not enough power generation on the grid to support the rapid pace of growth.
How will HBM4 evolve?
Generative AI and high-performance computing (HPC) are fueling growth in data center dynamic random-access memory (DRAM) demand.
Yole Group predicts that high-bandwidth memory (HBM) bit shipments will rise by around 48% between 2023 and 2029, with HBM’s share of the DRAM market rising above 6% from around 2% in 2023.
A Yole Group’s teardown shows that NVIDIA has introduced an innovative design in its Hopper Tensor Core H100 GPU that uses HBM3 offering twice as much DRAM bandwidth as its predecessor, the A100. The GPU is paired with the Grace CPU using NVIDIA’s ultra-fast chip-to-chip interconnect, delivering 900GB/s of bandwidth, to provide up to 10 times higher performance for applications running terabytes of data for the HPC, AI, and gaming markets.
HBM is in the hands of three main players – SK Hynix, Samsung and Micron. SK Hynix is likely to be the first to introduce HBM4 in 2026, but it remains to be seen if hybrid bonding will be used . Next-generation hybrid bonding enables the use of smaller connections for more dense stacks.
“The need for HBM4 is clear, but it is still unclear whether hybrid bonding will be used to achieve the manufacturing. It will be dependent on entry barrier cost and if yields are high enough..”
Jérôme Azémar
Sales Force Effectiveness Director, Yole Group
Packaging trends towards glass substrates and chiplets
The current average substrate size of AI accelerators is around 70*70 mm² and the next-generation products are expected to scale to larger sizes, but this trend starts to be limited at 100*100 mm². Beyond this limit, the yield drops drastically.
Not all the current AI accelerators are using the maximum layer count, but next-generation products are expected to quickly adopt higher layer count to face the signal routing and power delivery challenges in parallel with an increased form factor.
One approach to increasing scalability could take the advanced substrates to the next level by adopting glass as a new core material. Glass core substrates can offer smaller L/S, fewer layers, and superior thermal conductivity, making them a promising candidate to resolve thermal management issues.
Yole Group expects to see the chiplet approach take hold in the future as manufacturers look to optimize cost against performance. Chiplets enable disaggregation so that the system-on-chip (SoC) monolithic die is partitioned into smaller chips with different functions and then interconnected in the same package. Chiplets also allow for duplication so that two or more SoC monolithic dies can interconnected in the same package to form a larger SoC.
“This allows designers to place most advanced functions requiring the smallest nodes in small dedicated chips, rather than having a large chip requiring the smallest node to handle all functions. If you can have some of the functions handled by other chips with less advanced nodes, then total cost would be lower.. So, it is becoming a plug-and-play approach where you need to have the optimum case for each and it’s a matter of balance between the space you want to take up, the cost of the chip, the performance you want to obtain and how you pack it.”
Jérôme Azémar
Sales Force Effectiveness Director, Yole Group
Taiwan’s central role in generative AI
While Taiwan does not have large chipmaking IDMs in its territory, the country plays a strong role in the AI ecosystem. It serves the likes of AMD and NVIDIA and has key players in processor foundries and packaging integration.
With the presence of TSMC, Taiwan is home to the main contributing foundry for generative AI. Taiwan is also essential in packaging, with TSMC being global leader and other players such as ASE and PTI well positioned for other chip packaging for data centers. Leading players in substrates involved in AI such as Unimicron) are also present in Taiwan.
The HBM market is dominated by Samsung, SK Hynix and Micron, but Taiwan has other important players including Nanya and Winbond, which are involved in supplying other types of memory for data centers. At the server level, Taiwan has strong companies too such as Quanta Computer.
Keep following as Yole Group continues to track the impact of generative AI on semiconductor technologies.
