Subscribe to Our Digest
How hyperscalers are optimizing the entire system, from silicon to server racks, to fine-tune performance per watt for specific AI workloads.
By: Patrick Mannion, Data Centre Digest Contributing Editor
To maximize performance per watt, hyperscalers are optimizing the entire system, from silicon to server racks, for specific AI workloads. This is dramatically altering the integration and partnership equations. While data centers will continue to scale in compute performance, memory bandwidth, and power density, hyperscalers are closely evaluating silicon, packaging, interconnect, thermal, and rack-level design choices much earlier in the development cycle. This is especially true as heterogeneous computing moves from the board to a system on chip (SoC) or system in package (SiP).
Take the case of Wiwynn, a cloud IT infrastructure provider for data centers, which recently entered into a strategic partnership with ASIC developer Global Unichip Corp. (Hsinchu, Taiwan) to align silicon readiness with data center system architecture. Wiwynn aims to combine its expertise in rack-scale system integration, liquid cooling, and optical interconnects (Figure 1) with GUC’s know-how in SoC design and 2.5D/3D advanced packaging.
The two companies are collaborating to offer solutions that will help hyperscalers evaluate critical system-level tradeoffs earlier and streamline optical I/O to deliver the bandwidth and power efficiency required for next-generation AI systems. They argue that this silicon-to-system approach is essential if hyperscalers are to develop scalable, efficient, and serviceable AI infrastructure, in which hardware and software are optimized simultaneously.
In current data center environments, expensive compute components are locked into vertically integrated, proprietary architectures that are difficult to scale. These vendor-optimized architectures stitch together compute and memory building blocks using proprietary networking and software. However, each AI workload has different compute, memory bandwidth, and networking requirements.
Alternatively, a heterogeneous architecture built around the silicon-to-system approach allows developers to stitch together right-sized compute, memory, and networking components to build flexible systems. These open, heterogeneous systems can also be customized and fine-tuned for specific AI workloads.
It all starts with silicon. Custom silicon, built specifically for AI data centers, can help developers codesign and architect the system for performance per watt, whether it’s expanding the scale-up network, creating a coherent memory domain between CPU and accelerator, or optimizing die-to-die interfaces for high-bandwidth memory (HBM). Intel CEO Lip-Bu Tan, while inking a strategic collaboration with Google Cloud, noted that scaling AI requires more than accelerators. It requires balanced systems, he said, referring to a combination of CPUs and infrastructure processing units (IPUs) in a heterogeneous arrangement. In this combo, while CPUs handle orchestration, data processing, and system-level control, IPUs offload networking, storage, and security functions from CPUs. This architecture bolsters processing utilization and efficiency, while facilitating more predictable performance across hyperscale AI environments.
The IPU is a custom ASIC that combines programmable infrastructure acceleration, networking, storage, and security offload functions traditionally handled by CPUs. This improves compute capacity and allows AI data centers to scale more efficiently without increasing overall system complexity.
Meta, another leading hyperscaler, has a similar arrangement with Arm to develop a new class of CPUs to support growing AI workloads. Arm’s agentic AI processor is expected to realize faster performance per rack and higher power efficiency than traditional CPUs, and it will work alongside Meta’s custom Meta Training and Inference Accelerator (MTIA) silicon, for which Meta partnered with Broadcom.
These partnerships underscore the critical need to reduce rack-scale integration complexity and accelerate the transition from silicon-ready innovation to system-ready AI infrastructure. They also enable hyperscalers like Google Cloud and Meta to efficiently transition from silicon definition to deployment-ready AI infrastructure.
Beyond ASIC implementation to meet denser compute requirements, the partnerships give hyperscalers greater exposure to 2.5D/3D advanced packaging at a time when the package is becoming the new motherboard with chiplets, die-to-die interconnects, and HBM stacks. Then there are scale-up networks that push the limits of conventional electrical interconnects; in these networks, optical I/O solutions, such as those from Lumentum, deliver the bandwidth and power efficiency required for next-generation AI systems.
The next generation of AI infrastructure won’t be a one-size-fits-all arrangement; it’ll be optimized from the ground up as an open, modular compute platform for data centers. This will allow data centers to optimize the total cost of ownership (TCO) at the system level for hardware, software, power, and liquid cooling. On the hardware side, the silicon-to-system approach enables designers to assemble diverse compute, memory, and networking technologies and build open, customized, fine-tuned heterogeneous systems for AI workloads.
The critical challenge lies in the software atop these heterogeneous hardware building blocks. It must deftly orchestrate and ensure memory coherence, workload scheduling, fabric management, and resource pooling. While it remains a work in progress, a unified software stack (Figure 3) can unlock the potential of the silicon-to-system approach by abstracting the complexity of heterogeneous components. This will ensure that data centers can take advantage of this open and flexible architecture.
It’s interesting to note that data centers are facing this architectural crossroads at a time when the larger AI infrastructure buildout is moving toward inference and agentic AI. These landmark technologies can benefit from the scalable, heterogeneous systems discussed here. A highly tuned system design could help designers flexibly stitch inference workloads and agentic AI applications into open, scalable systems. For instance, the fine-tuned interconnectivity between compute platforms could help designers determine how much memory will be required and how much latency can be tolerated in inference workloads.
Efficiency and performance per watt in data centers start with silicon architecture, but they certainly don’t stop there. In the AI era, competitive advantage may increasingly depend on how effectively hyperscalers co-optimize silicon, packaging, interconnects, power delivery, cooling, and software as a unified system. These design imperatives require a comprehensive system-level approach that spans board-level innovation, advanced packaging, and rack-scale integration.