Subscribe to Our Digest
The compute layer beneath modern artificial intelligence has become one of the most fiercely contested arenas in semiconductor history. Training a frontier large language model (LLM) now demands tens of thousands of accelerators operating in lockstep, while inference at scale punishes any architecture with insufficient memory bandwidth or interconnect throughput. The result is a rapidly diversifying landscape in which NVIDIA’s dominance is being challenged by AMD, Intel, Google, and AWS, each with a distinct philosophy on how AI silicon should be built. This article walks through the principal AI GPU and accelerator families on the market today, the architectural trade-offs that distinguish them, and the workloads each is best suited to handle.
AI accelerators have evolved a long way from the rasterization engines that gave them their name. Modern AI silicon is judged on four interlocking metrics: peak compute throughput at low precision (FP8, FP4, BF16), high-bandwidth memory (HBM) capacity, memory bandwidth, and chip-to-chip interconnect speed. Because LLM inference is memory-bound rather than compute-bound — the tensor cores can process tokens faster than memory can feed them — capacity and bandwidth often matter more than raw FLOPs. This explains why each generational leap focuses heavily on widening the memory pipe and growing on-package memory.
NVIDIA’s Hopper architecture, introduced in 2022, set the modern benchmark for AI training. The H100 SXM features 14,592 CUDA cores, 528 fourth-generation Tensor Cores, 80 GB of HBM3 memory, and 3.35 TB/s of memory bandwidth, delivering 3,958 TFLOPS of FP8 tensor throughput. NVLink provides 900 GB/s of GPU-to-GPU bandwidth, and PCIe Gen5 connects to host systems at 128 GB/s (Spheron).
The H200, released in late 2024, retains the same Hopper compute engine but doubles down on memory: 141 GB of HBM3e at 4.8 TB/s — a 76% capacity increase and 43% bandwidth improvement over the H100 (NVIDIA). Because LLM serving is memory-bound, this translates to roughly 45% higher tokens-per-second on Llama 2 70B in MLPerf testing, despite identical tensor core specifications. The H200’s 141 GB capacity is also enough to fit a 70-billion-parameter model in FP16 on a single GPU, eliminating tensor-parallelism overhead and halving GPU count for medium-sized inference deployments.
Blackwell, NVIDIA’s 2025 successor to Hopper, represents the largest generational jump the company has ever shipped. The B200 is a dual-chiplet design with 208 billion transistors, 192 GB of HBM3e memory, 8 TB/s of bandwidth, and a 1,000W TDP. It introduces fifth-generation Tensor Cores supporting FP4 precision — a first for production AI silicon — alongside FP6 and FP8. Combined, these advances deliver up to 9,000 TFLOPS of FP8 and 18,000 TFLOPS of FP4 throughput, with NVLink 5 providing 1.8 TB/s of GPU-to-GPU bandwidth (Clarifai).
The GB200, also known as Grace Blackwell, packages two Blackwell GPUs with an NVIDIA Grace CPU on a single board. The flagship rack-scale configuration, GB200 NVL72, links 72 GPUs into a single NVLink domain, enabling unprecedented parallelism for trillion-parameter model training. NVIDIA claims up to 4× faster training and 30× faster inference versus H100 for long-context workloads, with energy per inference token dropping from 12 J on Hopper to 0.4 J on Blackwell (NeevCloud). For organizations training frontier-scale models, the GB200 NVL72 has rapidly become the reference platform.
AMD has positioned its Instinct line as the principal challenger to NVIDIA in data center AI. The MI300X, launched in late 2023 on the CDNA 3 architecture, set an early memory benchmark with 192 GB of HBM3 and 5.3 TB/s of bandwidth — at the time, more than double the H100’s capacity. It delivers 1,307 TFLOPS of BF16/FP16 and 2,615 TFLOPS of FP8 performance (AMD). The MI325X, an incremental refresh, raised memory to 256 GB of HBM3e and bandwidth to 6 TB/s while maintaining the same compute envelope.
The MI350 series (MI350X and MI355X), built on the new CDNA 4 architecture, represents AMD’s direct answer to Blackwell. Each accelerator carries an industry-leading 288 GB of HBM3e and 8 TB/s of bandwidth, with expanded datatype support including MXFP6 and MXFP4 to match Blackwell’s low-precision capabilities (AMD). For memory-hungry workloads — fitting a 405-billion-parameter Llama model on fewer GPUs, or extending KV-cache to 128k-token contexts — AMD’s capacity advantage gives it a credible structural edge over NVIDIA’s flagship for certain inference scenarios.
Google’s Tensor Processing Units take a fundamentally different approach. Rather than general-purpose GPUs, TPUs are application-specific integrated circuits (ASICs) built around large systolic-array matrix-multiply units (MXUs) optimized exclusively for neural-network training and serving. The TPU v5p remains the workhorse for large-scale training in Google Cloud, with 8,960-chip pods linked by a high-bandwidth 3D torus interconnect.
Trillium (TPU v6e), Google’s sixth-generation TPU, delivers a 4.7× peak compute increase per chip versus v5e by enlarging MXU size and increasing clock speeds. Each v6e chip provides approximately 918 TFLOPS of BF16 throughput, 32 GB of HBM, and 1,640 GB/s of memory bandwidth, with doubled inter-chip interconnect bandwidth at 3,584 Gbps (Google Cloud). Pods scale to 256 chips in a 2D-torus topology, reaching 234.9 PFLOPs of BF16 compute and 102.4 TB/s of all-reduce bandwidth. Trillium is also Google’s most energy-efficient TPU, offering a 67% efficiency improvement over v5e — a meaningful consideration as data center power becomes the dominant constraint on hyperscale AI.
Intel’s Gaudi 3, manufactured on a 5-nanometer process, is positioned as a price-performance alternative to NVIDIA’s H100 for both training and inference. It contains 64 Tensor Processor Cores (TPCs) and 8 Matrix Multiplication Engines (MMEs), 128 GB of HBM2e memory, and 3.7 TB/s of bandwidth. Peak compute reaches 1,835 TFLOPS in FP8 and BF16 — a 4× improvement over Gaudi 2 (IONOS).
Gaudi 3’s principal architectural distinction is its embedded networking: 24 RDMA NIC ports of 200 GbE deliver 1.2 TB/s of bidirectional bandwidth per accelerator, allowing scale-out clusters to be built using standard Ethernet rather than proprietary interconnect fabrics. This Ethernet-native design simplifies data center integration and reduces vendor lock-in, making Gaudi 3 attractive for enterprises wary of NVIDIA’s proprietary stack (GIGABYTE).
AWS Trainium represents a different category again: custom silicon designed exclusively for the AWS cloud and exposed through Neuron SDK rather than CUDA. Trainium 2, the third generation of the family, packages eight NeuronCore-V3 cores delivering 1,299 FP8 TFLOPS and 667 BF16 TFLOPS per chip, paired with 96 GB of HBM3 at 2.9 TB/s. NeuronLink-V3 provides 1.28 TB/s of chip-to-chip bandwidth, and AWS’s Trn2 UltraServer architecture connects 64 Trainium2 chips across four instances into a single coherent training domain (AWS).
Trainium 3, AWS’s first 3-nanometer AI chip, doubles compute to 2.52 PFLOPs of FP8 and increases HBM3e capacity by 1.5× to 144 GB at 4.9 TB/s (AWS). Trainium’s value proposition is economic: AWS reports 30–40% better price-performance versus equivalent NVIDIA-based instances, making it compelling for workloads where AWS-native deployment is acceptable and CUDA dependency can be engineered away.
The table below summarizes headline specifications and intended use cases for each accelerator covered in this guide. Figures reflect publicly disclosed specifications from each vendor; rack-scale figures for the GB200 NVL72 represent the full 72-GPU domain.
Selection ultimately turns on three questions: software ecosystem, workload characteristics, and total cost of ownership.
The AI accelerator market has moved decisively beyond a single-vendor narrative. NVIDIA still leads on raw performance and software ecosystem maturity, but AMD’s memory-led strategy, Google’s purpose-built TPU economics, Intel’s Ethernet-native Gaudi 3, and AWS’s vertically integrated Trainium silicon each occupy defensible ground. For technical buyers, the right answer increasingly depends less on chasing peak FLOPs and more on matching architectural strengths to specific workload profiles, software dependencies, and deployment constraints. As frontier model sizes continue to scale and inference economics tighten, the diversity of this silicon landscape will only become more strategically important.