Subscribe to Our Digest
Artificial intelligence has changed data center behavior faster than almost any previous technology shift. Two years ago, most environments were still dominated by familiar patterns: web servers, transactional databases, virtual machines, and steady application hosting. Those workloads were predictable. Capacity planners could rely on fairly stable CPU use, storage growth, and network traffic.
AI workloads have blown up that predictability. Training and inference place very different stresses on compute, storage, networking, and power, and they do it at scales that legacy designs were never meant to handle. NVIDIA’s GTC 2026 keynote captured this shift with the language of “AI factories”: end‑to‑end systems built specifically to run fleets of intelligent agents around the clock.
By early 2026, inference has overtaken training as the main consumer of AI‑optimized infrastructure spending, accounting for well over half of total AI compute outlay and still climbing. That pivot from “train once, use occasionally” to “serve continuously, everywhere” changes how data centers are sized, funded, cooled, and operated. GTC 2026’s Vera Rubin platform, for example, is explicitly designed to prioritize high‑volume inference and agentic workloads, not just one‑off training runs.
At the rack level, the contrast is just as stark. Cutting‑edge training clusters can pull close to a megawatt per rack when running frontier models, forcing the use of dense GPU sleds and liquid cooling. Inference clusters usually run at 30–150 kW per rack — still far above traditional compute — but with tighter latency targets, different storage access patterns, and more bursty demand. Understanding these differences, and mapping them to platforms like Rubin GPU racks and Vera CPU racks unveiled at GTC 2026, is the key to investing in infrastructure that becomes a competitive asset instead of a very expensive dead end.
In practice, most AI environments are a mix of five workload families. Each has its own pattern of compute, storage, and network use.
Training is where models actually learn. Large neural networks ingest massive datasets, compute predictions, compare them to ground truth, and adjust millions or billions of parameters until performance stabilizes. A single state‑of‑the‑art run can cost millions of dollars in GPU time and tie up thousands of accelerators.
To support this, infrastructure has to deliver:
The Vera Rubin platform showcased at GTC 2026 packages these needs into reference designs: GPU‑dense NVL72 racks with 72 Rubin GPUs and 36 Vera CPUs, built for large‑model training and fine‑tuning at scale.
Fine‑tuning takes a pre‑trained foundation model and adapts it to a specific domain—law, medicine, customer support, internal knowledge bases, and so on. Each run is far cheaper than original pre‑training, often just a few percent of the compute, but the number of fine‑tunes explodes as more teams adopt AI.
Post‑training techniques and continuous refinement can, when aggregated, rival or exceed initial training costs. At GTC 2026, GPU vendors and cloud providers emphasized Rubin‑based clusters tuned for this middle ground: not full frontier training, but high‑throughput, cost‑sensitive fine‑tuning and instruction‑following work.
Inference is where models earn their keep. User prompts, images, sensor feeds, or transactions are passed to a trained model and turned into predictions in real time. Examples range from automatic photo tags and conversational assistants to fraud scores and warehouse robotics.
Analysts expect inference to dominate AI compute by the end of the decade, taking a large share of overall data center demand along the way. The growth is driven by always‑on, “agentic” systems that break problems into steps, call tools, and run continuously behind applications. GTC 2026 reinforced this with the launch of NeMoClaw and an Agent Toolkit aimed at enterprises that want fleets of trustworthy agents rather than isolated chatbots.
Inference looks nothing like training from an infrastructure perspective. Training tolerates long epochs and big batches. Inference needs sub‑second—or often sub‑100‑millisecond—responses, rapid autoscaling, and efficient per‑request cost. Rubin inference racks and Groq LPX accelerator racks announced at GTC 2026 are tuned for exactly this kind of high‑throughput, low‑latency serving.
Most real‑world systems chain together multiple steps: ingesting raw data, cleaning it, extracting features, running one or more models, and post‑processing the results. This “boring” part of the stack can quietly consume more compute than the model pass itself.
The same is true for training data prep: joining dozens of tables, handling missing values, transforming categorical fields, and splitting into training and validation sets can take serious horsepower at scale. Many GTC 2026 “AI factory” reference architectures and partner offerings focus heavily on GPU‑accelerated analytics and high‑throughput data pipelines for exactly this reason.
Kubernetes has become the default control plane for many AI platforms, providing a common way to package workloads, schedule them on clusters, and scale them automatically. GPU operators extend Kubernetes with awareness of accelerators, drivers, and networking so pods can claim the right resources and expose health metrics.
That said, running AI on Kubernetes is not a magic switch. Initializing nodes, pulling large container images, and wiring up drivers can add minutes of startup time—acceptable for month‑long training runs, but noticeable when teams are iterating rapidly. Several GTC 2026 blueprints and vendor demos addressed this with AI‑specific “factory” stacks that bundle orchestration, storage, and monitoring patterns out of the box.
Training modern foundation models on a single server is no longer realistic. Clusters now span many machines, each with multiple GPUs, all working together on the same job. Network design suddenly matters a lot; a poorly tuned fabric can leave expensive accelerators idle, cutting effective utilization in half.
When the pieces are put together correctly, the payoff is dramatic. A job that might have taken a month on a single node can finish in a couple of days on a well‑designed 100‑plus‑GPU cluster, letting data science teams iterate on model ideas far faster. Rubin NVL72 systems are NVIDIA’s attempt to turn that complexity into a repeatable building block that integrates networking, cooling, and management from day one.
To use all those GPUs, training frameworks combine data parallelism, model parallelism, and pipeline parallelism in different ways. Libraries such as Fully Sharded Data Parallelism (FSDP) can drastically cut memory use but may add runtime overhead, so architects have to balance time‑to‑results against hardware limits.
This only works if the network keeps up. Collective‑communication libraries and RDMA‑capable fabrics allow GPUs to communicate directly, instead of bouncing every message through CPUs. GTC 2026 extended this with Rubin‑class optical networking and DSX reference platforms that marry switches, DPUs, and optics specifically for AI‑factory traffic.
Inference platforms live and die on latency. If responses take too long, users abandon the experience and product teams turn models off. That reality affects every design choice—from which accelerators you pick, to where models are hosted, to how you batch requests.
Training favors large, flexible accelerators. Inference often runs best on more specialized parts tuned for lower‑precision operations and better performance per watt, including Rubin inference GPUs and Groq 3 LPUs.
Agent‑style systems make this even harder. Instead of one model call, a single user request might trigger a dozen or more chained inferences and data lookups. Each link adds latency, so architects have to squeeze delays out of every stage to stay within tight budgets.
If users are global but your inference cluster sits in one region, network round‑trip alone can blow your latency targets. The alternative is spreading model serving across regions and clouds, routing requests to the nearest healthy location and updating models in a controlled way. Many GTC 2026 partner announcements—across clouds and telcos—centered on exactly this kind of Rubin‑based, multi‑region footprint.
Inference demand is also notoriously spiky. Modern serving stacks (KServe, TensorFlow Serving, TorchServe, and managed services) spin up multiple replicas, balance requests, and add or remove capacity automatically as load shifts. Cloud providers at GTC 2026 highlighted GPU autoscaling and agent‑aware routing as key pieces of keeping inference costs in line while still meeting aggressive response‑time goals.
AI is as much a storage problem as a compute problem. Training jobs read huge volumes of data continuously; if storage can’t keep up, GPUs sit idle waiting for input. High‑bandwidth file systems and carefully sharded object stores have become essential in large training environments.
Inference has its own data needs. Models must load quickly when new versions go live, and features have to be fetched from databases or feature stores with millisecond‑level access times. GTC 2026 even pushed this thinking beyond Earth, with Rubin‑based “Space‑1” modules aimed at orbital data centers that process satellite streams in place.
Long‑running training campaigns will almost certainly hit hardware faults. Without good checkpointing, that can mean throwing away days of work and a lot of money. Resilient systems regularly save model state so runs can resume from a recent point rather than starting over, using asynchronous and differential techniques to reduce overhead.
The center of gravity for AI infrastructure is shifting from rare, enormous training runs to continuous, large‑scale inference. Training will always matter — it is how new capabilities are created — but value is increasingly realized through systems that run 24×7, embedded deep into business processes.
GTC 2026 drove this home. Rubin platforms, Vera CPUs tuned for agentic AI, Groq LPX inference racks, NeMoClaw agent runtimes, and DGX Station/Spark “desktop data centers” all point in the same direction: AI infrastructure is becoming an industrial system, not a side project.
For CIOs and data‑center architects, the takeaway is clear. AI workloads are not a minor variation on traditional compute. They are a distinct category that demands purpose‑built solutions—at the chip level, the rack level, and the software‑stack level. Organizations that align their designs with this reality now will be the ones that turn AI from an exciting experiment into a durable competitive advantage.