Enterprise IT infrastructure isn’t built for AI inference
AI inference requires fundamentally new, distributed enterprise infrastructure
by https://www.techradar.com/author/michael-reid · TechRadarOpinion By Michael Reid Published 13 October 2026
Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter
As global AI investment continues to climb — it surpassed $300 billion in 2025, according to IDC — enterprise IT remains primarily focused on supporting the development of training models, with architecture such as large GPU clusters and high-density compute taking center stage.
But now AI inference is becoming the dominant workload, and the architecture that supports models in production, at scale, and to real users and systems has quickly become the future of AI.
The demand for AI inference is there — according to Deloitte, inference already accounts for roughly two-thirds of compute, while McKinsey projects that it will represent between 30% and 40% of total data center demand by 2030 — however, the dedicated infrastructure is not.
Latest Videos FromTechRadarWatch full video here: Michael Reid
CEO of Megaport.
Inference is more than a scaled-down training environment
What training and inference require from an IT point of view are very nearly polar opposites. Enterprises have facilitated AI training via data centers with massive, centralized data GPU clusters for extreme power density and specialized, high-speed interconnects meant to maximize communication within a single room.
Inference, on the other hand, is defined by a continuous flow of information and unpredictable user demand as opposed to a pre-defined workload. Once the prompts start coming in, you need geographically distributed compute that responds to and scales with real-time demands.
Because inference runs 24/7 and requires millisecond-fast responses for end users, it relies on commodity ethernet, geographically distributed edge sites, and low-latency network connections rather than isolated throughput within a single data center. The very design choices that make an enterprise data center optimal for training actively work against running inference at scale.
The cost structure is entirely different, too. Training is a one-time capital event — once a model is built, that cost is already sunk. Organizations that continue to optimize their spending for training are now being hit with the reality that they have the wrong foundation.
Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors