
Inside AI Infra Summit 2026: Agentic AI Takes Over the Infrastructure Conversation
By 10 a.m. on Day 1, the word had already lost its novelty. ‘Agentic AI’ had been splashed across ten slides of three keynotes before the first coffee & stretching break. From exhibit booths, conference keynotes, and fireside chats to around the lunchroom, Agentic AI quickly became an official backbone of the AI infrastructure discussion.
The AI infra summit, from September 15-17 in Santa Clara, kicked off with speakers from Google, NVIDIA, Intel, and Dell all discussing how to scale agentic AI quickly and efficiently, considering the nearly infinite computing growth required for large models.
— Ian Buck, VP, Hyperscale & HPC Computing, NVIDIA
In a research paper cited during discussions, MIT Sloan associate professor John Horton loosely defined agentic AI as “autonomous software systems that perceive, reason, and act in digital environments to achieve goals on behalf of human principals, with capabilities for tool use, economic transactions, and strategic interaction.”
That definition matched what played out across the show floor all week — agentic AI wasn’t just a marketing term, it was the lens speakers used to talk about everything from chip design to data strategy.
Moving from training to the inference era of agentic AI
Sanjay Basu, Senior Director, GPU & GenAI Solutions at Oracle, noted that the focus on agentic AI represents a shift from the training era to the inference era. Less than five years ago, the AI community’s focus was on utilization and efficiency, especially in terms of power. In today’s inference era, we’re focusing on the tokens necessary to solve inference-based queries and keep models performing well.
“You will see that exponential curve, that every year we are increasing 10x model size,” Basu said. What used to be a simple one-shot Q&A with a single agent has turned into a semi-autonomous system whose goal is to complete the core of an ask, request, or task with minimal human oversight.
Model sizes are growing due to agentic load – one question does not have one simple answer. A complicated user query can have upwards of 20 inference checks before responding – especially if coding, planning, tools, and APIs are involved.
The relationship is straightforward — more agentic AI means more tokens, bigger models, and infrastructure that has to scale right along with them. Scott Wallace, Head of Engineering & Architecture at Digital Realty explained in “Tokenomics 101” that as orgs are building agentic workflow models they are using more and more tokens, increasing prices, physical power demands, and context bloat.

Enter the heavyweight class of chip infrastructure
“At the heart of agentic AI is CPUs,” the moderator noted to a crowd of C-Suite professionals nodding along.
Agentic AI is what’s driving the current infrastructure buildout – including complications, constraints, budgets, and loads, CPU and GPU builds. Internal use, computational demands, and external/client-facing dominated another main stage keynote on Agentic AI Infrastructure.
We’ve been focusing on the infrastructure to generate tokens, but we also need to focus on building infrastructure that can use tokens at scale, said Anil Nanduri, VP, AI Products & GTM, Intel Data Center Group.

“As you run these expectation tasks, these agents are running on CPUs. You’re going to need a lot of CPUs to keep up,” Nanduri continued.
It used to be that for every 8 GPUs you needed 2 CPUs, but now you almost need 1:1 to run agents, Nanduri quipped. GPU and CPU weights and building scalable infrastructure dominated lunch table conversations among CTOs and builders.
In a fireside chat, Ian Buck, VP, Hyperscale & HPC Computing at NVIDIA, discussed building infrastructure for frontier AI at scale.
“No AI model, except for maybe the smallest ones, fit on a single GPU,” Buck said. You have to design a compute token factory end-to-end. CPU, storage, GPU, and networking issues have been the biggest challenges over the last two years as we start building AI factories.
“We have to help the world build a supply chain that can build this infrastructure,” Buck continued. Scaling AI infrastructure is not a one-time strategy, but a continuous Silicon Valley and tech collaboration. The scale is unprecedented.
Enterprise AI Capability and mid-scale corporations
Nicola Tan, Director, Enterprise AI GTM at AMD, introduced what we all know, “You’re not optimizing for a single chatbot, you’re optimizing for a whole system,” Tan said.
Enterprise AI is not hyperscale AI – the priorities of OpenAI, Oracle, and Microsoft are not those of a mid-size enterprise using AI to just improve internal workflows.
For mid-level enterprises using AI as an enabler, the quest to optimize agentic AI relies on the removal of siloes. The biggest crunch to this? Data. Employees using AI across departmental siloes have scattered (sometimes sloppy) data.
“The model is table stakes. The value is in the data,” Tan said.
Data is distributed across clouds, on-premises, and client systems. Proprietary data is critical. She then made a bold claim. If AI pilots fail in mid-level orgs, it’s because of the lack of data outside a POC, sandbox environment. Use smaller models for agentic AI – break up your compute into chiplets.
Agentic workloads are heavier, less predictable, and more failure-prone than the last generation of AI, and that’s driving demand across compute, networking, storage, and the orchestration/tooling layer.
Teamwork makes the agentic dream work
Another buzzword? Partner. Ask ten CTO’s how they scale their infrastructure stack, and all ten will answer with some iteration of “we partner with…,” or “Our partners at [org] enable us to…”
Partnerships help as the scale continues to grow. As scale increases, it becomes a system optimization problem of cooling, GPUs, CPUs, and networking, said Sachin Katti, VP, Compute Strategy & GPT-Infra, OpenAI.
Exhibit hall booths bolstered the goal of building out ecosystems rather than individualizing AI’s parts.
Alexera, a Dutch company creating AI acceleration technology, had a 20-foot-tall block showing its Voyager Wingman, “an agentic platform built with the knowledge base of the Voyager SDK that turns plain language into working AI — whether you are a first-time developer or an experienced engineer looking to move faster.”

Parsec AI also bolstered their optimization of agents – not models. Their talking track showed off their approach to trimming agent context — claiming up to 78% fewer tokens and roughly 40-50% lower costs by stripping out information agents don’t actually reuse.
MiTAC Computing, the Taiwan-based server manufacturer, framed its booth around ‘scalable AI infrastructure for the agentic era’ — leaning on AMD-powered systems and liquid-cooling tech designed for the density and heat demands of agentic workloads at rack scale.

When can we expect the immeasurable cost to be measured?
As coverage rolls on, expect the cost question to keep surfacing. Unlimited infrastructure growth is a strategy for hyperscalers with unlimited budgets — it isn’t one for mid-size organizations or smaller enterprises trying to scale without adding a billion dollars to the line item.
Tan’s point about data over models is a preview of where that pressure will show up first: enterprises without hyperscaler budgets will have to get more out of what they already have, not just buy their way to scale. We’ll be watching for vendors, speakers, and orgs willing to put a real number on what advancing AI actually costs — starting with the token economics conversation that dominated the latter half of the show.