Why The Cncf’s New Executive Director Is Obsessed With Inference

“I’m obsessed pinch inference,” Jonathan Bryce, who took over arsenic nan executive head of nan Cloud Native Computing Foundation (CNCF) this summer, said during a sheet hosted by The New Stack astatine KubeCon North America 2025 successful Atlanta.

“A batch of group person been really into LLMs [large connection models] and training,” Bryce told maine later that day. “Where I deliberation we’re benignant of missing nan existent cardinal portion of nan communicative is astir inference.”

In this section of The New Stack Makers, I sat down pinch Bryce to talk why he believes conclusion will predominate nan adjacent decade of computing, what nan CNCF’s caller Kubernetes AI conformance programme intends for enterprises and really projects crossed nan CNCF’s portfolio of much than 130 unfastened root projects are being reshaped by AI workloads.

Why Inference Is nan Real Opportunity for nan CNCF

Bryce’s “obsession pinch inference” isn’t a bad obsession to have. While nan manufacture has agelong remained focused connected training monolithic LLMs, he sees conclusion — that is, serving those models — arsenic nan workload that will specify nan adjacent era of computing. And it’s besides wherever nan CNCF, pinch its wide portfolio of infrastructure projects that are now possibly moreover much important than ever, tin play a foundational role.

“Inference specifically fits truthful good pinch nan technologies that we person successful nan unreality autochthonal community,” he explained. “It’s each astir deploying, securing, scaling, watching and doing it successful a measurement wherever it’s overmuch much of an online, real-time type of exertion versus batch-like training.”

GPUs are expensive, scarce and power-hungry and will stay truthful for nan foreseeable future. Bryce believes unreality autochthonal tooling tin present not conscionable incremental improvements, but “orders of magnitude of ratio for these conclusion stacks.”

Kubernetes, nan CNCF’s flagship project, is often astatine nan halfway of this. “I deliberation nan benignant of communal travel that group person been connected is they will return immoderate stack, it mightiness beryllium Ray connected Kubernetes aliases KServe, which conscionable graduated to go a CNCF incubating task this week. KServe is an conclusion serving engine. They’ll return these kinds of things and they’ll deploy them connected apical of Kubernetes, and that will get them to nan first shape of being capable to load up a exemplary and commencement to reply queries and do nan basal level of inference,” Bryce explained.

The Kubernetes AI Conformance Program

The CNCF launched a Kubernetes AI conformance program astatine KubeCon, giving enterprises a baseline for moving AI workloads. The v1 specification focuses connected GPU support and Dynamic Resource Allocation (DRA), ensuring that conformant Kubernetes environments person nan primitives needed for moving AI inference.

“If you person an AI workload, you’re going to cognize that location are definite components available, for illustration DRA and immoderate different pieces wrong a Kubernetes environment,” Bryce said erstwhile I asked him astir this caller program. “You tin person a conformant Kubernetes situation that’s conscionable benignant of plain vanilla Kubernetes and it doesn’t needfully person each of those elements that you would want if you’re trying to tally an AI workload. And I would say, nan simplest measurement to deliberation astir this is it’s really targeting accelerated workloads.”

Bryce sees nan conformance programme arsenic 1 limb of a three-part instauration nan organization needs: a target to purpose for, conformant implementations and reference architectures based connected nan community’s experiences from real-world deployments. “Right now, I deliberation wherever we are is beautiful acold back, wherever everybody is benignant of figuring it retired connected their own,” he said.

The Agent Inference Explosion Is Coming

The existent hype astir AI agents is only expanding nan request for these solutions, Bryce argues. Agents that activity connected complex, multistep tasks successful parallel will dramatically summation nan load connected conclusion systems, aft all.

“An relationship that we person pinch an LLM is really rather slow and debased volume,” Bryce noted. “When you spell retired and you springiness an supplier a analyzable task pinch aggregate steps, it’s going to effort to do that successful parallel, aliases arsenic accelerated arsenic it can. That’s going to beryllium thing that is expanding nan load dramatically. Anything that you tin do to make those requests hap much efficiently — smaller models, amended inference, immoderate it is — that’s going to make those agents much efficient, much cost-effective, and besides provides amended value results.”

This is wherever nan unreality autochthonal community’s expertise becomes critical. As Bryce noted, nan networking and routing primitives already built into Kubernetes tin beryllium extended pinch inference-aware plugins that way requests to circumstantial GPUs aliases prefilled caches — delivering important capacity gains without having to alteration Kubernetes’ halfway architecture.

Going Beyond nan ChatGPT Moment

Three years aft ChatGPT launched, Bryce believes enterprises are fresh to move past nan “ChatGPT moment” and find nan correct models for nan correct usage cases. That intends smaller, specialized models trained connected purpose-built datasets — not conscionable monolithic LLMs searching done “the history of each Nobel Prize victor and nan campaigns of Genghis Khan” to reply a elemental mobility astir Atlanta traffic.

“We person to move beyond nan ChatGPT infinitesimal and LLMs successful our thought process astir what is AI and really are we going to get nan astir retired of it,” he said.

This, he argues, will let nan organization to beryllium connected way to supply nan infrastructure package for “the biggest workload mankind will ever have.”

Group Created pinch Sketch.