Kamera Uses Simulation To Verify Kubernetes Controller Logic

Tim Goodwin, a postgraduate student astatine nan University of California Santa Cruz has been reasoning a batch astir nan early of Kubernetes. And he sees a world beyond clusters, wherever Kubernetes tin beryllium a cosmopolitan power level for conscionable astir anything.

Making this imaginable is nan magic of controllers, which are built connected nan Kubernetes controller runtime. The expertise to constitute controllers — and to yoke them together — is what gives Kubernetes a batch of its power,

Controllers, however, are really difficult to negociate and debug. So Goodwin created a controller simulation software, called Kamera, which provides targeted instrumentation to seizure nan behaviors of individual controllers and nan interactions betwixt them, utilizing simulation and exemplary checking.

Kamera useful pinch existent controller code, and you don’t request a cluster to tally it — successful fact, it’ll activity conscionable good connected a developer’s laptop.

Goodwin introduced nan exertion astatine KubeCon + CloudNativeCon NA 2025 earlier this month.

Kubernetes arsenic a Universal Control Plane

Kubernetes was created arsenic a measurement to tally containerized apps connected clusters, and it has since been extended to negociate different resources arsenic well.

Databases, CI/CD systems for illustration Argo CD, outer infrastructure platforms specified arsenic Crossplane, and, astir recently, machine learning (ML) systems are each controlled by Kubernetes these days.

Kubernetes has nan imaginable for being a “universal power level for thing we want,” Goodwin said. “Pretty overmuch thing that tin beryllium described declaratively and reconciled continuously, we tin negociate pinch Kubernetes.”

With Kubernetes, nan personification describes successful a declarative language nan desired authorities of their system, successful position of resources. It is nan controllers that will continuously reconcile nan existent authorities to nan desired authorities done a power loop. They show a bid of alteration events that are sent from nan API server, and respond pinch nan basal changes backmost to nan server.

You tin besides constitute controllers to tally non-Kubernetes resources arsenic well, pinch nan thief of a custom assets definition (CRD) to specify nan resource.

Controllers tin besides beryllium aggregated and controlled collectively, paving nan measurement for as-yet-unseen platforms.

“It’s this creation method that allows america to raise nan level of abstraction,” Goodwin said.

The Challenges of Managing and Debugging Controllers

Controllers, however, are difficult to manage, particularly successful ample numbers. They get into title conditions. They run from old information and tin nutrient nondeterministic results if severely written.

With power loops, “the business logic is beautiful simple, but there’s a batch of things that you request to watch retired for to guarantee that this business logic runs soundly wrong a reconcile loop,” Goodwin said.

In summation to azygous controllers, assembling them successful an aggregate will besides bring challenges, arsenic nan devs besides request to logic astir nan composite logic of aggregate controllers, Goodwin said.

Ensuring that each individual controller useful arsenic planned doesn’t guarantee that they, erstwhile tally collectively, won’t do immoderate benignant of damage. Two controllers managing a azygous assets whitethorn vie for control, causing a title condition.

“When thing goes incorrect successful these interactions, it tin beryllium a immense headache to debug,” he said.

These sorts of bugs tin travel up successful codification changes, aliases changes successful configuration, aliases changes successful resources. And these title conditions whitethorn hap each nan time, aliases only rarely, if you are really unlucky.

Today, location are fewer devices to help. In galore cases, nan champion a developer tin do is propulsion up nan log files of a controller and conjecture nan cluster authorities astatine nan clip erstwhile everything went wrong.

“Because these types of issues tin reproduce successful very circumstantial and possibly fleeting conditions, that makes our wide debugging process moreover harder erstwhile there’s a reproducibility challenge,” he said.

Welcome to distributed programme debugging hell.

Exploring nan Execution Space With Kamera

Goodwin wrote Kamera to thief nan developer amended understand a cluster state.

“For a fixed reconciliation process, Kamera is going to show each of nan controller actions that went into that process,” he said. “So if there’s immoderate issue, we tin understand precisely what’s going connected to create it.”

In bid to prevention time, Kamera simulates a cluster. Usually, controllers interact pinch nan strategy done nan Kubernetes API server; Kamera simulates nan server pinch a mocked API customer interface.

“If you person controller runtime implementations, you tin ligament them up to this instrumentality without immoderate further codification changes,” Goodwin said.

The software, written successful Golang, runs connected a azygous CPU thread and tin beryllium tally connected a laptop. No cluster needed.

To work, nan package represents nan cluster (or, much generally, nan system) and walks done nan creation of ReplicaStates — pursuing immoderate different controllers that are called — and proceeds until each nan controller states are converged and location are nary pending reconciliations.

“So what this intends is that we’re successful immoderate authorities wherever each controller that has a liking successful nan contents of that authorities has observed those contents and decided that thing needs to alteration astir it,” he explained.

If it works, that intends your power level logic is sound.

Distributed Program Debugging Blues

If nan simulation gets caught successful a loop, that intends thing is wrong. There whitethorn beryllium a title issue, aliases a nonidempotent cognition producing a different consequence each clip it’s executed. Or possibly location is immoderate different nondeterministic cognition that keeps shifting nan authorities into immoderate different (presumably) converged state.

If you’re moving these operations connected a existent cluster, you’d beryllium successful superior threat of toppling it complete pinch nan bid of actions described crossed each nan controllers. Good point it’s a simulation.

Modeling nan Execution Space

Kamera tin not only simulate nan actions of components, but it will besides exemplary nan full execution abstraction successful bid to verify that nary of nan bugs mentioned supra will ever happen.

“We tin usage our execution exemplary to comprehensively hunt each imaginable execution way and cheque for properties of liking for illustration unchangeable convergence,” Goodwin said.

This process, called exemplary checking, “exhaustively explores nan abstraction of each imaginable states that our strategy tin participate into to verify that definite properties that we attraction astir ever hold.”

Every azygous execution way of each controller is exhaustively tested, successful effect creating a graph of each imaginable strategy states (to immoderate extent you specify). Each resulting converged abstraction is tracked and compared. Each converged authorities tin beryllium explored, via an action-by-action stepper, to spot what execution paths were taken.

“So what this lets america do is measurement done each reconcile, and inspect successful a granular measurement really cluster authorities is evolving,” he said.

To trial nan Kamera connected existent unreality autochthonal software, Goodwin utilized nan package to look astatine really various services wrong nan Knative serverless package reconcile pinch each other. Not surprisingly, he recovered nan programme to beryllium sound.

Kamera takes a different attack than SimKube, which acts arsenic much of a replay mechanism of actions that person already happened connected a cluster.

Kamera is unfastened source, but Goodwin warned that nan package is only “research-ready” and is looking for much feedback from nan Kubernetes community.

Despite immoderate unsmooth edges, this package could beryllium a very useful instrumentality for debugging stubbornly elusive misbehaviors wrong Kubernetes clusters.

Or readying for systems for Kubernetes to power that haven’t ever been thought of before.

View nan full talk here:

Group Created pinch Sketch.