Tech Veterans’ New Approach To Eliminate ‘configuration Hell’

There’s gotta beryllium a amended way!

As pinch truthful galore startups, that was nan thought down ConfigHub, focused connected delivering some dev and ops folks retired of “configuration hell.”

“Config hellhole is erstwhile you’re trying to hole nan system, everything’s broken, everybody’s screaming, and you can’t moreover fig retired which of nan 1,000 YAML files contains nan existent error,” said CEO Alexis Richardson successful an interview.

It’s 1 of nan problems he and his cofounders person dealt pinch for years. Richardson was nan laminitis of unreality autochthonal instrumentality guidance level Weaveworks and connection agent RabbitMQ. Former Google package engineer Brian Grant, ConfigHub’s CTO, was nan original lead designer of Kubernetes; and Jesper Joergensen, nan caller company’s main merchandise officer, held various roles astatine Salesforce, including pinch Heroku, earlier joining Twilio to lead its voice, video and level teams.

“We knew location had to beryllium thing amended than this. … What we person alternatively is group wading done pages and pages and pages and pages of YAML, Terraform, HCL [HashiCorp Configuration Language] … immoderate it’s called. You know, caller languages each nan clip for supposedly solving this problem, and it’s incomprehensible to astir people. And erstwhile you person an outage, you don’t person clip to study it. You conscionable sewage to hole it,” said Richardson.

Their Solution: Treat Configuration As Data

Misconfiguration perennially makes nan OWASP Top 10 database of nan astir communal information vulnerabilities. It is blamed for monolithic outages including caller ones astatine Cloudflare, Amazon Web Services and Azure, arsenic good arsenic nan July 2024 Crowdstrike nonaccomplishment estimated to person cost airlines, banks and different companies $5.4 billion. These are problems that approaches for illustration Infrastructure arsenic Code (IaC), Infrastructure from Code (IfC) and others person aimed to fix.

“Brian’s ray bulb thought was conscionable to return each nan configuration information and abstracted it retired … return nan manifest, render it each retired into values, nary variables, nary programming loops, nary mysterious templates, thing that has to beryllium generated arsenic portion of nan configuration process, and each of that past becomes operational facts astir your business,” Richardson said.

Grant explained successful a caller KubeCon North America talk that during an outage, ops folks don’t want to wade done a maze of YAML files, git folders and joined-up dev tools, they want a azygous pane of solid they tin break to find and hole nan error. Yet GitOps wants a write-through connected git earlier a hole tin return place.

To simplify things, nan ConfigHub squad created a database wherever configuration is stored and managed arsenic system data. It is serialized utilizing modular information formats, specified arsenic YAML, and stored pinch revision history and immoderate metadata. Since nan information is maintained successful a unrecorded state, it’s ever fresh to go.

Code that operates connected that configuration is abstracted from configuration data connected via API.

“The configuration information is not parameterized. The configuration of every variant is stored independently successful its native, afloat rendered, WET [Write Every Time] form. There are nary templates, variables, conditionals, loops aliases generators that create configuration connected nan fly. You besides don’t request to constitute and support patches by manus nan measurement you do with Kustomize. Every worth circumstantial to an situation is stored virtually successful nan config…. Make a elemental edit to nan config and nan unrecorded resources tin beryllium updated much quickly and pinch fewer, simpler steps than deploying done git and CI/CD pipelines,” Grant explained successful a blog post.

In nan question and reply Joergensen explained: “If you look astatine Kubernetes, which we’re focused connected correct now, and you look astatine a emblematic champion believe setup today, you’ll spot teams … fundamentally usage Helm arsenic first measurement of nan generation, and past what it generates gets sent complete to CI/CD. … But a batch of coding, a batch of processing, procreation is happening wrong nan CI/CD step, and location is nary visibility astatine this level,” he said.

“The DevOps folks cannot spot what gets deployed earlier it gets deployed, because it’s each being generated, and that is creating a bunch of problems. And truthful we alternatively say, ‘Start by immoderate you want to do pinch software,’… but what you extremity up pinch is simply a database of what we telephone literal config, meaning that nan config is afloat rendered, afloat generated, successful nan shape that it needs to beryllium successful correct earlier it goes to nan infrastructure. And that is your root of truth. That is your strategy of record.

“And that is what you’re looking at, some arsenic humans and arsenic automations and AI and immoderate other you want to constituent astatine it, and that gives you a full different level of fidelity astir what will hap next. And that changes things a lot.”

With one-to-one mapping, each moving entity and truthful each correction tin beryllium located and updated, moreover successful bulk, utilizing modular database operations specified arsenic semantic study based connected information schemas. It enables reverse aliases “bidirectional” GitOps, allowing users to spot changes to beryllium made earlier deployment and nan results afterward.

Since each config is isolated, changes impact only that 1 situation pinch nary chance they will interfere pinch thing else.

How ConfigHub Works: Core Components

In a Kubernetes deployment, ingress hostnames, situation variables, image tags, work dependencies, assets requests and much are each stored straight successful nan YAML alternatively than being generated via template variables and input values, explained advisor and level engineering guru Artem Lajko successful a deep dive into IaC issues and ConfigHub’s take connected it.

“Make a elemental edit to nan config, and nan unrecorded resources tin beryllium updated without moving a analyzable CI/CD process,” he noted, adding “Policy enforcement tin cheque nan configuration straight and study results immediately, dissimilar templated config that first needs rendering.”

The halfway components of ConfigHub are:

• A Unit maintains a sequential database and history of revisions of nan config data.
• A Target is wherever nan configuration is to beryllium applied. It abstracts entree specifications and credentials to Kubernetes clusters, unreality accounts and different services, meaning users tin negociate resources without straight handling credentials.
• A Worker is simply a abstracted process connecting ConfigHub and extracurricular entities. Similar to a Kubernetes GitOps Operator aliases CI runner, it operates wrong your cluster. Its 2 main types of activity are Functions and Bridges.
• Functions are executable pieces of codification that run connected configuration information wrong Config Units. They tin beryllium publication only, mutating aliases validating, and widen ConfigHub’s automation capabilities.
• Bridges link ConfigHub pinch your destination assets into nan correct API calls and relay cognition events and position backmost to ConfigHub.

Lajko reports that alternatively than being simply a database, ConfigHub allows teams to support their existing workflows — and create automated ones — and provides a azygous root of truth and wide position of nan situation they’re moving in.

The Real-World Impact of Misconfiguration

At KubeCon, Erick Bourgeois, head and Kubernetes level technologist lead astatine RBC Capital Markets, explained really 1 other abstraction successful its monolithic tangle of config files collapsed its systems. The institution had YAML wrong YAML wrong YAML. A regular ConfigMap spot crossed astir 30 Grafana instances turned into a three-day investigation because of unintended interactions successful templated config.

By treating configuration arsenic data, you tin return advantage of being capable to nary longer spot it arsenic text, but queryable content. Once each validation and mutation is complete, you person a resulting “unit” that tin tube done a worker to execute your goal, Bourgeois said connected LinkedIn.

Richardson said successful nan position pinch Bourgeois: “That was a bully illustration of what I’ve been calling ‘config sprawl,’ wherever you’ve sewage tons of different files successful tons of different repos owned by different people, different templates which make different formats astatine different times connecting to tons of different systems. … Controlling this and managing it is getting harder and harder and harder. And if you look astatine nan statistics, you’ll spot group for illustration nan DORA folks who are now astatine Google will show grounds that nan velocity of DevOps conscionable has not sewage immoderate faster.

“That wasn’t what we hoped would hap erstwhile astir 10 years agone we and others came up pinch these caller automated technologies astir Kubernetes. We thought we would make things easier, much automated, safer, much compliant. And instead, group look to beryllium getting much and much stuck. And that’s partially because systems person grown. We’re nan victims of our ain success. ….”

A New Way of Thinking About Configuration Management

He urged nan assemblage to effort retired its early shape Software arsenic a Service (SaaS) technology, which is successful nan preview stage.

“It makes each nan configuration that you’re going to request for nan applicable parts of managing infra and apps centralized and provides building truthful that limitations betwixt exertion components tin beryllium seen and understood successful narration to history, versioning and perchance argumentation arsenic well. So you tin use things for illustration triggers and compliance and functions astir that. That will let you to do really important things that are a small tricky correct now, for illustration seeing what happens earlier you deploy and past being capable to cheque afterwards to spot if that was correct,” he said.

He stresses that it’s still very early days for nan company. The trio began moving connected this institution successful 2024, and nan Menlo Park, California-based startup emerged from stealth successful March, announcing $4 cardinal successful funding.

“We’ve been astir this manufacture a agelong time. We cognize really agelong it takes to build things, and we don’t want to beryllium benignant of sounding for illustration we person each nan answers,” Richardson said successful nan interview.

But nan cofounders judge nan clip is correct to effort to spur change.

“We deliberation that there’s capable pent-up energy, group sitting connected nan sidelines, benignant of having accepted nan position quo, but having ideas and having perspectives that, if we tin works a mini seed of a caller measurement of doing things, we mightiness spot personification jump in, right? … If we tin get a mini flywheel going successful caller measurement of thinking, we would for illustration to spot much group jump connected it.”

Group Created pinch Sketch.