Kelsey Hightower On Engineering At Scale

“There’s a saying I like: ‘Some group person bully ideas. Some ideas person people.’ When your thought outlives you, that’s success.” — Kelsey Hightower

Kelsey Hightower’s profession is simply a cleanable illustration of that. His ideas person taken connected a life of their own, extending acold beyond his activity astatine Puppet, CoreOS, to KubeCon and Google. And he continues to standard his effect pinch his signature unscripted keynotes arsenic good arsenic nan definitive book, “Kubernetes Up and Running.”

We were thrilled that Hightower joined ScyllaDB’s Monster SCALE Summit to stock his experiences and proposal connected engineering astatine scale. And to standard his insights beyond those who joined america for nan unrecorded keynote, we’re capturing immoderate of nan astir memorable moments and takeaways here.

Monster SCALE Summit 2026 will spell unrecorded March 11-12, featuring antirez, creator of Redis; Camille Fournier, writer of “The Manager’s Path and Platform Engineering”; Martin Kleppmann, writer of “Designing Data-Intensive Applications”) and much than 50 others. The arena is free and virtual; registry and subordinate nan organization for immoderate lively chats.

Fail Before You Scale

The question and reply pinch big Tim Koopmans began pinch a pointed informing for attendees of a convention that’s all astir scaling: Don’t conscionable pursuit standard because you’re fascinated by others’ scaling strategies and achievements. You request to really acquisition immoderate symptom personally first. As Hightower put it:

“A batch of group spell spot a convention talk — I’m astir apt blameworthy of this myself — and past effort to ‘do standard things’ earlier they moreover person acquisition pinch what they’re doing. Back astatine Puppet Labs, tons of group wrote ammunition scripts pinch bad correction handling. Things went awry erstwhile conditions weren’t perfect. Then they moved connected to configuration management, and those who made that travel could understand nan trade-offs. Those who started straight pinch Puppet often didn’t.”

“Be judge you person a reason,” he said, “So earlier you over-optimize for a problem you whitethorn not moreover have, you should ask: ‘How bad is it really? Where are nan metrics proving that you request a much scalable solution?’ Sometimes you tin do thing and conscionable hold for scaling to go nan default option.”

Ultimately, you should dream for nan “good problem” wherever expanding request causes you to deed nan limits of your tech, he said. That’s overmuch amended than having fewer customers and over-engineering for problems you don’t have.

Make nan Best Choice You Can … for Now

The speech shifted to what level of standard teams should target pinch their first tech stack and each consequent iteration. Should you optimize for a early authorities that hasn’t happened yet? Play it safe successful lawsuit nan marketplace changes?

“If you’re not judge whether you’re connected nan correct stack … I committedness you, it’s going to change,” Hightower said. “Make nan champion prime you tin for now. You tin walk each twelvemonth optimizing for ‘the champion thing,’ but it whitethorn not beryllium nan champion point 10 years from now. Say you prime a database, spell each in, study champion practices. But put a small footnote successful your creation doc: ‘Here’s really we’d alteration this.’ Estimate nan switching cost. If you do that, you won’t get stuck successful sunk costs fallacy later.”

Rather than trying to foretell nan future, deliberation astir really to debar getting trapped. You don’t want limitations aliases extensions to limit your expertise to migrate erstwhile it’s clip to return a amended (or conscionable different) path.

“Change isn’t failure,” he emphasized. “Plan for it; don’t fearfulness it.”

In Hightower’s view, scaling decisions should commencement connected a whiteboard, not successful code.

“When I was astatine Google, we’d do method whiteboard sessions. Draw a statement — that’s time. “Today, we’re here. Our level allows america to do these things. Is that bully aliases bad?” Then tie ahead: “Where do we want to beryllium successful 2 years?”

He continued, “Usually that’s driven by teams and customer needs. You can’t do everything astatine once. So crippled milestones — six months, a year, etc. You tin push things retired successful clip for erstwhile caller libraries aliases devices arrive. If thing caller shows up that gets you 2 years up instantly, great. Having a timeline gives you state without guilt that you can’t vessel everything today.”

Are You Really Prepared for a 747?

Following up connected nan Google thread, Koopmans asked, “I’d emotion to perceive applicable ways Google avoids over-engineering erstwhile designing for scale.” To exemplify why “Google-scale” solutions don’t ever fresh everyone else, Hightower utilized a memorable analogy:

“I had a customer erstwhile say, ‘We want BigQuery on-prem.’ I said, ‘You do? Really? OK, really overmuch money do you have?’ And it was 1 of those companies that had plentifulness of capital, truthful that wasn’t nan issue. I told them, ‘That would beryllium for illustration going to nan airport, looking retired nan window, seeing a brand-new 747 and telling nan hose that you want that plane. Even if they fto you bargain it, you don’t person a pilot’s license, you don’t cognize really to substance it. Where are you going to parkland it? Are you going to thrust it down your subdivision, decapitating nan roofs of your neighbors’ houses?” Some things conscionable aren’t meant for everyone.”

Ultimately, whether it’s over-engineering aliases not depends connected nan target user. Understand who they are, really they activity and what devices they use, past build pinch that successful mind.

Hightower besides warned against treating “best practices” arsenic cosmopolitan truths: “One mobility that astir customers show up pinch is, ‘What are nan champion practices?’ Not needfully nan champion practices for me. They conscionable want to cognize what everyone other is doing. I deliberation that mightiness beryllium different anti-pattern successful nan mix, wherever you only attraction astir what everyone other is doing and you don’t bring nan basal discourse for a bully recommendation.”

How Leaders Should Think About Dev Tooling

“Serializing engineering culture” (Hightower’s phrase) like Google did pinch its google3 monorepo makes it elemental for thousands of caller engineers to subordinate nan squad and commencement contributing almost instantly. During his tenure astatine Google, everything from gRPC to deployment devices was integrated. Engineers conscionable opened a browser, added codification and reviews would commencement automatically.

However, there’s a good statement betwixt serializing and stifling. Hightower believes that prohibiting engineers from moreover installing Python connected their laptops, for example, is overkill: “That’s for illustration telling Picasso he can’t usage his favourite brush.”

He continued: “Everyone useful differently. As a leader, study what devices group really usage and beforehand sharing. Have engineers show their workflows — nan shortcuts, setups and plugins that make them productive. That’s wherever productivity and velocity travel from. Share nan nuance. Most group deliberation their tricks are excessively mini to matter, but they do. I want to spot your dotfiles! You’ll animate others.”

Watch nan Complete Talk

As Hightower noted, “Some group person bully ideas. Some ideas person people.” His attack to standard — pragmatic, context-driven and quality — shows why immoderate ideas really do outlive nan group who created them.

You tin spot nan afloat talk below.

Fun fact: it was genuinely an unscripted question and reply — Hightower insisted! The squad met him successful nan edifice lobby that morning, chatted a spot during a java run, prepped nan camera angles … and abruptly Hightower and Koopmans were broadcasting to 20,000 attendees astir nan world.

Group Created pinch Sketch.