It’s Time To Kill Staging: The Case For Testing In Production

Staging has ever been a basal evil. New approaches to isolation and on-demand sandboxes person yet made it conscionable plain evil.

For decades, nan staging situation has been a fixture of package development. And for conscionable arsenic long, it has been hated by developers everywhere. It’s nan proverbial postulation jam that each developer is forced to beryllium successful conscionable to get their activity validated.

Yet staging is nary longer necessary; its clip has travel and gone. Newer isolation methods alteration developers to trial safely successful unrecorded environments, providing fast, high-fidelity feedback that is intolerable to execute successful a staging environment.

It is clip to termination your staging environment.

The Problem We All Know

Staging environments make sense, successful theory. We must trial codification successful a production-like situation earlier we vessel to production. Anything other would beryllium madness.

But nan cure has go its ain disease. In immoderate statement pinch much than a fistful of microservices, nan staging situation inevitably becomes a wasteland of developer symptom and burned cash.

• It’s a bottleneck: When 50 developers merge code, staging becomes a shared queue. Tests neglect not because of bad code, but because different developer deployed a conflicting change.
• It’s not production: We telephone it “production-like,” but it ne'er has nan aforesaid information scale, postulation patterns aliases personality and entree guidance (IAM) policies. This fidelity spread is wherever vulnerable bugs hide.
• It kills velocity: Commit code, hold for CI, hold for a deploy slot, tally a 40-minute trial suite. This multihour rhythm destroys travel state.
• Nobody maintains it: Teams dainty it arsenic a dumping crushed for unstable builds, further diverging from production.

We person accepted this surgery workflow for 20 years. We believed it was nan only way.

From Environment Isolation To Request Isolation

Staging exists because of nan presumption that testing must beryllium isolated astatine nan situation level. To trial a caller type of nan costs service, you must deploy it to an situation that besides contains a cart service, personification work and auth service.

This presumption is outdated and obsolete.

The caller exemplary is request-level isolation. Instead of cloning an full environment, you rotation up only nan work you’re changing. This exemplary is enabled by Kubernetes-native platforms that supply on-demand sandboxes for each request.

Here is really it works:

• A caller work type is spun up successful an on-demand, isolated “sandbox.”
• When a trial petition is sent (tagged pinch a unsocial header), it is routed to nan sandboxed service.
• As that work calls its dependencies, those calls are routed backmost to nan stable, baseline services successful production.
• The test petition remains isolated arsenic it travels done nan stack, while each different postulation flows normally.

With this approach, you get high-fidelity testing (real dependencies, existent web policies) without nan downsides of shared environments (no collisions, nary queues, dramatically little cost).

Request-level isolation tin beryllium implemented into a accepted section > staging > accumulation deployment travel to amended it, eliminating contention and agelong waits for a CI pipeline. But its existent powerfulness lies successful bypassing nan request for staging altogether, enabling testing successful production.

The Safety Model

Testing successful accumulation sounds dangerous. It’s not erstwhile you person nan correct guardrails.

• Strict information isolation is critical. The biggest interest is that a sandboxed work could corrupt information successful different services. The solution is simple. The aforesaid routing header that isolates trial postulation besides routes database operations to abstracted trial databases. For example, erstwhile a trial petition flows done nan system, each work recognizes nan trial discourse and directs each database writes to isolated trial information stores, wholly abstracted from accumulation databases. Test users interact only pinch trial data. Production information remains untouched.
• Multitenancy provides nan foundation. Virtual backstage web (VPN) restrictions guarantee trial postulation originates only from authorized soul networks. Audit logs way each sandbox convention for compliance.
• Request routing provides blast radius control. Your sandbox is isolated astatine nan petition level. Your colleagues’ activity is unaffected. Production postulation flows normally.
• Progressive rollout remains essential. Sandboxes grip preproduction validation, but you still usage canary deployments, characteristic flags and observability to safely rotation retired to existent users.

Answering nan Hard Questions

Testing successful accumulation is nan logical improvement of nan activity to displacement testing left. But making specified a foundational alteration to your CI/CD pipeline people brings up immoderate captious questions:

• “How do you guarantee trial postulation doesn’t corrupt accumulation data?” Test writes are isolated. The isolation header redirects each database writes to ephemeral, nonproduction information stores that are destroyed aft nan test. Production information is ne'er touched.
• “What astir nan blast radius? How do you extremity a bad trial from DDoSing a downstream service?” Sandboxes are “shadow” deployments built pinch guardrails for illustration circuit breakers and web policies. A buggy trial pinch runaway web requests is automatically throttled and contained, preventing it from overwhelming baseline services aliases affecting different users.
• “This sounds good for elemental APIs, but what astir Kafka aliases gRPC?” The isolation exemplary is protocol-agnostic. The isolation header is propagated complete gRPC aliases arsenic a Kafka connection header. A sandboxed consumer, for example, sounds from nan main taxable but only processes messages pinch its unsocial sandbox ID.
• “What astir compliance and audit requirements?” This exemplary is much auditable. Every sandbox is tied to a circumstantial personification and a propulsion request/dev session. All trial postulation is explicitly tagged pinch a sandbox ID and personification identity, creating a granular audit log that is acold superior to a shared staging environment.

Addressing these and making nan displacement to testing successful accumulation will inevitably impact immoderate upfront engineering investment. However, nan return connected finance for that activity is not conscionable nan savings from eliminating nan nonstop infrastructure costs of your staging. It’s besides a amended developer experience, faster merchandise transportation and less opportunities are mislaid to competitors that tin iterate and vessel faster than you.

It’s Time To Let Go

Eliminating staging whitethorn sound for illustration a tube dream, but respective salient unreality autochthonal teams for illustration DoorDash and Uber person already made nan displacement near to testing successful production. Driven by their highly analyzable microservice stacks and a request to get amended testing fidelity, they are besides realizing immense infrastructure costs savings.

Teams for illustration these deprecating their staging environments to trial successful accumulation correspond a broader trend: nan rejection of approximation successful favour of reality. Staging environments are artifacts of an era erstwhile duplicating infrastructure was a harder problem to lick than coordinating humans astir shared resources.

That era is ending.

The early isn’t astir building amended approximations of accumulation aliases optimizing your CI pipeline. It’s astir adopting an wholly caller paradigm. The teams taking this measurement aren’t conscionable moving faster and cutting costs. They’re besides shipping much reliable code.

It’s clip to termination your staging environment.

Group Created pinch Sketch.