An End-to-end Cloud Native Observability Framework

It has been established that observability is basal to understand really our systems are performing and erstwhile thing goes incorrect pinch them. Through my activity pinch enterprises adopting observability services for their captious unreality autochthonal workloads, I spot observability is adopted successful silos. The requests are often separated: focusing connected exertion traces aliases only connected Kubernetes metrics/logs aliases only connected CI/CD pipelines telemetry.

However, my attack is to deliberation astir end-to-end observability from Day 1 and not dainty it arsenic a bolt-on.

Here’s a demonstrated end-to-end observability model utilizing a elemental two-microservices exertion built/deployed connected a managed Kubernetes cluster done CI/CD pipelines. It focuses connected nan telemetry that tin beryllium collected from each layer— application, Kubernetes and CI/CD — and really each contributes to faster troubleshooting and amended strategy health.

Application and Observability Architecture

Image 1 shows nan architecture, including telemetry collection. The demo exertion includes 2 microservices — retail-web and retail-api — moving connected a managed Kubernetes cluster successful a unreality environment. For this application, observability is covered done nan pursuing strategy:

• Traces from nan exertion are collected utilizing OpenTelemetry collector, which offers a vendor-agnostic measurement to receive, process and export telemetry data.
• Logs from Kubernetes cluster are collected utilizing fluentd daemonsets (one log postulation pod per node).
• Kubernetes infrastructure metrics are collected utilizing nan unreality platform’s proprietary agent. This could besides beryllium done utilizing a Prometheus Node Exporter.

Because this exertion runs connected a unreality platform, telemetry from unreality autochthonal components specified arsenic CI/CD pipelines, compute nodes and nan Kubernetes power level is collected done nan platform’s observability services.

This demo unit exertion is focused connected 1 feature: nan checkout request. The user’s checkout petition hits nan load balancer, and is processed by nan web frontend, which calls nan API backend. The API backend past executes nan 3 cardinal operations – checking inventory, charging costs and creating nan order, past each 3 operations deed a database. The API backend uses SQLite for objection purposes; successful production, this would beryllium a managed database service. The observability patterns stay applicable sloppy of nan underlying database.

Note: A managed Kubernetes situation besides includes respective different cloud-managed components, specified arsenic networking and load balancing. It’s important to alteration and show their telemetry arsenic good since these services straight impact nan reliability of Kubernetes workloads.

Capturing Application Traces pinch OpenTelemetry

A trace captures nan end-to-end travel of a azygous personification request. A azygous trace is made up of aggregate spans depending connected really nan petition traverses nan distributed system. In my application, a azygous personification petition starts erstwhile nan personification hits nan Checkout fastener connected nan UI, moving connected to nan backend and SQLite database. A azygous trace search this petition will person aggregate spans and, successful this example, individual spans will seizure each HTTP petition and operations specified arsenic “verifying inventory,”“executing payment”and “create order” arsenic shown successful Image 2.

The exertion uses a operation of OpenTelemetry auto-instrumentation (for Flask and HTTP calls) and manual spans (around nan business logic stages), and each spans are exported done nan OpenTelemetry (OTel) Collector, which enriches them pinch Kubernetes metadata earlier sending them to nan unreality autochthonal exertion capacity service.

Image 3 shows a codification snippet from my telemetry module, wherever I specify a civilization bootstrap() function. This usability configures OpenTelemetry for my work by mounting assets attributes specified arsenic service.name, service.namespace, and deployment.environment. These attributes go portion of each span.

Inside nan aforesaid function, I initialize an OpenTelemetry TracerProvider and connect an OTLP (OpenTelemetry Protocol) span exporter, which is nan constituent responsible for sending spans to nan adjacent destination successful nan pipeline, which successful this lawsuit is nan OTel Collector.

If You’re New to OpenTelemetry:

OpenTelemetry Protocol (OTLP) is nan modular measurement telemetry signals (traces, logs, metrics) are transmitted betwixt components. The OTLP span exporter sends spans to whichever OTLP endpoint is configured.

Although an OTLP span exporter candirectly nonstop traces to an exertion capacity monitoring (APM) backend, I intentionally nonstop them to nan OTel Collector instead. There are 2 reasons for this:

1. Vendor neutrality and futureproofing.
   When nan collector sits betwixt nan exertion and APM backend, I tin way nan aforesaid traces to immoderate backend (Grafana Tempo, Jaeger, cloud autochthonal APM services, etc.) without modifying exertion code.
2. Span enrichment and processing.
   I usage nan collector to inject Kubernetes metadata (pod name, node name, deployment, etc.) into spans. The collector tin besides execute batching, sampling, transformations and routing, if needed.

Image 4 shows a codification snippet pinch an auto-instrumentation conception wrong bootstrap(). Here, I alteration OpenTelemetry’s instrumentation libraries for:

1. Requests (Python’s HTTP client) to automatically create client-side spans whenever nan exertion makes outbound HTTP calls (such arsenic retail-web calling retail-api).
2. Flask to automatically create server-side spans whenever nan exertion receives inbound HTTP requests.

Without nan RequestsInstrumentor, nan downstream API calls would not look arsenic portion of nan aforesaid trace, and nan distributed parent-child narration would beryllium lost. Flask instrumentation handles inbound traffic, while requests instrumentation handles outbound calls. Because microservices often telephone each other, some are required to support a complete distributed trace.

Image 5 shows a codification snippet from nan exertion record (app.py), wherever nan bootstrap() usability is called instantly aft creating nan Flask exertion instance. This is what activates auto-instrumentation for nan moving work and applies nan assets attributes defined earlier.

Manual instrumentation intends explicitly creating spans successful your exertion codification to seizure business logic, database operations aliases immoderate activity that auto-instrumentation cannot infer.

In nan retail-web service, by nan clip /checkout/execute handler runs, Flask auto-instrumentation has already created a SERVER span for nan HTTP request. In nan handler, I fetch that span pinch trace.get_current_span() and past I adhd a manual genitor span called Process_Checkout_Flow pinch 3 kid spans: Verify_Inventory_Status, Execute_Payment_Charge and Create_Order_Record. Process_Checkout_Flow sits betwixt nan HTTP-level SERVER span and these business steps. These spans representation straight to my business steps, and I connect attributes for illustration retail.flow and retail.stage truthful I tin later select traces by travel and shape if I want to.

Image 6 beneath is nan codification showing nan manual genitor span and business span for nan verifying inventory step.

Image 7 beneath shows nan afloat span character for 1 checkout request. The first span, retail-web: POST /checkout/execute, is nan entry-point server span created automatically by nan Flask auto-instrumentation. The adjacent span, retail-web: Process_Checkout_Flow and nan nested spans underneath it (inventory check, costs complaint and bid creation) are manually instrumented. These manual spans travel from nan code-level instrumentation arsenic shown successful image 6.

Auto and manually instrumented spans service different purposes. If nan latency comes from auto-instrumented spans, it tin beryllium attributed to overhead processes specified arsenic web connectivity issues whereas if it’s coming from manually instrumented spans, past it’s mostly because of exertion logic specified arsenic inefficient loops.

OpenTelemetry generates 3 types of spans:

• Server spans — created whenever a work receives an incoming HTTP petition (such arsenic retail-web receiving POST /checkout/execute, retail-api receiving GET /inventory/check).
• Client spans — created whenever a work makes an outgoing HTTP telephone (such arsenic retail-web calling retail-api). These show nan outbound information of nan round-trip.
• Internal spans — spans created wrong nan work to correspond soul units of work. All manual spans (such arsenic Process_Checkout_Flow, Verify_Inventory_Status Execute_Payment_Charge, DB operations) autumn nether this class because they correspond code-level operations performed wrong a service, and I didn’t explicitly group nan span benignant to immoderate different type. Image 8 beneath shows different types of spans successful nan span tree.

I utilized nan OTel Collector to enrich spans pinch Kubernetes context. Image 9 shows really I configured nan OTel Collector to extract Kubernetes metadata.

Once this configuration is applied, each span sent done nan collector includes Kubernetes attributes arsenic shown successful Image 10. This enrichment becomes highly valuable erstwhile troubleshooting capacity issues. If a span slows down, I tin instantly spot which pod and node executed nan code.

Observability for Managed Kubernetes Environments

In a managed Kubernetes situation connected a unreality level location is simply a shared work model. The unreality supplier manages nan Kubernetes power level components specified arsenic API server, etcd, scheduler, controller manager, node provisioning and exposes only nan logs aliases metrics that nan level chooses to make disposable for power plane. Everything that runs wrong nan nodes (pods, containers, exertion processes) is afloat user-managed and nan personification needs to ain their observability setup.

In essence, while each awesome managed platforms (Oracle, Google, Amazon, Azure) expose power level metrics for their managed Kubernetes, but a personification must mostly opt-in aliases usage nan provider’s autochthonal monitoring solution to devour them. Essential power level metrics see API server requests, API server petition latency and etcd latency to understand really nan power level is performing and raise work tickets pinch nan unreality supplier erstwhile you spot unexpected behavior.

Node Health – Understanding node wellness is important arsenic nodes are nan ground wherever exertion workloads run. A bully starting constituent is to support a watch connected node-level metrics arsenic they are our early informing strategy for assets exhaustion. Image 11 shows CPU utilization for 3 nodes belonging to a Kubernetes cluster. Here we spot adjacent distribution crossed 3 nodes, and that’s healthy. If 1 metric statement was astatine 10% while nan different was astatine 95%, that would person shown that 1 node is overburdened while nan different is underutilized.

Logs besides tin supply insights.Logs tin thief successful troubleshooting, but they besides make for a awesome overview for illustration nan node wellness summary shown successful Image 12. Through this widget, I item each 3 unit values for each node, which contributes to nan readiness of a node to big pods. These conditions travel straight from kubelet and are surfaced successful node status.

The node-pod relationship is important, providing nan position of pods moving connected nodes. Image 13 shows a machine-learning-based visualization that brings aggregate dimensions together: node, namespace, pod position and count of pods. This makes it easy to spot patterns specified arsenic uneven pod placement, pods stuck successful pending/failed states aliases nodes consistently hosting problematic workloads. That’s nan existent powerfulness of observability: being capable to spot relationships alternatively than isolated metrics.

Pod Health

Logs from pods tin beryllium utilized to supply snapshots for illustration Image 14 which shows specifications of each moving pods including namespace, placement, controller and controller kind.

But for knowing pod wellness simply knowing pod shape (Running / Pending / Succeeded / Failed / Unknown) is not enough. A pod whitethorn show a Running shape while still being unhealthy because nan shape only reflects nan life rhythm state, not whether nan instrumentality is reachable aliases functioning correctly.

Image 15 is simply a bully example. The pod shape highlighted is moving but erstwhile I correlated it pinch Kubernetes Events, I recovered “readiness probe failure.” This relationship exposes nan existent truth:Pod position does not adjacent pod health. Health depends connected signals specified arsenic readiness, liveness, restart counts and probe failures. These seldom show up successful nan high-level shape alone. This is wherever observability helps by connecting pod logs, events and metrics together truthful that nan “real” wellness becomes visible.

Gaining Insights from CI/CD Pipeline Observability

In CI/CD setup, metrics and logs springiness developers a feedback loop that improves reliability and accelerates shipping code. Here I focused connected observability signals from build and deployment pipelines for retail-web and retail-api microservices, utilizing cloud-native DevOps telemetry.

Build Pipeline Trends

Images 16 and 17 show metrics graphs showing successful and grounded build trends for some microservices.

For successful builds, we want to spot this statement support a patient level, but if nan successful builds statement drops comparative to nan expected number of builds, it tells america thing important astir nan value of codification being merged aliases nan stableness of our testing environment.

The grounded builds floor plan is simply a timeline of developer frustration. What instantly stands retired is nan ample spike successful nan ray bluish line, representing nan retail-api pipeline. This instantly prompts nan DevOps technologist to ask: ‘What was committed correct earlier that time? Did we present a breaking change, an incompatible dependency aliases a configuration error?

Build Run Duration (P95)

Next, we look astatine nan P95 build tally duration, which tells america nan clip it takes for our codification to spell done nan build and trial process.

What is P95? P95 build long tells america really agelong 95% of builds take, which is overmuch much useful than averages.

In Images 18 and 19 show a accordant shape that grounded builds return importantly longer than successful ones. This signals that failures successful this illustration are happening late, aft lengthy trial runs.

The hole is to “shift left”: Add earlier validation truthful builds neglect fast, redeeming compute clip and improving developer experience.

Deployment Pipeline Trend

Image 20 shows deployment nonaccomplishment counts per hr for some retail-web and retail-api pipelines.

Notice nan retail-web pipeline (dark blue) shows a persistent level of nonaccomplishment pinch chopped spikes. This tells america nan problem isn’t needfully nan codification itself, which passed nan build. This indicates biology aliases configuration issues for illustration image propulsion errors, networking timeouts, missing secrets aliases Kubernetes API throttling.

Observing deployment failures helps abstracted code-level issues from cluster-level problems, which keeps developers focused while DevOps teams debug nan existent bottleneck.

Deployment Execution Time (P95)

P95 deployment long for some services shows precocious variance pinch melodramatic spikes shown successful Images 21 and 22. Instead of a flat, unchangeable line, we spot high, melodramatic spikes and dips.

The clip it takes to deploy our codification is highly unpredictable. It mightiness return 1 infinitesimal connected 1 day, and complete 11 minutes to deploy connected different day.

Investigating logs from nan CI/CD strategy helps place whether it’s nan registry, network, cluster API aliases rollout strategy causing nan slowdowns.

From Raw Logs to Intelligent Insights pinch AI

We’ve tracked our build and deployment failures but diagnosing them from thousands of log lines is painful. This is wherever log clustering becomes a existent accelerator. The level applies instrumentality learning-based clustering to automatically group akin log patterns together.

In Image 23 complete 3,000 earthy log entries from nan retail-web build pipeline were reduced into conscionable 52 clusters instantly highlighting recurring issues and outliers.

Now we tin attraction connected nan large picture: spotting recurring nonaccomplishment signatures aliases configuration errors that mightiness different beryllium buried successful noise.

Taking it 1 measurement further, Image 24 shows really an AI adjunct summarizes these clusters successful plain language. By correlating logs crossed 52 clusters, it tin aboveground probable causes down recurring errors specified arsenic why BUILD_EXECUTION Failed appears aliases whether it overlaps pinch authentication issues, missing limitations aliases pull-rate limits

This does not switch an engineer’s expertise. Instead, it acts arsenic a copilot that accelerates reasoning, highlights relationships crossed log patterns and reduces time-to-root-cause.

Conclusion

With this article my extremity is to springiness readers a practical, end-to-end position of what observability tin look for illustration crossed your application, your Kubernetes situation and your transportation pipelines. I dream nan lessons thief you build systems pinch observability arsenic nan first-class creation principle.

Group Created pinch Sketch.