How Are Opentelemetry And Fluent Bit Related?

Editor’s Note: The pursuing article is an excerpt from Chapter 1 of The Manning Book: “Fluent Bit pinch Kubernetes,” a guideline to configuring Kubernetes logging, metrics and trace pipelines. This excerpt focuses connected Fluent Bit’s narration to OpenTelemetry and its improvement successful capturing not only logs but besides section metrics specified arsenic CPU, representation and retention use. To analyse much intimately really Fluent Bit and OTel execute different functions, download nan afloat book.

How OpenTelemetry Came to Life

Before OpenTelemetry (OTel), nan superior specifications that informed nan observability of metrics, traces and logs came from respective standardization efforts wrong Cloud Native Computing Foundation (CNCF) successful nan shape of OpenTracing, OpenCensus and — implicitly, fixed its power — Fluentd, on with, by association, Fluent Bit for nan building of logging.

Different standards often required different devices to seizure specified data. Fluent Bit has ever caught immoderate metrics data; nan Internet of Things (IoT) ecosystem needs to support package footprints small, truthful 1 work capturing some logs and metrics is preferable. As a result, it made consciousness for Fluent Bit to seizure not only logs but besides section metrics specified arsenic CPU, representation and retention use.

Bringing each these information sources together has driven nan simplification of operational monitoring, resulting successful accelerated uptake and proving to beryllium disruptive. Fluent Bit’s support of nan OTel standards and its expertise to activity wrong nan OTel ecosystem haven’t required immoderate extremist changes, though they person driven immoderate upgrades to parts of its implementation. In immoderate respects, nan upgrades person formalized what Fluent Bit was already doing. With this alignment, Fluent Bit is well-equipped to support nan adoption of OTel standards without imposing them, allowing its take to beryllium much incremental.

Learnings from Later Manning Book Chapters

In early chapters, erstwhile we commencement digging into nan input and output capabilities of Fluent Bit, we’ll look further into nan narration pinch OTel and starring products successful nan observability space, specified arsenic Prometheus, which has helped propel OTel further forward, and Grafana. We’ll besides look astatine commercialized vendors that person worked to support OTel’s standards, creating a quickly increasing ecosystem of connectable monitoring tools.

Note: If you request a speedy reference connected nan acronyms and terminology, you tin find a useful glossary here. Also, Appendix B lists respective fantabulous resources. (Download nan book to position Appendix B.)

How OTLP and OTel Fit Together

The bosom of OTel is nan OpenTelemetry Protocol (OTLP), which specifications nan information structures, encoding and transmission of nan telemetry data.

Currently, OTLP supports transmission utilizing distant process telephone (gRPC) with HTTP/2 utilizing Protocol Buffer (Protobuf) and JSON pinch HTTP synchronously. OTLP promotes nan usage of gRPC arsenic nan first-choice attack to connection and JSON arsenic a step-down aliases fallback.

OTel, arsenic a project, goes acold beyond defining OTLP. It besides provides implementations of nan functionality described successful nan modular (sometimes described arsenic a reference implementation), on pinch devices and libraries.

The devices and libraries are implemented successful aggregate languages; we tin usage them to thief inject logic into applications and quickly get information applications producing traces. OTel besides has functionality, specified arsenic log appenders, that let logging frameworks to nonstop nan logs utilizing nan OTLP specification.

How Fluent Bit Fits successful an OpenTelemetry Solution

To understand really Fluent Bit could fresh into an OTel solution, let’s look astatine what Fluent Bit tin do utilizing OTel terminology:

• Given its expertise to stitchery monitoring and observability information from different sources and toggle shape it into nan OTLP structure, Fluent Bit tin capable nan domiciled of an OpenTelemetry Collector.
• Because Fluent Bit was built to activity successful a distributed situation and tin walk information successful OTLP format to immoderate different OpenTelemetry-compliant collector (which could beryllium a Fluent Bit node aliases different product), we tin picture Fluent Bit arsenic being capable to execute arsenic an OTLP Exporter.

Figure 1 shows really Fluent Bit tin fresh into an OpenTelemetry situation pinch its expertise to grip logs (L), metrics (M) and traces (T) generated by an exertion pinch aliases without nan thief of OTel libraries aliases tools, on pinch its expertise to interact pinch an OpenTelemetry Collector.

Collector vs. Exporter: Choosing nan Right Role for Fluent Bit

Because OTel provides implementations of collector and exporter capabilities, calling Fluent Bit an OpenTelemetry Collector aliases OpenTelemetry Exporter tin beryllium a root of confusion. The modular itself is called OTLP, truthful referring to Fluent Bit arsenic being OTLP-compliant is clearer, moreover if little evident astir nan task we mightiness deploy Fluent Bit to perform.

In addition, location is immoderate sensitivity wrong nan OpenTelemetry organization astir nan quality betwixt nan project’s ain implementation of a collector (called OpenTelemetry Collector) and different implementations of that capability. We are erring connected nan broadside of describing Fluent Bit arsenic an OTLP Collector (after all, protocol compliance is cardinal to nan collector’s function) and reducing ambiguity among CNCF projects.

Protocol Buffers (Protobuf)

Protocol buffers are a cardinal exertion for gRPC, which OTel uses. Protocol buffers person a concisely defined schema, which is utilized pinch nan Protobuf tooling to make nan codification for sending and receiving payloads. A well-defined schema allows nan tooling to create nan codification that creates a compressed binary payload representation. This schema is some a spot and a imaginable constraint.

The spot comes from nan businesslike payload transmission. The downside is that a schema alteration affects some nan supplier and user and makes realizing nan tolerant scholar integration shape much challenging. Also, fixed that nan Protobuf-generated payload is simply a compressed binary format, it is simply a batch harder to inject into immoderate connection middleware that tin accommodate transformation. Links to OTel, Protobuf and related technologies are successful Appendix B.

Looking Ahead: Explore Fluent Bit and OTel Even Further

As we advancement done nan “Fluent Bit pinch Kubernetes” book, which tin beryllium downloaded successful its entirety here, we’ll analyse much intimately really Fluent Bit and OpenTelemetry execute different functions. Note that OpenTelemetry protocol support earlier Fluent Bit v3 was restricted to HTTP and JSON. Version 3 brought enhancements that support HTTP/2, enabling Fluent Bit to usage gRPC. This, successful turn, intends that Fluent Bit tin supply a afloat compliant OTLP implementation without needing to return advantage of nan measurement down to HTTP and JSON.

Group Created pinch Sketch.