Fixes Required For Prometheus’ Opentelemetry Integration

We person covered really location person been immoderate conflicts betwixt OpenTelemetry and Prometheus compatibility for a number of reasons. Much of this has to do pinch really Prometheus has been — and remains — a very tried and trusted unfastened root metrics solution that preceded what OpenTelemetry has offered arsenic an alternate measurement of standardization and different attributes that decidedly radiance for observability.

During PromCon, Prometheus’s yearly personification convention held precocious successful Munich, Julius Volz, a co-founder of Prometheus, offered circumstantial insights during nan opening keynote into really utilizing OpenTelemetry pinch Prometheus still poses problems and what needs to beryllium done to reside those issues.

First, nan bad: There is simply a basal nonaccomplishment of work find and progressive propulsion erstwhile utilizing OpenTelemetry, arsenic good arsenic nan complexity of OpenTelemetry’s SDKs. Performance issues abound arsenic well. In 1 benchmark trial — which RevCom has not verified — Volz noted location was up to a 22 times velocity quality successful Go benchmarks erstwhile OpenTelemetry instrumentation is utilized compared to autochthonal Prometheus instrumentation.

Obviously, improving capacity is key, successful peculiar for codification meant for observability, which runs very often and is meant to thief amended performance. But semantic conventions besides airs problems, arsenic we discussed successful our erstwhile articles. There is besides a continued request for collaboration betwixt nan Prometheus and OpenTelemetry teams.

“I did want to constituent retired nan drawbacks of [integrating OpenTelemetry pinch Prometheus], and possibly make you deliberation about, ‘OK, do we really want to spell down this way if you mostly attraction astir metrics and utilizing them pinch Prometheus?’” Volz said. “Also, successful position of nan work find and SDK slowness drawbacks, I would really for illustration different group to deliberation astir that much and possibly amended things, to not conscionable propulsion distant nan babe pinch nan bath h2o and suffer each these benefits that we built very cautiously successful Prometheus complete a agelong time.”

There person been galore improvements since Richard “RichiH” Hartmann started a general effort to amended interoperability betwixt nan 2 projects successful 2020 (his business title is head of organization astatine Grafana Labs).

While OpenTelemetry’s deficiency of work find will ever create other activity for operators, astir of nan basal problems person been resolved, Hartmann told maine station conference. OpenTelemetry has changed, arguably fixed, their histogram bucket definitions successful favour of Prometheus, Hartmann said. “Prometheus past expanded support for information labels to support OpenTelemetry,” Hartmann said. “And some projects collaborated early connected autochthonal histograms based connected nan activity of Björn “Beorn” Rabenstein, starring to afloat compatible releases connected some sides.

Meanwhile, OpenTelemetry is “here successful afloat unit and is not going to spell away, and organizations activity to usage it pinch Prometheus, instrumenting their services pinch OpenTelemetry and past sending nan metrics portion to nan Prometheus system, Volz said. “However, location are plentifulness of downsides successful comparison to utilizing Prometheus’ ain autochthonal instrumentation customer libraries,” Volz said. “It is important to beryllium alert of these earlier choosing nan OpenTelemetry way if 1 mostly cares astir metrics and Prometheus.”

A speedy opposition betwixt nan 2 systems shows that Prometheus is an full monitoring system, focusing only connected nan metrics awesome type. In contrast, OpenTelemetry “only cares about” generating nan signals — including logs, metrics, traces and profiles — and past passing them connected to immoderate benignant of third-party backend system, Volz said. This corresponds to really OpenTelemetry’s creators’ extremity is to standardize connected nan emanation facet and reflects nan galore different retention vendors represented successful OpenTelemetry.

The transportation pieces coming a cardinal difference: OpenTelemetry uses OTLP to nonstop those metrics via push, while Prometheus uses a text-based format and actively pulls them, Volz said. Sending metrics via an OpenTelemetry collector to an OTLP receiver endpoint successful a Prometheus server introduces respective drawbacks, Volz said.

Losing Active Pull and Health Monitoring

The first and astir unfortunate downside is nan enactment of throwing distant a batch of what makes Prometheus bully and capable: nan integration of work find pinch a pull-based progressive target monitoring, Volz said. Prometheus solves this by talking to systems for illustration nan Kubernetes API server to get an always-up-to-date view. It past actively tries to propulsion aliases scrape metrics, signaling an up metric pinch a worth of zero aliases one, which is basal for target wellness alerts, Volz said.

Since OpenTelemetry doesn’t person a built-in installation for this functionality, it becomes harder to show if a target is moving but not sending metrics, aliases if it is down, Volz said. Conversely, information controls are bypassed erstwhile unexpected metrics are pushed. “A batch of group disregard this completely, treating their Prometheus server arsenic a random receptacle of metrics, and past they do not cognize if a process that should beryllium moving is not,” Volz said. “The thought of a synthetic up metric for OTLP ingestion has been heard, but it does not beryllium yet.”

The 2nd downside is nan resultant changed metric names aliases somewhat “ugly PromQL selectors,” Volz said. OpenTelemetry introduces characteristic group differences, allowing characters for illustration dots and slashes that were not supported antecedently successful Prometheus 3. “This suggests that nan group standardizing OpenTelemetry did not highly prioritize really a metric would beryllium utilized successful a query connection for illustration PromQL,” Volz said.

Prometheus conventions adhd suffixes for some units and types of metrics to instantly explain nan meaning. OpenTelemetry, however, says “don’t put portion and type into nan metric name.” As a result, nan Prometheus ingestion furniture adds backmost those suffixes during translation. With nan extended characteristic set, PromQL selectors go much analyzable and little easy to constitute and publication than nan autochthonal selector, Volz said.

Indeed, OpenTelemetry and its SDKs are rather analyzable and tin beryllium rather slow. Benchmarking successful Go showed that autochthonal Prometheus customer libraries are up to 22 times faster than nan OpenTelemetry SDK for antagonistic increments, arsenic mentioned above. “Even adding 2 labels makes nan OpenTelemetry SDK 90% slower,” Volz said. “OpenTelemetry’s complexity is baked successful and is difficult to remove, making it nan XML aliases CORBA of telemetry, because it attempts to lick each problems astatine once.”

Roll up Sleeves

To reside wellness checks, early activity whitethorn impact a synthetic up metric for OTLP ingestion, Volz said. This characteristic would usage work find and correlate expected information pinch incoming information to make an up metric erstwhile information is missing, Volz said.

“On nan metrics side, nan Prometheus squad is actively trying to amended things, including creating an experimental Delta to cumulative processor to support OpenTelemetry’s Delta temporality,” Volz said. “There is besides a recognized regret for not having semantic conventions successful Prometheus onshore from nan beginning, suggesting early collaboration pinch OpenTelemetry group could make consciousness to present a akin standardized naming structure.”

Group Created pinch Sketch.