Dbt Labs Open Sources Metricflow: An Independent Schema For Data Interoperability

Data model supplier dbt Labs precocious released arsenic unfastened root MetricFlow, a SQL procreation instrumentality that reinforces nan dbt semantic furniture via nan Apache 2.0 license. The implications of this improvement span nan furthest reaches of nan information ecosystem.

It reaffirms dbt Labs’ committedness to nan Open Semantic Interchange (OSI) Initiative, an effort led by like-minded vendors specified arsenic Snowflake, Salesforce, Atlan and Alation, to create standards for exchanging information crossed platforms and tools.

Part of nan open-sourcing of MetricFlow includes making disposable its JSON-based metadata layer, which provides a cosmopolitan schema independent from nan definitions and metrics engine. Thus, moreover without adopting MetricFlow, nan unfastened root organization tin still utilize this semantic furniture arsenic a communal speech for knowing information crossed devices and vendors. Organizations tin besides proceed to entree it via MetricFlow.

The open-sourcing of this metadata furniture whitethorn very good beryllium nan cardinal to nan long-awaited interoperability betwixt information systems that galore person longed for, yet fewer person achieved. The main driver for each of these developments is nan request to supply transparent spot successful statistical AI applications — peculiarly those involving move agents and nan occurrence of nan emergent MCP protocol for agents interacting pinch tools.

“There’s 2 ways to deliberation astir semantic layers,” said Ryan Segar, main customer serviceman astatine dbt Labs. “The aged measurement lets you specify thing and gives you an reply erstwhile you inquire for it, but not nan traceability nether nan hood to understand more, for illustration nan JOIN paths, wherever it came from, whether aliases not it’s trusted, and whether it’s been tested.”

“You can’t spend to do that successful nan AI era because erstwhile you’re using an LLM [large connection model], and you’re talking astir MCP [Model Context Protocol], nan measurement to get much meticulous answers is not conscionable to springiness nan surface-level reply of what gross intends and locomotion away. You request to beryllium capable to springiness a intelligibly defined and tested metadata way to nan models.”

Transforming MCP

MetricFlow — and its JSON-based metadata furniture — tin service arsenic nan starting constituent for providing specified granular accusation to agents, nan connection models powering them and to humans monitoring and auditing those agents. Although nan existent take rates of MetricFlow since dbt Labs’ open-sourcing of nan instrumentality person not been intensely scrutinized conscionable yet, nan anticipation of its effect connected MCP’s evolution is very real. Even if nan unfastened root organization only embraces its cosmopolitan schema specification without nan remainder of MetricFlow, it tin perchance toggle shape nan measurement MCP itself functions.

At best, it tin reshape nan protocol from a terminus to a launching constituent for nan knowing and spot basal for enterprises to get nan results they desire from agentic deployments. Via this ideal, “MCP is not conscionable an endpoint that gives you what you want and past is done,” Segar commented. “It’s nan gateway to standardizing really immoderate exemplary thinks astir interacting pinch your information and, much importantly, your metadata.”

The Universal Schema Specification

Realizing this perfect requires much than conscionable MetricFlow aliases its JSON-based semantic furniture that enables devices — including those for Business Intelligence (BI), AI, information warehousing, databases and much — to stock metrics, terminology and definitions pinch 1 another. It requires a translator instrumentality for illustration dbt to facilitate nan provenance for gleaning wherever nan information came from for answers to questions, and conscionable what was done to that information to guarantee that it’s nan correct information to usage for a peculiar exertion aliases query. MetricFlow’s cosmopolitan schema specification, however, is nan launching constituent for tooling crossed vendors, whether that’s Databricks and Snowflake, Power BI and Tableau, aliases thing else, to efficaciously pass pinch each other.

Subsequently, sloppy of wherever metrics were created, users tin input them into this world schema and still understand their meaning crossed vendor ecosystems. According to Segar, this world JSON schema aliases metadata stack functions arsenic “the Rosetta chromatic that’s successful nan middle. It’s nan communal crushed truthful companies don’t person to merge straight pinch each different anymore. They tin merge and adopt this metadata spec that’s communal crossed each of america and that’s what’s going to let them to publication and parse.” If users prime to entree this metadata stack independently of nan remainder of MetricFlow, they tin trust connected metrics that they’ve been utilizing successful a BI instrumentality for years, for example, and still person different devices and products understand nan underlying semantics.

Defining Metrics

Since MetricFlow is now accessible to nan unfastened root community, it’s conscionable arsenic easy to create metrics and their accordant definitions pinch its engine. MetricFlow efficaciously translates those definitions into SQL, pinch each of its ubiquitous benefits passim nan information space.

For example, “You tin specify nan meaning of gross margin, and what MetricFlow does is compile that meaning into SQL,” Segar explained. “That SQL is not conscionable location to opportunity ‘you asked for gross separator and here’s nan answer.’ It understands that if you talk astir gross margin, calendars travel up. So, nan fiscal calendar, really do you grant that and what’s nan logic underneath?” Naturally, organizations tin still avail themselves of nan world metadata modular that’s portion and parcel of MetricFlow, if they like, successful summation to being capable to entree it without nan remainder of nan MetricFlow offering.

Interoperability

The big of usage cases surrounding nan interoperability that becomes imaginable erstwhile implementing MetricFlow’s JSON schema is innumerable. Still, nan astir pressing 1 astatine nan infinitesimal seems to beryllium making statistical AI deployments much trustworthy, reliable and accurate. These benefits look to beryllium redoubled erstwhile applying them to deployments of agent-based AI, peculiarly erstwhile 1 considers that galore are infused pinch LLMs that organizations haven’t trained aliases fine-tuned.

“In this AI world wherever everyone is worried astir accuracy scores and really nan exemplary derived nan answer, you request it to beryllium explainable,” Segar mentioned. “If you want trust, you person to person transparency. Transparency needs to not conscionable beryllium human-readable. It needs to beryllium repeatable and portable truthful that AI tin interact pinch it, and understand, and crawl done really nan metrics are defined.”

Group Created pinch Sketch.