Restack Gives Product Teams The Reins To Own Ai Agent Behavior

Andrés Tapia, co-founder and CEO of Restack, says companies are going astir AI each wrong.

Tapia and his chap co-founder and CTO, Philippe Brulé, first worked together 10+ years agone astatine Mister Spex, an omnichannel optician. After going their abstracted ways, nan brace teamed up again to thief companies wrangle their data: really to move it; really to edit it.

Thus was nan first loop of Restack: a unreality work for information engineers to deploy applications pinch 1 click. For astir 2 years, that was nan ngo — until ChatGPT entered nan segment successful 2022 and wholly upended technology, work, and nan measurement we deliberation astir everything.

Suddenly, Restack’s customers began approaching Tapia pinch caller requests: “‘I’m utilizing Restack for moving information from present to here,’ Tapia paraphrases. ‘We are now starting to build immoderate AI products. Can we usage your level arsenic good for these AI products?’”

Technically, he says not overmuch needed to alteration to suit nan caller demands — but it proved a awesome turning point. Soon, almost 90% of their users were seeking thief pinch AI: “So we said, ‘We’re going to extremity what we[‘re] doing, and we’re going to attraction 100% connected nan AI usage case,” Tapia recalls.

But they weren’t nan only ones moving that way.

So Many New AI Tools — So Many New Mistakes

Just arsenic quickly arsenic ChatGPT and near-ubiquitous AI agents seemingly cropped up overnight, truthful did nan devices to thief build AI products: Langchain, Bercel, etc. And contempt its comparative newness, nan process of building an supplier has quickly go comparatively standard: Define what your supplier is expected to do (for many, that’s customer support); link nan correct devices and information sources (like FAQs); spell live.

“But nan problem is, it doesn’t extremity there,” says Tapia. Once you spell live…, your supplier is answering existent group — and [it] start[s] to hallucinate.”

One healthcare supplier learned this nan difficult measurement earlier coming to Restack.

They had created a customer support supplier to section questions like, “How overmuch is X going to cost?” and “Does my security screen this dentist aliases not?” Handy successful mentation — but not erstwhile astir of nan answers to those questions are wrong.

“They ha[d] a batch of cases where… nan customer complained, ‘I went to a dentist, and actually, this dentist is not covered pinch my policy,’” Tapia explains.

Unfortunately, anyone who’s chatted pinch a bot knows these hiccups aren’t uncommon.

Tapia thinks he’s pinpointed nan guidelines of nan failure. From his perspective, astir companies place nan 2nd — and very important — portion of creating AI agents: testing behavior.

The Part Everyone Misses: Testing Agent Behavior (Not Just Outputs)

“First, you build nan method part. But then…you request to guarantee nan supplier useful decently — that’s [where] we didn’t spot that location was thing successful nan market,” Tapia says — and wherever he says Restack offers thing different.

Companies commencement by building an agent. Then, they tin specify what Tapia calls “behavioral metrics,” qualifiers that spell beyond nan emblematic functional aliases capacity metrics astir devices trust on.

For example, for nan aforementioned healthcare provider, behavioral metrics see questions like, “Is my supplier mentioning my competitor?” “Is my supplier checking nan argumentation earlier it recommends a dentist?”

Unlike nonsubjective signals (like supplier consequence clip aliases solution rate), nan answers to these behavioral questions are subjective and frankincense not easy checked pinch a elemental usability aliases rule-based trial — but they do much to extremity chatbot answers from going sideways.

There’s different furniture of evaluation, too. “We [have] this LLM arsenic a judge,” explains Tapia. “Basically, different LLM is going to cheque if this LLM is really doing nan correct point aliases not.”

And if nan supplier doesn’t meet nan LLM judge’s expectations? Then it’s clip to iterate, i.e., “change nan prompts; alteration nan devices you provide; alteration nan context,” he says.

When AI Goes Live, nan Mistakes Show Themselves

It’s nan behavioral metrics and nan test-and-iterate process that Tapia says astir companies are missing erstwhile they statesman AI projects.

“Every institution wants to instrumentality AI,” he says. “They cognize successful nan backmost of their caput that—somehow—they could prevention money and beryllium much efficient, [but they do] not precisely [know] nan usage case.”

Understandably, a batch of companies are jumping into AI eyes-closed, headfirst because they fearfulness nan oft-touted maxim: “Join in, aliases get near behind.” And merchandise teams are often nan ones nether unit to fig it retired and present — fast.

In a hunt for viable usage cases, customer support usually seems for illustration a earthy fit. The adjacent measurement is assessing disposable frameworks and past kicking it to engineering teams to tinker for a fewer months earlier going live.

That’s erstwhile Tapia says nan mistakes uncover themselves.

During nan first unrecorded trial (say, sending 10% of customer support conversations to nan agent), “People [i.e., customers] commencement to complain,” says Tapia. “And past guidance complains… Then nan merchandise squad sa[ys], ‘This is nan merchandise I ship; I’m responsible for what it’s doing.’”

But erstwhile merchandise teams circle backmost to engineers to inquire what went incorrect and really to hole it, answers aren’t trim and dry.

There’s No Quick Fix

From nan engineering side, everything was correct — correct tools, correct integrations. But nan non-deterministic quality of LLMs intends outputs are still going to vary.

It’s not encouraging. Perhaps that’s why Gartner predicts 40% of agentic AI projects will beryllium cancelled by 2027. Internally, Tapia says discussions often echo nan aforesaid concerns:

“[For companies that…build the[ir] first agent…, location [is] a large chunk of group saying, “AI—bye. It doesn’t work. Let’s extremity it.’”

It’s only pinch these failures, he claims, that teams recognize they can’t measure behaviour pinch a classical Q&A. Namely, Tapia says, they recognize “‘We request to trial successful a different way. We request to telephone nan supplier ourselves, effort to simulate cases, and see… nan capacity of nan supplier and nan behaviour for those questions.'”

But that testing is easier said than done.

“When nan merchandise squad wants to trial that, they find retired they can’t — because everything was built from a constituent of position of package development,” explains Tapia. “The instrumentality is built there. If they want to alteration anything, they request to spell to nan package squad and show them, ‘Change nan prompt. Change this tool. Remove this discourse from nan discourse store.’”

Obviously, a multistep loop process adds bulk and time. But building AI agents is caller terrain, truthful it makes consciousness that companies revert to known processes for package development.

There’s nan catch, Tapia points out. He argues that building AI agents isn’t a package improvement challenge; it’s a merchandise challenge. And until companies alteration their approach, he predicts continued nonaccomplishment ahead:

“More companies [are going to] jump into AI and beryllium unhappy pinch nan results…because, basically, they didn’t understand that AI is not nan aforesaid arsenic building a package solution.”

Stop Treating AI Like Software; Treat It Like an Employee Instead

So, really should companies deliberation astir AI implementation?

“You person to deliberation astir AI [the same] arsenic you deliberation astir caller labor successful nan company,” Tapia advises.

That intends onboarding, training, and regular improvement truthful that, complete time, some parties tin amended understand what needs to beryllium done. This behavioral refinement, he claims, is nan cardinal to occurrence — and what different devices overlook.

Done nan Restack way, merchandise teams are empowered to ain AI supplier behaviour and trial and vessel it for illustration normal merchandise features.

“You build your supplier nan measurement you build it, but nan merchandise squad is going to person a furniture of features…so they can—by themselves successful our personification interface—iterate and alteration those prompts, tools, everything—without needing to spell to nan tech team.”

It’s a displacement from nan default AI deployment processes — 1 that gives merchandise teams much power complete supplier behaviour and possibly yet brings it successful statement pinch personification expectations.

Group Created pinch Sketch.