Ai Agents Create A New Dimension For Database Scalability

Everyone wants their database to scale. In fact, if you are aged enough, you mightiness moreover retrieve erstwhile saying your database was web scale was each nan rage.

In principle, “scale” conscionable intends nan expertise to do more. But much what? More data? More queries? Talking astir standard only makes consciousness if we specify nan axis connected which scaling takes place.

It’s clip to move beyond reasoning of databases successful position of conscionable capacity and throughput. The emergence of AI agents requires a caller axis of scalability: How galore databases tin you create and maintain? Multitenancy is ending; nan property of hyper-tenancy has begun.

Sponsor note: If you’re into database scalability, there’s a full convention for that! Monster Scale Summit is simply a free + virtual convention connected utmost standard engineering, pinch a attraction connected data-intensive applications. Join your peers and study from luminaries for illustration antirez, creator of Redis; Camille Fournier, writer of “The Manager’s Path” and “Platform Engineering”; Martin Kleppmann, writer of “Designing Data-Intensive Applications” and much than 50 others, including engineers from Discord, Disney, Pinterest, Rivian, Datadog, LinkedIn and UberEats. Register and subordinate nan organization for immoderate lively chats.

Changing Dimensions for Scalability: Two Examples

This surely wouldn’t beryllium nan first clip that caller technologies triggered america to rethink scalability.

Around 2005, booting your emblematic Linux distribution took much than a minute. The kernel unsocial took dozens of seconds to initialize and, aft that, myriad services had to beryllium brought online. That was a mostly sequential operation, driven by handcrafted bash scripts (SysV Init!).

There were galore efforts to trim that time, led by engineers (myself included) who simply couldn’t guidelines it. It offended our engineering sensibilities. You cognize engineers — if thing tin beryllium fast, why would we fto it beryllium slow?

But location was nary existent unit to amended this metric, nary existent business need. As a result, galore successful nan organization saw those efforts arsenic futile. After all, successful server hardware, nan firmware sanity checks unsocial sometimes took much than that. And moreover connected desktop PCs, really often do you footwear your machine anyway?

Everybody wanted Linux to beryllium accelerated and wanted Linux to scale, but that meant executing circumstantial strategy calls fast, being capable to tally multiprocessor systems, etc. Time to footwear was ne'er a portion of nan scalability axis.

What changed nan calculation was nan emergence of virtualization, nan exertion that led nan unreality revolution. Machines were nary longer beingness entities; they were logical entities that were created connected demand. Initially, those machines conscionable played nan aforesaid domiciled arsenic nan aged beingness ones, isolated from overmuch cheaper and pinch much flexibility. But that soon started to change.

Taking advantage of nan caller capabilities afforded by nan cloud, developers started moving each sorts of ephemeral machines: They would travel online, execute a circumstantial task, past spell away. Suddenly, you had an service of those machines alternatively of nan 1 large server. The business unit to make nan operating strategy footwear clip a portion of nan scalability matrix was abruptly real. Today, for immoderate specialized applications, particularly those being deployed successful unikernels, nan clip to footwear tin beryllium astir a twelve milliseconds.

An moreover much melodramatic illustration comes from nan hardware itself. The first commercialized transistors were astir 10mm successful size. That is, by each worldly standards, small. But location was nary meaningful unit to make it moreover smaller.

Engineers moving connected transistors wanted to make them better, but nan axis connected which “better” was judged was conscionable different. For example, 1 could measurement really nan switching velocity (how accelerated nan transistor could move a high-frequency signal) aliases nan sound fig (the magnitude of sound introduced successful nan circuit by this element). But really physically mini they were? Not a factor.

Size only started to go a interest pinch nan emergence of integrated circuits, nan grandfather of modern CPUs (and now GPUs). As miniaturization became an important aspect, nan transistor size became a cardinal business concern. Today, a modern CPU holds much than 200 cardinal transistors, each of them successful nan bid of a mates of nanometers — 1000 times smaller than their predecessors.

A New Scalability Axis for Databases

In each case, nan shape is identical: A technological disruption fundamentally changes nan measurement we interact pinch and deliberation astir an existing building block. Suddenly, a caller axis of scalability emerges.

Databases are facing akin pressures today, driven by nan emergence of agentic systems. Up until recently, nan word “database” was commonly preceded by nan definite article: “the database.” You constitute an exertion and that exertion has a database that goes pinch it. Scaling “the database” traditionally meant 1 of 2 things: either shop much information aliases execute much queries.

But nan emergence of AI agents changes nan game. Agents are spun up by nan millions. Each supplier is responsible for a portion of a larger task, pinch immoderate agents having nan task of coordinating nan activity of their subagents. Some agents beryllium for conscionable a mates of seconds. And each of them person information needs that are backstage to nan agent. Which devices were called? Did nan instrumentality succeed? What was nan consequence of nan past instrumentality call? Which files were generated? Which files, information and metadata tin beryllium added to nan discourse of this supplier to amended its performance?

Traditional multitenancy breaks down erstwhile you request microsecond provisioning and strict information isolation guarantees. This calls for hyper-tenancy, a overmuch finer-grained isolation pinch a batch much elastic scalability.

Agent builders want this information to beryllium quickly disposable during nan supplier operation, arsenic good arsenic summarized for later usage for auditability and observability reasons. Some agents woody pinch delicate information that must beryllium encrypted, isolated and ne'er time off nan discourse of that 1 agent.

Consider a customer work AI handling a analyzable request. It spawns 1 supplier to cheque inventory, different to verify nan customer’s acquisition history and a 3rd to cipher refund amounts considering existent promotions. A coordinator supplier manages these three, while a compliance supplier logs each determination for regulatory audit. Each exists for a mates of seconds. Each needs to way its state, cache its discourse and grounds its instrumentality calls. Once a caller personification opens a caller ticket, a full caller postulation of agents gets created. Now ideate that serving millions of users. And that is for a azygous strategy successful an organization.

Databases do each of that — and person been doing this for nan past 5 decades. But nan clip it takes for a database to travel online, and nan number of databases you tin support astatine once, was ne'er a pressing interest — until now.

Databases will look nan unit to supply trillions of instances, moving independently. Picture a fleet of hundreds of agents serving millions of users and now multiply that by each nan companies crossed nan economy.

Traditional multitenancy — sharing 1 database pinch logical separation — breaks down erstwhile you request microsecond provisioning and strict information isolation guarantees. This calls for hyper-tenancy, a overmuch finer-grained isolation pinch a batch much elastic scalability.

Databases will look nan unit to supply trillions of instances, moving independently. They will germinate from being deployed to being spawned. If nan “trillions” bid of magnitude sounds absurd, conscionable look astatine our erstwhile examples and ideate we are surviving astatine nan opening of nan agentic curve. Picture a fleet of hundreds of agents serving millions of users and now multiply that by each nan companies crossed nan economy.

The Right Tool for nan Job

As it turns out, a database that tin beryllium spawned by nan trillions already exists: SQLite. On its website, SQLite claims a full of trillions of installations worldwide, making it nan astir deployed database successful nan world. It tin do this because it doesn’t travel nan accepted client-server model: It is conscionable a file, accompanied by an in-process room code. SQLite has besides inspired a procreation of different in-process databases pinch nan aforesaid model, for illustration Turso and DuckDB.

To understand why this exemplary useful truthful well, see what an supplier really needs from its local, backstage database:

• Instant availability: When an supplier spawns, its database must already beryllium there. Not “available aft relationship establishment,” but there. The infinitesimal you person a record handle, you person a database.
• True isolation: Each agent’s information must beryllium abstracted from each different agent’s data. Separate tables could work, but past you would request billions of tables. Separate database files let for difficult isolation betwixt supplier data, pinch nan added use of being easy to encrypt information for different agents pinch abstracted encryption keys.
• Co-location: The database must unrecorded wherever nan supplier lives. If nan supplier is successful a container, nan database is successful that container. If nan supplier runs successful a browser, nan database runs successful that browser. If nan supplier moves to a different machine, nan database moves pinch it. Data and computation recreation together. As agents get deployed everywhere, databases must travel them everywhere.
• Zero coordination: An supplier shouldn’t request to discuss pinch a database cluster astir connections, transactions aliases assets limits. It owns its database completely. No relationship pools to exhaust. No different tenants to impact performance. No shared fate.

Client-server databases simply can’t supply these properties, nary matter really well-optimized they are. In-process databases can.

The Gap

But here’s nan challenge: SQLite was designed for a different era and has been slow to evolve. It’s superb astatine what it does, but agents request capabilities that spell beyond what it tin do. They need:

• Vector search: Agents activity pinch embeddings constantly. Semantic hunt complete documents, similarity matching for Retrieval-Augmented Generation (RAG), clustering of results. SQLite has nary autochthonal knowing of vector operations. You tin bolt connected extensions, but they’re not first-class features integrated into nan query planner.
• Native encryption: SQLite tin encrypt data, but it requires extensions and cardinal guidance becomes nan developer’s problem. Agents dealing pinch delicate information request encryption that’s transparent, pinch keys that unrecorded and dice pinch nan supplier life cycle.
• Observability: When you person billions of database instances, you can’t SSH into each 1 to debug. You request system logging, metrics and traces that aggregate automatically. SQLite is silent, optimized for embedded contexts wherever observability was personification else’s problem.
• Network capabilities: While nan halfway database should beryllium embedded, agents often request to sync state, replicate to durable retention aliases coordinate pinch different agents. SQLite is purely local. The infinitesimal you request to move information betwixt machines, you’re connected your own.
• Developer abstractions: Agent builders aren’t database administrators. They request high-level APIs that grip communal patterns: “Give maine a database for this agent,” “Sync this information erstwhile you can,” “Clean up everything erstwhile I’m done.” SQLite gives you SQL and record handles. You build everything else.

The New Generation

Looking astatine nan past for lessons connected what to expect for nan future, I judge we will spot nan emergence of a caller category: databases that inherit SQLite’s embedded architecture, but germinate it for modern needs. These systems are forged to beryllium building blocks for larger systems, not monolithic solutions. They supply nan primitives that supplier frameworks need: instant instantiation, built-in encryption, vector-native operations and automatic synchronization.

Databases for illustration DuckDB fresh nan bill. DuckDB, often dubbed “SQLite for OLAP,” is an in-process online analytical processing (OLAP)-oriented database that provides local, actionable intelligence for agents. But there’s besides a request for an online transaction processing (OLTP) balanced that will play nan aforesaid domiciled arsenic nan original SQLite.

That’s what led to nan creation of Turso, an OLTP-oriented database that is afloat file- and API-compatible pinch SQLite. It’s a afloat modern rewrite successful Rust that intends to supply a much modern and feature-rich telling of what SQLite tin beryllium (with autochthonal encryption, alteration information capture, vector search, among others), while keeping nan file-based quality of SQLite.

I, for one, americium excited astir nan future!

Group Created pinch Sketch.