We're building Postgres in Rust. Using the LLVM of databases

Today we are announcing the beginning of a new and exciting journey: we will write a modern version of Postgres, in Turso (which is itself written in Rust). Some of you reading this phrase will think it makes absolutely no sense: what does it even mean to write Postgres in Turso? Are you drunk? (the answer in the case of Pekka is likely yes, but that is irrelevant). This will make sense soon.

Not to bury the lede, here’s the short version: Turso is becoming the LLVM of databases. One modern, reliable core; many database frontends compiled down onto it. SQLite was the first frontend. Postgres is next. Others will follow.

But the end game is clear: we want a new database, compatible with Postgres, written from the ground up with a modern architecture (not using processes for connections, able to be run embedded in the browser, as a file, with self-updating materialized views, etc). And we want you to build this with us, in the best possible spirit of Open Source. We have done this successfully many times, and we are confident we can do it again.

What we are announcing today is the first step. That step already goes quite far. In this post we will explain the architecture, what works today, the current limitations, and what we believe the future will look like.

#What is was Turso ?

Some of you may be familiar with Turso: Turso is, or was until recently, exclusively a full rewrite of SQLite in Rust using a modern architecture. It is file-compatible with SQLite, meaning it can open and generate SQLite files. But it has a couple of very interesting features: It supports concurrent writes using MVCC (like Postgres), it has a rich type system (like Postgres), it has support for Materialized Views (unlike Postgres, those views actually auto-update!), it is fully asynchronous, meaning it runs natively very well on browsers and other challenging environments, and the list goes on.

Let’s pause on that materialized views one, because we said it too casually. It works now, and it showcases exactly how we aim to be much more than just a 1:1 translation of Postgres to Rust. If you have ever babysat a REFRESH MATERIALIZED VIEW cron job, or faked live views with a pile of triggers, read that sentence again: our views update themselves, live. This is the feature Postgres users have quietly wanted for twenty years.

Turso is also written with extreme reliability in mind: it is tested using a Deterministic Simulation Testing suite, Antithesis testing, Oracle testing, Fuzzying, and Formal methods. One of the features of SQLite that we love the most is how rock solid it is, and we wouldn’t be able to do it justice if we didn’t replicate that one feature! Yes, reliability is a feature.

More importantly, Turso is an Open Contribution project: we have over 260 contributors today, with an extremely healthy Open Source community. AI hasn’t changed our stance on Open Source a bit. In fact we have doubled down: we welcome AI contributions, as long as they are *good* contributions. We, the creators of Turso, spent the first 10 years of our careers working in the Linux Kernel. And we have then spent the rest of our time trying to replicate that same environment elsewhere: a place where code speaks, the bar is high, and you can - through grit and work - have a seat at the table, and help shape the *direction* of the project (not just fix things here and there). Turso is already such a place, with a large body of work coming from total strangers (which often become good friends!), albeit at a smaller scale than Linux (at least for now)

#The truth about SQLite

SQLite is, very deservedly, a piece of technology that users love to love. I fell in love with it over 20 years ago, and I am proud to have built so much on it. But there are lots of things about SQLite that, in our experience, people just don’t know. For example: SQLite doesn’t accept external contributions, and its legendary test suite is proprietary.

Another thing many don’t know is that in a sense, if you squint, SQLite is not really a database: SQLite is a Virtual Machine. Yes, you have heard this right: SQLite (and we also adopted this design in Turso), has a very unique design: it compiles SQL (in the SQLite dialect) to its own bytecode language, called the VDBE. That is not a general purpose language, like you would get from the JVM, .NET or WASM: it is a database-specific language with higher level operations like “find something in a B-tree”, but it is a bytecode language nevertheless.

It works more or less like this:

This is really an implementation detail, and the language is not exposed or specified, nor is it modularized. But it is there.

#It really is a virtual machine!

I know this must sound hard to digest. You are probably thinking we are exaggerating or out of our minds. After all, if this was truly a VM language, you would be able to run Doom on it. Right? RIGHT?

Not only the Turso VDBE is sufficient to run Doom, since Turso runs in the browser, we don’t have to show you a video. Here’s the freaking demo itself, in all its glory: the actual Turso engine, compiled to WASM, running Doom as VDBE bytecode right here in this tab. Click to play.

That is real Doom. We wrote a small C to VDBE compiler, and the result is executed by the database exactly the way it runs a SELECT. Each frame is a result row. Memory areas are represented by blobs that are read from and written to. If you want to read more, you can dig deeper in this example repo.

#What does it mean to write Postgres IN Turso?

Many people before us tried to marry SQLite and Postgres. Since SQLite is a monolith, this is usually done at the query layer, by translating Postgres queries into SQLite. But this has a very large impedance mismatch: on the outside, the databases are just too different.

But the irony is: On the *inside*, the databases are not actually *that* different. Look close enough, and every SQL database is just a fancy collection of B-Trees with a bunch of Indexes.

Now yes, yes, we can hear the Postgres purists already typing. Actually, Postgres keeps rows in heap files with separate index B-trees, tuple versioning, TIDs, and vacuum, while SQLite clusters everything into the B-tree itself. All true. But those are differences in how the bytes are arranged on disk, not in the fundamental operations a database performs. The things that Postgres needs to do are either already representable in the VDBE, or we could extend the VDBE to do it (like we extended it for features that go beyond SQLite).

So we attempted a new design: Since Turso goes beyond SQLite, what if we made the frontend pluggable? What if we could turn Turso into the LLVM of databases? Parse the Postgres language into a common AST, compile that AST to the Turso bytecode… and… do we have Postgres?

We do! A while back, we brought this crazy experiment to life through a project called pgmicro. Our goal was to prove to ourselves that this could be done. And we are delighted with the results. Now, it is time to make it official. This is an official project backed by Turso. The initial experiment is now fully merged into the tree, and it is time to get our weight behind it.

To be clear about where we are: today this is a foundation, not a finished product. But it is a real, running foundation. We gave the world a new, improved version of SQLite. It is time to do the same for Postgres.

We will talk about the trade offs in a FAQ at the end of this article. But first, some closing remarks

#The LLVM of Databases

I know what you are thinking: Could I do MySQL? And the answer is yes. Could I do Redis? And the answer is also yes. What about my crazy experimental database that I have been thinking of? OH YES.

And here is where it stops being an architecture diagram and starts being something you can feel. Picture it: a real Postgres running in a browser tab, with no server anywhere behind it. An agent spinning up a fresh Postgres database per user, a billion of them, each one just a file. A local-first app that syncs its Postgres down to a single file and back up again. Same database, same SQL, none of the operational weight.

This idea is so powerful, that from this day forward, we will speak of Turso as much more than a rewrite of SQLite: We are now the LLVM of databases. Yes, we are a rewrite of SQLite, and yes, we will be a rewrite of Postgres. And given the constraints of time and energy that still exist even in the post-singularity world, that’s all we personally intend to do in the near future.

But the door is open. And we’d love to have you with us

#Run it, and come build with us

That’s the whole idea: turn the core into the LLVM of databases, and let the frontends bloom. Postgres is the one we’re building. The next one could be yours.

There are no published packages yet, but you can build and run it today, straight from the source:

cd postgres/cli && cargo run

The door is open. Come build it with us.

#FAQ

What does this mean for Turso as a rewrite of SQLite?

Nothing really changes for Turso/SQLite. Our goal is still to be fully compatible with SQLite, and extend its set of capabilities for the workloads of tomorrow. If anything, this is easier now because we can rely on Postgres tests to implement increased functionality when needed and ship things faster.

Will Turso/Postgres also be fully, 100% compatible with Postgres?

Unlike SQLite, our current belief is that we need to be compatible enough, especially at the core functionality, but not really 100%. Postgres is a much more complex target. Some of the architectural changes we should make may pose the question of whether or not to keep things 1:1 and the answer may vary. Our Materialized Views, for example, are live. It is unclear at this point if it makes sense to implement Materialized views that are *not* live just for the sake of perfect compatibility. True to Paretto, we do believe that the vast majority of users using the majority of common features should be able to run their applications unmodified. But ultimately, especially insofar as it adds more features, Turso will be its own thing. Our real ambition is to build the future of what Postgres could be in the LLVM of databases.

Will my existing apps, ORMs and psql just connect?

Yes. Although we have highlighted the cool new things you can do by running this in-a-file, Turso/Postgres will also include the wire protocol and a server implementation.

Does that mean that we’ll be able to run Turso/Postgres like SQLite? Embedded, in phones, devices, etc?

Yes.

Turso has features that allow databases to be synced to a local file. Will that be possible with Turso/Postgres too?

Yes.

What is your stance on using AI?

We have written about that extensively before. We like AI tools and we have been AI-pilled from the beginning. But we insist on reviewing code closely, and understanding architecturally what is being done. Perhaps one day tools will be good enough so that this doesn’t have to be done, but we do not believe we are there yet, especially for a component as critical as a database.

Doesn’t that mean you will move slower than other people just running Fable 24x7?

Sure. Slower per commit. But we are not starting from a blank canvas. We are starting from a database that already works, is already tested half to death, and already shares most of Postgres’ guts. Slow-and-correct is exactly why we will still be here, and still be right, in three years. A database is not the place to move fast and break things.

But how would you implement PL/pgSQL? That sounds impossible!

It is not impossible. It is just a procedural language. PL/pgSQL is also heavily criticized by many people. We are likely to implement *something else*, and then a compatibility layer to expose PL/pgSQL. If you have ideas on how to make this awesome, remember: Turso is Open Source and Open Contribution!

But what about Postgres extensions?

We have a proof of concept of loading some Postgres extensions using a WASM container. There is a performance hit, but it allows arbitrary extensions to be loaded as long as they are using the symbols that the WASM container provides. We do not guarantee that this is the direction we will take, but looks promising. Ultimately, we believe that an extension system that allows *any* arbitrary code to be executed is a bad idea, and in practice some extensions will never load. But in practice most will, if compiled to WASM.

It sounds like this will take forever!

Way less than you think. The key realization is that we’re not starting from scratch: the whole point of this article is to show that we are starting from a working, reliable, tested SQL database that bears a lot of commonalities with Postgres, and turning our core into the LLVM of databases.

One thing is true, however: if you look at the development history of Turso, you will see that for us, doing it right trumps doing it fast. This is not a pet project for us. It is the infrastructure that we believe will power the applications of the future. Even in the so-called “age of AI”, doing things right takes time. Features have a long tail, some need some time to simmer until we reach the right architecture. It may take some time to get *every* feature of Postgres implemented. But both Glauber and Pekka are around 44 years old now. We have at least 20 more years to do it and we’re not in a hurry. In practice, we believe we can get to somewhere that implements a large enough subset of Postgres to make this very useful in practice within a couple of months. It also depends on you: when we implemented multi-process support for the SQLite dialect, for example, we didn’t implement it for Windows because none of us have Windows machines. Within a week, one developer that is a heavy user of Turso and uses Windows took an interest and nailed it. We believe very strongly in the power of Open Source, and we believe that our already strong community will get even stronger around this mission.

Will this be licensed under a permissive license?

Of course. While we expect some more experimental frontends to be developed out-of-tree, the Postgres frontend will be developed in-tree as an official frontend from day 1. It is therefore subject to Turso’s existing license, the MIT.