---
type: WebPage
title: [FREE] Databricks introduced a new database that could c
description: ""
resource: https://vutr.substack.com/p/databricks-introduced-a-new-database?utm_source=share&utm_medium=android&r=59q6qa
tags: []
timestamp: 2026-09-06T23:00:49.406486Z
---

[![VuTrinh.](https://substackcdn.com/image/fetch/$s_!2JXp!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb91e59e-4df2-4ac8-b3b4-1f4e830f7b42_256x256.png)](/)

# [VuTrinh.](/)

SubscribeSign in

# [FREE] Databricks introduced a new database that could change everything (?)

### Lakebase

[![Vu Trinh's avatar](https://substackcdn.com/image/fetch/$s_!UWAa!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4805f673-db97-4f7c-85c4-44b345a8de80_256x256.png)](https://substack.com/%40vutr)

[Vu Trinh](https://substack.com/%40vutr)

Sep 05, 2026

25

3

2

Share

> *Reminder: I’m offering a limited-time **50% discount** on the annual plan:*
>
> [50% OFF FOREVER](https://vutr.substack.com/subscribe?coupon=b1c9d853&utm_content=213836638)
>
> *Once you claim it, the discount will be applied **forever**.*
>
> Now, with only **$5/month,** you will have access to:
>
> * ***200+** deep-dive data engineering articles*
> * ***[practice-spark](https://spark.vutrinh.net/problems)***: **65 LeetCode-style problems** to practice **Spark SQL/DataFrame**
> * ***[learn-spark/dbt/airflow](https://learn.vutrinh.net/#tool)**: **CLI tools** to master **Spark/dbt/Airflow***
>
> *If you’re a **Vietnamese user**, please DM me for an upgrade due to payment issues*

---

[![](https://substackcdn.com/image/fetch/$s_!NSig!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60968962-dfda-44ea-bce3-ed0bb095a5a6_2000x1429.png)](https://substackcdn.com/image/fetch/%24s_%21NSig%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/60968962-dfda-44ea-bce3-ed0bb095a5a6_2000x1429.png)

---

# Intro

In May 2025, Databricks acquired Neon, the company that provides serverless Postgres.

A few months later, Databricks introduced Lakebase, a serverless, Postgres-compatible transactional database architecture that separates compute from storage to enable low-latency applications and AI agents.

Lakebase is the result of a collaboration between Databricks and Neon (Databricks invested in Neon long before the acquisition).

When I first heard about Lakebase, I thought, “nah, just another effort to decouple compute and storage for an OLTP database. “

However, after having a chance to read a [Databricks blog](https://www.databricks.com/blog/object-storage-wal-lakebase-postgres-agentic-era) recently, I realized it’s more than that.

In this article, join me to understand what Lakebase is and Databricks’ vision behind this new database.

> *This article is not sponsored by or affiliated with any organization. I write it just because I was curious out Lakebase. Also, I wrote this article based purely on my research and understanding, feel free to give me feedback.*

But before going into the details of Lakebase, I think it would be helpful to review some PostgreSQL concepts and internal mechanisms.

---

# PostgreSQL

## PostgreSQL is an OLTP database.

[![](https://substackcdn.com/image/fetch/$s_!U_PM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e385dd5-2d17-4fbc-b587-0f9c68ea5f6b_1292x540.png)](https://substackcdn.com/image/fetch/%24s_%21U_PM%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/3e385dd5-2d17-4fbc-b587-0f9c68ea5f6b_1292x540.png)

OLTP databases primarily handle the ingestion of new information. Their data operations typically read/write/update a small amount of data each time.

An e-commerce company requires an OLTP database to store order information.

Or, a bank needs an OLTP database to store accounts, transactions, etc

[![](https://substackcdn.com/image/fetch/$s_!eGIG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5bfa440-4a1b-4a77-9b6a-3429dfee4db1_560x332.png)](https://substackcdn.com/image/fetch/%24s_%21eGIG%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/f5bfa440-4a1b-4a77-9b6a-3429dfee4db1_560x332.png)

Their goals are:

* **High Concurrency:** To support thousands of users and processes simultaneously reading and writing data without interfering with each other.
* **High Throughput for Writes:** To quickly process a large volume of minor, frequent updates, inserts, and deletes.

So, the way they manage the data changes is optimized for a small number of rows; most OLTP systems manage data in a row format (records are stored next to each other). In more detail, an OLTP relational system, like PostgreSQL, stores row data side by side in pages

[![](https://substackcdn.com/image/fetch/$s_!ZVqN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46068d46-3dd6-4786-9f15-119018dbb0ec_1228x606.png)](https://substackcdn.com/image/fetch/%24s_%21ZVqN%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/46068d46-3dd6-4786-9f15-119018dbb0ec_1228x606.png)

## Page

A database page is the I/O unit created and managed by the database software itself. This is different from the disk page, which is a unit of storage managed by the disk hardware.

[![](https://substackcdn.com/image/fetch/$s_!pude!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F339e7423-146c-41a1-b42f-6e1a9e1d96e4_1174x610.png)](https://substackcdn.com/image/fetch/%24s_%21pude%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/339e7423-146c-41a1-b42f-6e1a9e1d96e4_1174x610.png)

A page contains metadata, actual data, or indexes, and it usually doesn’t include a mix of data types (e.g., a page for data only or a page for indexes only).

When the database needs to read data, it doesn’t fetch a single row from the disk; it reads the **entire page** containing that row into memory. Likewise, when it writes data, it writes the whole modified page back to disk.

[![](https://substackcdn.com/image/fetch/$s_!RRrW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ae623a5-97de-49c4-a0ea-fd1fb4fcf9fa_1292x860.png)](https://substackcdn.com/image/fetch/%24s_%21RRrW%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/9ae623a5-97de-49c4-a0ea-fd1fb4fcf9fa_1292x860.png)

A row’s unique address of a record is a composite key ⟨Page ID, its position in that page⟩. There are mechanisms (such as a B+ Tree index) that help the database locate the required page during reads and writes. When it has the page, the database will unpack it and jump to the row’s position.

## WAL

When you modify data, the affected page is brought into memory, changed there, and eventually flushed back to disk. But memory is volatile; if the machine crashes before that flush happens, the change is gone.

To prevent this, OLTP databases such as PostgreSQL use a Write-Ahead Log (WAL) to provide durability.

[![](https://substackcdn.com/image/fetch/$s_!KzPz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffcd314fa-6aae-4fbb-9e2a-3a9ef748a969_1040x520.png)](https://substackcdn.com/image/fetch/%24s_%21KzPz%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/fcd314fa-6aae-4fbb-9e2a-3a9ef748a969_1040x520.png)

WAL is a separate, append-only structure on disk that records every change as it happens. The rule is simple: before a modified page is allowed to be written to disk, the log record for that change must be written to disk first.

If the machine dies before the modified page made it to disk, the database replays the WAL and reconstructs the page from there.

By default, PostgreSQL acknowledges a commit as soon as the change is durably in the WAL; it doesn’t wait for the actual page to be written back to disk at all. This is faster because appending to a log is cheaper than scattered page writes.

Keep these details about Page and WAL in mind, because they play important roles in the way you understand how Lakebase works.

Now, let’s jump on Lakebase.

# Motivations

Lakebase's motivation is that Databricks wants to introduce a new-generation transactional database architecture.

—

Hardware and the way we develop applications have changed a lot. Networks are faster, storage is cheaper, and pay-as-you-go is the dominant model.

However, major OLTP players such as PostgreSQL and MySQL were designed at a time when networks were slow, and buying servers upfront was the common way to deploy applications.

[![](https://substackcdn.com/image/fetch/$s_!PX3Y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3d79531-18f3-4f4b-b443-42049812bbd9_390x520.png)](https://substackcdn.com/image/fetch/%24s_%21PX3Y%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/c3d79531-18f3-4f4b-b443-42049812bbd9_390x520.png)

Coupling storage and compute was reasonable to achieve good performane at the time.

However, this approach doesn’t fit into today's context.

[![](https://substackcdn.com/image/fetch/$s_!dJe8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d15db9a-3edb-4b02-9b45-519e37f3f03b_1356x632.png)](https://substackcdn.com/image/fetch/%24s_%21dJe8%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/6d15db9a-3edb-4b02-9b45-519e37f3f03b_1356x632.png)

If you want to add storage, you have to add compute.

[![](https://substackcdn.com/image/fetch/$s_!gYrE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11effe0a-715f-4793-b205-c6efc8ea1e83_1346x632.png)](https://substackcdn.com/image/fetch/%24s_%21gYrE%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/11effe0a-715f-4793-b205-c6efc8ea1e83_1346x632.png)

If you want to add compute, you have to add storage as well.

This prevents users from leveraging the pay-as-you-go model. Resource utilization is also a problem; you add more compute to handle peak workloads, but when it returns to normal, you can’t simply turn it off, given that compute nodes also store data.

It lacks elasticity.

—

Efforts such as AWS Aurora and Google AlloyDB solve this problem by providing PostgreSQL- and MySQL-compatible solutions while separating compute and storage.

[![](https://substackcdn.com/image/fetch/$s_!4cdM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facf908ef-5f8f-46fd-8f68-0aff7babeda3_430x644.png)](https://substackcdn.com/image/fetch/%24s_%214cdM%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/acf908ef-5f8f-46fd-8f68-0aff7babeda3_430x644.png)

In return, you have to store the data in these services’ proprietary formats; only these database engines can access it.

[![](https://substackcdn.com/image/fetch/$s_!lzuJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21ca99b3-7531-460f-af83-5f8ce362324c_710x540.png)](https://substackcdn.com/image/fetch/%24s_%21lzuJ%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/21ca99b3-7531-460f-af83-5f8ce362324c_710x540.png)

Databricks doesn’t see that as an ideal approach because users will be locked into a vendor, limiting “portability and the ability to use multiple engines”.

—

Databricks observes that traditional databases do not fit into modern development workflows. For a new feature or bug fix, developers just need to branch out the codebase and work on that isolated branch.

[![](https://substackcdn.com/image/fetch/$s_!qpQA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72df8d2b-294a-4db0-a93b-d46d8e1c060b_1324x574.png)](https://substackcdn.com/image/fetch/%24s_%21qpQA%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/72df8d2b-294a-4db0-a93b-d46d8e1c060b_1324x574.png)

The process is very lightweight thanks to version control systems (like Git); when you branch out, it’s only a metadata operation; the new branch is just a new pointer to the latest commit of the existing branch, where the commit is a snapshot of the entire project at a specific point in time.

[![](https://substackcdn.com/image/fetch/$s_!C9IQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F151f2392-5f13-4ef2-a296-c200e5aa6c86_844x928.png)](https://substackcdn.com/image/fetch/%24s_%21C9IQ%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/151f2392-5f13-4ef2-a296-c200e5aa6c86_844x928.png)

But when it comes to databases, things are not that simple. Having a separate database in an isolated manner costs time and money because spinning up a new instance is not as lightweight as branching a codebase, and the process requires moving the full data rather than just pointer operations as in a version control system.

—

Applications are no longer built only by humans.

Agents can do that too.

An agent building an application isn’t following a single linear path to a finished product.

It’s experimenting: try a schema, run the migration, see if it breaks something, try a different approach, test it against.

[![](https://substackcdn.com/image/fetch/$s_!6_zg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cf2c9-ebf5-44a4-bb5b-a2c6630ecd25_1062x592.png)](https://substackcdn.com/image/fetch/%24s_%216_zg%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/752cf2c9-ebf5-44a4-bb5b-a2c6630ecd25_1062x592.png)

Each of those attempts wants its own sandbox, because you don’t want migration attempt #3 corrupting the data that migration attempt #2 was tested against. You definitely don’t want any of them touching production.

Stack enough of these attempts up.

One per prompt/experimenting, sometimes several in parallel to compare approaches- and you get a lot of short-lived databases.

Spinning up a new database is neither cheap nor fast. With many iterations, the overhead is multiplied and becomes a real bottleneck.

It would be great if we could somehow “branch“ the database.

—

From the insights above, Databricks wants to build an OLTP database that:

* Has compute and storage separate.
* Stores data in an “open“ format to enable interoperability.
* Has the ability to “branch out“

---

> *Reminder: I’m offering a limited-time **50% discount** on the annual plan:*
>
> [50% OFF FOREVER](https://vutr.substack.com/subscribe?coupon=b1c9d853&utm_content=213836638)
>
> *Once you claim it, the discount will be applied **forever**.*
>
> Now, with only **$5/month,** you will have access to:
>
> * ***200+** deep-dive data engineering articles*
> * ***[practice-spark](https://spark.vutrinh.net/problems)***: **65 LeetCode-style problems** to practice **Spark SQL/DataFrame**
> * ***[learn-spark/dbt/airflow](https://learn.vutrinh.net/#tool)**: **CLI tools** to master **Spark/dbt/Airflow***
>
> *If you’re a **Vietnamese user**, please DM me for an upgrade due to payment issues*

---

# Design decisions

For those characteristics, Databricks makes the following technical decisions for Lakebase.

The first one is to leverage object storage for the storage layer.

Object storage services are famous for durability and availability. However, object storage alone won’t deliver the desired low-latency performance for OLTP.

[![](https://substackcdn.com/image/fetch/$s_!Ejfu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc8716a4-2f33-45f3-a113-89b8ce8b6bc6_1060x836.png)](https://substackcdn.com/image/fetch/%24s_%21Ejfu%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/dc8716a4-2f33-45f3-a113-89b8ce8b6bc6_1060x836.png)

There are stateless compute nodes that run managed PostgreSQL with local SSDs for caching, helping achieve low-latency performance; the data-serving process usually doesn’t need to access object storage unless there is a cache miss.

In such a case, a dedicated component will fetch data from object storage, give it to the compute to cache it on disk, and serve it from the cache; subsequent requests will benefit from this cache.

—

Due to the separation, compute can be scaled independently (no need to rebalance data between nodes).

[![](https://substackcdn.com/image/fetch/$s_!BNBh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F373e236c-28ba-4e66-aee9-d714d661bcf8_1084x618.png)](https://substackcdn.com/image/fetch/%24s_%21BNBh%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/373e236c-28ba-4e66-aee9-d714d661bcf8_1084x618.png)

Lakebase resizes compute continuously; CPU and memory scale up and down in small increments based on live workload, all the way down to zero when nothing's happening, then back up in under a second when a request arrives, thanks to a warm pool of compute kept ready in advance by Databricks.

—

The second decision is to treat WAL as a first-class citizen.

The WAL is not used only for recovery anymore; it’s the backbone of Lakebase. To understand this point, let’s see what WAL’s records look like; here is an example from Databricks’ blog:

[![](https://substackcdn.com/image/fetch/$s_!KnCt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1694849-d6aa-436b-9e15-4cc3a0cd0b0a_1536x234.png)](https://substackcdn.com/image/fetch/%24s_%21KnCt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/f1694849-d6aa-436b-9e15-4cc3a0cd0b0a_1536x234.png)

[Source](https://www.databricks.com/blog/object-storage-wal-lakebase-postgres-agentic-era)

It records not only the insert transaction, but it also records which pages are involved (the *blkref* identifier) and the timeline order (the LSN, a monotonically increasing identifier; anything that happened later always has a higher LSN)

This means if you have the whole WAL history, you will have the complete timeline of changes applied to the database.

[![](https://substackcdn.com/image/fetch/$s_!hKQf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c3dde89-4f1a-4de2-8947-0aad096718e5_1604x496.png)](https://substackcdn.com/image/fetch/%24s_%21hKQf%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c3dde89-4f1a-4de2-8947-0aad096718e5_1604x496.png)

—

In traditional Postgres, WAL is only for protection. WAL records can be safely discarded when data is persisted on disk. The data files are the real database, modified in place, and the log protects them along the way. No persisted history.

[![](https://substackcdn.com/image/fetch/$s_!qXFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ff5b992-de64-457f-9542-3639c86c055c_566x452.png)](https://substackcdn.com/image/fetch/%24s_%21qXFU%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/4ff5b992-de64-457f-9542-3639c86c055c_566x452.png)

However, Databricks flips it.

WAL becomes the actual database. Data files are the result of the WAL materialization process. Instead of in-place modification, changes are persisted in immutable files (which work well with object storage).

[![](https://substackcdn.com/image/fetch/$s_!zb3P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda703e0d-5f64-433b-997d-fe36f64a98d1_760x552.png)](https://substackcdn.com/image/fetch/%24s_%21zb3P%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/da703e0d-5f64-433b-997d-fe36f64a98d1_760x552.png)

Immutable files aren’t just one thing here; they split into two kinds:

[![](https://substackcdn.com/image/fetch/$s_!HUog!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b97406d-3cf5-4119-9cc3-43586cfc10ff_1162x954.png)](https://substackcdn.com/image/fetch/%24s_%21HUog%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/4b97406d-3cf5-4119-9cc3-43586cfc10ff_1162x954.png)

* **Delta layer**: only the changes: raw WAL records for a range of *keys* over some span of LSNs. (Worth being precise here: a “key” isn’t a row or a primary key; it’s a page identifier, the same *blkref* you saw in the WAL record earlier. A “key range” is just a slice of pages, e.g., blocks 1,000–2,000 of a table.) If a key didn’t change, it’s not in the delta layer.
* **Image layer**: full snapshot. Every key within its key range, every page in that slice, is materialized, frozen at one specific LSN, even pages that haven't changed in months.

Neither one gets modified in place. A delta layer is appended once and never touched again. An image layer (the snapshot), once written, stays exactly as it was. It is later superseded by a new image layer (a newer snapshot) produced in the background via compaction, which replays the accumulated delta layers onto the old image to produce a fresh one.

[![](https://substackcdn.com/image/fetch/$s_!iGxq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff7adcdb9-c219-4b34-8356-852b8bac5e38_1156x838.png)](https://substackcdn.com/image/fetch/%24s_%21iGxq%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/f7adcdb9-c219-4b34-8356-852b8bac5e38_1156x838.png)

> *You can think of it like a CDC implementation: there’s a base table, and a changes table streaming into it. To get the current state, you “merge” the changes into the base.*
>
> *Constructing a new image layer works the same way: the merge produces a fresh image layer (applying delta changes on top of existing image) that becomes the new base going forward*

Compared to the traditional approach, history is preserved now.

By treating WAL this way, Lakebase can address the whole database timeline. But no single file holds “the whole database’s current state”.

An image layer covers only a slice of the database at a single LSN, and many of them can coexist for the same page key range, each frozen at a different point in history (LSN), because none of them are ever overwritten.

[![](https://substackcdn.com/image/fetch/$s_!FDbO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd64e352-186e-411e-ada5-bf227aff8681_1502x610.png)](https://substackcdn.com/image/fetch/%24s_%21FDbO%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/cd64e352-186e-411e-ada5-bf227aff8681_1502x610.png)

To reconstruct a state, Lakebase picks the right image layer for each key range and replays any deltas on top of it.

---

# Architecture

As discussed, Lakebase separates the architecture into two parts: compute and storage.

## Compute

The compute nodes will run PostgreSQL. They will do what a traditional PostgreSQL would do: accept user queries, validate them, parse the SQL, plan and execute them, and handle indexes.

[![](https://substackcdn.com/image/fetch/$s_!oMFd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa396fd0b-b022-4ad3-aa63-ccd4dcb8d195_1046x608.png)](https://substackcdn.com/image/fetch/%24s_%21oMFd%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/a396fd0b-b022-4ad3-aa63-ccd4dcb8d195_1046x608.png)

Compared to traditional PostgreSQL, compute nodes no longer hold data; they still have RAM and disk, but they’re used for data processing and caching rather than persisting data.

That is the job of the storage layer.

## Storage

It has three components:

[![](https://substackcdn.com/image/fetch/$s_!ciI3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F740721cd-9f89-406f-b297-4a3089e7f8b9_930x826.png)](https://substackcdn.com/image/fetch/%24s_%21ciI3%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/740721cd-9f89-406f-b297-4a3089e7f8b9_930x826.png)

* Safekeepers (SK): they absorb WAL records from compute and replicate them between Safekeepers. Users receive an ACK only after a quorum of SKs write the WAL records to disk (to ensure durability). SKs then upload the WAL records to object storage (object storage writing happens later to reduce user request latency)
* Pageserver (PS): This component streams WAL from SK and materializes those WAL records into PostgreSQL pages (more details later). One of the APIs that SK exposes for the compute node to read data is GetPage@LSN, which allows it to read a page at a specific LSN. When the compute node calls this API, the PS finds that page's most recent complete snapshot (from the image layer) at or before the requested LSN, replays the delta records (from the delta layer) up to that LSN on top of it, and returns the page.
* Object storage: it holds all WAL and immutable files (delta and image layers).

---

# Write process

Here's what’s under the hood of the Lakebase’s write process:

[![](https://substackcdn.com/image/fetch/$s_!cCIa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ac4b28c-4e5b-43b0-8ef2-da2cb6de2011_1374x816.png)](https://substackcdn.com/image/fetch/%24s_%21cCIa%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/9ac4b28c-4e5b-43b0-8ef2-da2cb6de2011_1374x816.png)

* At first, things will proceed as in traditional PostgreSQL: changes are applied in memory, and WAL records are prepared for write.
* However, instead of persisting WAL records on local disk as in traditional PostgreSQL, the compute node sends them to the SKs over the internet.
* Once a quorum of SKs write the WAL records to disk, the transaction is considered committed, and the user will receive the ack.
* After that, one of the SKs will be chosen to upload the WAL records to the object storage. It will be load-balanced between the SKs; not always the same SK will do the upload.
* PSs will stream the WAL from the SKs asynchronously to materialize pages from WAL records, then upload them to object storage. The user doesn’t need to wait for the pages to materialize to get the ack. PS will write WAL records in an LSM-tree-inspired structure, which enables a high WAL ingestion rate and efficient page-level access via an exposed API for compute.
* For more details, PS first appends WAL records to an in-memory buffer; this is cheap and fast; it just appends. This is what lets the Pageserver keep up even under write-heavy load. Once that buffer is big enough or old enough, it gets flushed out as a delta layer.
* Delta layers stack up over time for a given key range. Once the stack passes a threshold, compaction kicks in, replaying the delta records onto the existing image to produce a fresh image layer.
* Both delta and image layers get uploaded to object storage for durability (object storage is the single source of truth)

---

# Read process

For the read process, here is what happened:

[![](https://substackcdn.com/image/fetch/$s_!nlmF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb87e9096-5a15-4f62-b8a1-6ac4e27b157b_1684x1054.png)](https://substackcdn.com/image/fetch/%24s_%21nlmF%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/b87e9096-5a15-4f62-b8a1-6ac4e27b157b_1684x1054.png)

* A read first checks the compute node’s own cache. First comes the RAM for Postgres shared buffers, then the compute node's local NVMe cache. If the required data exists, the compute node will serve it immediately. Most reads never leave the compute node at all.
* On a local miss, the compute node issues a `GetPage@LSN` request to the PS, asking for a specific page at a specific LSN.
* The PS looks up which layer covers that page’s key at this LSN, searching across the delta and image layers produced by that LSM-inspired structure from the write path. PS uses an index built specifically to make that lookup fast even across millions of layer files. The layers can be fetched from PS’s local disk or downloaded from object storage.
* Once the right layer is found, the Pageserver locates the most recent image of the page at/before the requested LSN, replays any delta records that apply on top of it to reconstruct the complete page at that LSN.
* That page is returned to the compute node, cached locally, and served to the request. Subsequent queries will benefit from this cache and don’t need to propagate down to object storage.

---

# Branching

The design of making WAL the database allows Lakebase to seamlessly “branch” a database.

Originally, that process involved provisioning a new instance and physically duplicating the data.

Lakebase skips all of that because a branch isn't a copy of data; it's a reference.

[![](https://substackcdn.com/image/fetch/$s_!Yobl!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcbf5b098-3d18-46bf-b79d-a86257779f76_1606x418.png)](https://substackcdn.com/image/fetch/%24s_%21Yobl%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/cbf5b098-3d18-46bf-b79d-a86257779f76_1606x418.png)

Branching in Lakebase is just writing down two things

* A new timeline ID
* The exact LSN in the parent’s history to fork from.

A **timeline** in Lakebase is a branch history; the chain of layer files (delta and image layers) belonging to it, tied to its own timeline ID.

The root database starts as the initial timeline (like the main branch in Git). Branching creates a new one that starts out as just a pointer back to the parent timeline and the LSN it forked from (like a branch is a pointer to a commit in Git).

No data pages are moved or touched (thanks to the fact that data is managed in immutable files, history is completely preserved)

As the branch takes writes, it has its own delta and image layers, forming its own history.

Just like how things work with Git when you “git checkout -b“. It doesn’t duplicate your repo; it creates a pointer to a commit, and the branch shares every object with its parent until something actually changes.

---

# Use cases

I don’t use Lakebase yet, but I believe the ability to lightweight-version-control the database like this will be super helpful for developers.

With this feature alone, Lakebase is already an attractive replacement for vanilla PostgreSQL.

[![](https://substackcdn.com/image/fetch/$s_!dd34!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbea2af45-e71f-41f8-9073-d20968de07e2_1374x390.png)](https://substackcdn.com/image/fetch/%24s_%21dd34%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/bea2af45-e71f-41f8-9073-d20968de07e2_1374x390.png)

Branching, rolling back, and recovering the database in seconds (only pointer operations) allow you to iterate faster while developing an application.

It takes 1 second to create a new code branch and maybe 5 more seconds to create a new database branch.

(Those who handle a lot of database migrations and rollbacks (including me) really like this)

But I bet Agents like this more.

With Lakebase, they can, for example, branch 20 databases simultaneously and run experiments on them. After that, when most of the branches are no longer needed, Lakebase can automatically scale the compute down to save costs.

Agents can now run at full speed; database creation is no longer the bottleneck.

—

Storing data in object storage not only makes Lakebase’s compute node stateless but also enables more opportunities for data sharing.

Traditionally, data is moved from an OLTP database to object storage before being loaded into an OLAP database.

[![](https://substackcdn.com/image/fetch/$s_!7X6M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c901112-ed7c-4e0b-8c02-1533cf217060_920x464.png)](https://substackcdn.com/image/fetch/%24s_%217X6M%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/7c901112-ed7c-4e0b-8c02-1533cf217060_920x464.png)

With Lakebase, data is already in object storage, so there is no need for the OLTP data exporting step. Databricks introduces native data sync to the Lakehouse using a logical decoding extension (wal2delta). The extension takes the WAL stream and writes directly to a Delta Lake table in SCD Type 2 format. Lakehouse system can consume the SCD2 table directly or compact it into the snaphot.

—

For the other direction, Databricks also supports syncing data from Lakehouse to Lakebase by creating a Spark streaming DataFrame that reads Delta changes from the analytical table, feeds them into the destination table in PostgreSQL, and updates it.

[![](https://substackcdn.com/image/fetch/$s_!V2fj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc115d1b5-3af6-4c0f-b0a9-a7b792f06cce_944x456.png)](https://substackcdn.com/image/fetch/%24s_%21V2fj%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/c115d1b5-3af6-4c0f-b0a9-a7b792f06cce_944x456.png)

This is useful for feeding analytical results to end users, such as personalized recommendations or reverse ETL.

—

Thanks to this openness, Lakebase is the foundation of Databricks’ LTAP offer, the architecture that unifies OLAP and OLTP on a single copy of data.

---

# Outro

In this article, we deep-dive into Lakebase, the new OLTP database generation introduced by Databricks. We first revisit some basic PostgreSQL concepts, such as pages and WAL, then move on to Lakebase’s motivations, design decisions, architecture, and read/write path. Next, we see how Lakebase offers lightweight branching, just like Git. Finally, we see some use cases where Lakebase can shine.

Thank you for reading this far.

---

# Reference

*[1] Databricks, [Lakebase: Serverless Postgres over Open Lake Storage](https://www.vldb.org/pvldb/vol19/p4385-pandis.pdf) (2026)*

*[2] Cassie Murray, Carlota Soto, [Object Storage + WAL: Lakebase Postgres for the agentic era](https://www.databricks.com/blog/object-storage-wal-lakebase-postgres-agentic-era) (2026)*

25

3

2

Share

#### Discussion about this post

CommentsRestacks

![User's avatar](https://substackcdn.com/image/fetch/$s_!TnFC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fdefault-light.png)

[![Thuan Thong Tra's avatar](https://substackcdn.com/image/fetch/$s_!n7Iv!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fblack.png)](https://substack.com/profile/50391684-thuan-thong-tra?utm_source=comment)

[Thuan Thong Tra](https://substack.com/profile/50391684-thuan-thong-tra?utm_source=substack-feed-item)

[1d](https://vutr.substack.com/p/databricks-introduced-a-new-database/comment/330232732 "Sep 5, 2026, 12:38 PM")

Liked by Vu Trinh

Thank you for sharing this great article. Can't wait for the next one.

I have been playing around with the "Lakebase CDF" feature recently to sync Lakebase tables to Lakehouse tables and create SCD2 tables, but I only had a limited understanding of how it actually works. Now it makes much more sense—looks like the wal2delta extension plays a key role behind the scenes.

ReplyShare

[![Jocelyn Sedes's avatar](https://substackcdn.com/image/fetch/$s_!ryS2!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fyellow.png)](https://substack.com/profile/195936096-jocelyn-sedes?utm_source=comment)

[Jocelyn Sedes](https://substack.com/profile/195936096-jocelyn-sedes?utm_source=substack-feed-item)

[2d](https://vutr.substack.com/p/databricks-introduced-a-new-database/comment/330101379 "Sep 5, 2026, 6:30 AM")

Liked by Vu Trinh

Thanks for sharing this new database’s insights. Very interesting. What about DML operations? Do you know how it is handle ?

ReplyShare

[1 reply by Vu Trinh](https://vutr.substack.com/p/databricks-introduced-a-new-database/comment/330101379)

[1 more comment...](https://vutr.substack.com/p/databricks-introduced-a-new-database/comments)

TopLatestDiscussions

No posts

### Ready for more?

Subscribe

© 2026 Vu Trinh · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)

[Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)

[Substack](https://substack.com) is the home for great culture

This site requires JavaScript to run correctly. Please [turn on JavaScript](https://enable-javascript.com/) or unblock scripts