Diskless Kafka: What Happens When Brokers Stop Owning the Data?

Apache Kafka was designed to make good use of sequential I/O. Producers batch records, brokers append those batches to ordered log segments, and consumers read sequential ranges. Kafka also relies on the operating system's page cache and uses zero-copy techniques such as sendfile where possible. The Kafka design documentation explains why this works: batching turns many small operations into larger sequential reads and writes, which storage devices and operating systems can handle efficiently.

In the cloud, another storage option is available. An object store can provide durable, replicated storage independently of broker compute. Could Kafka use that service for the active log and let brokers scale without moving the durable dataset?

That is the idea behind diskless Kafka: retain Kafka's client API while moving durable payload storage away from broker-owned partition logs. The implementation still has to preserve ordering, assign offsets, track committed writes, and serve reads efficiently. Different systems divide these responsibilities differently, and their feature coverage varies.

Upstream status, August 25, 2026: KIP-1150 is accepted, but native Diskless Topics are not yet a generally available Apache Kafka feature. KIP-1163 and KIP-1164, which define the core implementation, remain under discussion.

How the broker and the log are coupled

In classic Kafka, a partition leader accepts records, assigns offsets, and appends batches to local log segments. Followers fetch from the leader and append the same batches to their own storage. With acks=all, the producer waits for the required replication before receiving a successful acknowledgement.

Batching reduces network and system-call overhead, while the page cache can serve recent data without a physical disk read. A broker therefore owns both the processing responsibility for its partitions and their local log segments and replica state. Changing where partitions run often means moving their data too.

The cloud changed the storage equation

Consider a cluster spread across three availability zones. Each record is stored by the leader and copied to followers in the other zones:

Replicating across zones lets Kafka tolerate zone failures when replica placement and acknowledgement settings are configured accordingly. It can also generate substantial cross-AZ traffic. KIP-1150 identifies this as a major cost driver on AWS and Google Cloud. The economics depend on provider and network configuration; payload replication is only one part of the total traffic bill.

Every broker also needs enough storage for its assigned partitions. Adding capacity can require partition reassignment; replacing a lost replica requires copying its data. A broker holding terabytes of partition state takes longer to replace than a worker whose durable state lives elsewhere. Storage capacity and data movement constrain how quickly compute can change.

Amazon S3, Google Cloud Storage, and Azure Blob Storage provide durable storage independently of Kafka brokers. With an appropriate regional storage class, the provider also handles replication across failure domains. The useful comparison is between that service and Kafka's complete replicated storage path. EBS itself replicates within one availability zone; Kafka's replicas can extend protection across zones.

KIP-1150 proposes delegating payload durability to object storage to reduce the cost of active-segment replication and broker block storage. The change is in who owns replication, and which data still needs to pass through Kafka's coordination mechanisms.

Diskless Kafka vs. Tiered Storage

Kafka already uses object storage through Tiered Storage. Completed log segments can be copied to remote storage and later removed from the local tier. The active segment still lives on broker storage, and followers still replicate incoming records.

Tiered storage separates retention from local disk capacity. A cluster can retain months of history without provisioning broker disks for the entire dataset. Diskless storage changes the active write path as well:

Some systems place a separate WAL before the object-storage dataset. In those designs, the WAL determines when a write can be acknowledged; we will look at AutoMQ below.

What “diskless” means

The name is slightly misleading. Diskless does not necessarily mean that no disk exists anywhere in the system; it means something more specific: broker-local disk is no longer the primary durable source of truth for user data. KIP-1150 puts it nicely: diskless is to "no disks" roughly what serverless is to "no servers." Disks may still exist for caches, metadata, or temporary state, but they stop being the storage abstraction operators need to manage for the message payload itself.

A simplified diskless write path looks like this:

Classic Kafka keeps payloads and their ordering together in partition logs. Diskless systems separate them: object storage holds the bytes, while metadata defines their order, offsets and commit status.

S3 is not a Kafka log

At first glance, building Kafka on object storage sounds trivial: receive a record, write it to S3, done. Unfortunately, that would be a terrible streaming architecture. Object stores are optimized around objects, not appendable logs - you generally create immutable objects and retrieve them later - and object-store operations have non-trivial latency and per-request cost.

Creating an S3 object for every Kafka record would quickly become expensive at high throughput. Every producer batch? Still far too many objects. One object per partition every few hundred milliseconds? Now a cluster with tens of thousands of partitions starts generating an unpleasant number of tiny objects and PUT operations.

Object storage wants:

FEWER
LARGER
OBJECTS

Kafka wants:

MANY
INDEPENDENT
ORDERED PARTITIONS

Reconciling those two requirements is where diskless streaming gets interesting.

How diskless Kafka batches across partitions

Most diskless Kafka architectures arrive at the same first optimization: do not create an object for every partition - batch data from many partitions together.

WarpStream is a good example. Its Agents buffer producer traffic from multiple clients and partitions and flush on a time or size threshold; the current default batch timeout is 250 ms. To be precise about what that number is: it is the longest an Agent will wait for a batch to fill, not the durable write latency - under real load the size threshold usually fires first, and the producer acknowledgement additionally waits for the object-storage upload and the metadata commit. The 250 ms is one component of a latency budget we will return to later in this article.

Files can contain records belonging to many different topic-partitions. The object is persisted first, its metadata is committed, and only then is the produce request acknowledged. Aiven's diskless implementation uses the same broad idea, and Redpanda Cloud Topics batches data across topics and partitions into what it calls L0 objects - "level zero," a name borrowed from LSM-tree storage engines for the raw, freshly ingested files - before recording lightweight metadata in Raft. Keep the term in mind; the levels will matter later.

Several teams arrived at this design independently because cross-partition batching amortizes the object-storage operation across a useful amount of data. The price is that the physical object no longer maps cleanly to a Kafka partition.

The physical Kafka log just disappeared

In classic Kafka, partition-7.log contains records belonging to partition 7, in order. The physical storage layout and the logical Kafka log are closely related. Now consider our diskless object:

object-123

P7 | P2 | P19 | P1 | P7

and another broker writing concurrently:

object-124

P1 | P7 | P5 | P19

Which P7 batch comes first? Object names cannot answer that, upload completion time is not enough, and the object itself cannot answer it either. Object storage contains bytes, but the Kafka abstraction requires something stronger: topic, partition, offset, order. So a diskless system needs another structure mapping the logical log onto physical storage - something like:

We removed the physical partition log, so now we have to reconstruct a logical partition log using metadata. Notice what we are actually building now: a log that exists only as strongly consistent metadata over shared objects - which is to say, a distributed database wearing Kafka's protocol.

Metadata defines the committed log

Ordering is metadata. Offset assignment is metadata. The relationship between offsets and object locations is metadata. Transactions need metadata, idempotent producers need state, retention eventually changes metadata, compaction changes metadata, and garbage collection depends on all of it. A diskless Kafka implementation can outsource the replication of message payloads to S3; it cannot outsource Kafka semantics to S3. That means diskless Kafka does not remove consensus; it relocates it. Payload durability can move into object storage, but the system still needs a strongly consistent place to decide what belongs at each offset and which writes have committed.

There is a fair objection to raise here: is the metadata layer not simply the original replication problem all over again? Partly, yes—and none of these designs pretends otherwise. The metadata store is itself a replicated, strongly consistent system, and it processes a commit for every batch, so the number of coordination operations does not go away. What changes is what each operation carries: a pointer-sized record describing an object and its offset ranges, instead of the payload itself. Cross-AZ traffic is billed per byte, not per operation, so shrinking the replicated unit from megabytes of records to a metadata entry is exactly where the savings live—and cross-partition batching keeps the commit rate proportional to batches, not to individual records. Diskless Kafka does not escape replication; it reserves it for the data that is small and genuinely needs consensus. Apache Kafka's own proposal reaches the same conclusion later in this article: the diskless coordinator keeps its state in a replicated internal Kafka topic.

Different implementations, different boundaries

"Diskless Kafka" is becoming a category rather than one specific architecture. The implementations below all move the payload away from Kafka's traditional replicated local log, but they keep different amounts of state in brokers and introduce different metadata paths.

WarpStream: write directly to object storage

WarpStream takes perhaps the cleanest version of the idea. Its Kafka-compatible Agents do not require local disks: Agents write directly to object storage, while metadata lives in WarpStream's separate metadata layer. Because no Agent owns durable partition data, compute nodes can come and go without moving the stored log.

Any Agent can serve any partition, which removes the usual coupling between partition ownership and durable storage. The trade-off is equally direct: object storage sits in the acknowledgement path, so latency and object-store request economics become first-class design constraints. WarpStream documents the architecture in its write-path and read-path guides.

AutoMQ: add a WAL

AutoMQ makes a different trade. It agrees that object storage should hold the actual dataset, but introduces a WAL to accelerate persistence and recovery:

An interesting detail is that the WAL is shared across writes from many partitions rather than maintaining Kafka's traditional per-partition log structure. AutoMQ describes it as fixed-size and cyclic, with sequential writes and group commits. So even after moving Kafka away from broker-owned partition logs, the same optimization appears again at another layer. Object storage changes where the durable dataset lives; it does not repeal the value of sequential I/O.

A natural follow-up question is how the WAL itself is replicated—and the answer is that AutoMQ deliberately does not replicate it. Its stated design principle is to delegate durability to cloud storage instead of running a replication protocol such as Raft: with an EBS-backed WAL, durability comes from the volume's own replication, and when a broker dies, the orphaned volume is simply attached to another broker, which flushes the remaining WAL entries to S3; with an S3-backed WAL (the open-source default), the WAL objects are already in object storage, so a new broker recovers them directly. Consensus does exist in AutoMQ—but only where it does everywhere else in this article: in the metadata plane, where AutoMQ reuses Kafka's KRaft.

Redpanda Cloud Topics: retain per-partition Raft metadata

Redpanda takes a hybrid approach it has described as "disk-lite": payloads go to object storage, critical metadata still goes through local Raft.

Once an L0 object has become durable, Redpanda writes placeholder metadata into the appropriate partition's Raft log and then acknowledges the producer. Notice what that implies for broker state: the placeholder log is still a per-partition Raft log hosted on specific brokers, so every broker remains a member of concrete Raft groups, holding local metadata state, votes, and leadership roles. It is small state - pointers rather than payloads - but it means a Redpanda broker is not interchangeable the way a WarpStream Agent is, because an Agent holds no partition state at all; its metadata lives entirely in the separate metadata service. That is the sense in which Redpanda's model leaves more state in the brokers - a deliberate trade, because keeping the metadata log inside brokers lets Redpanda reuse its existing transaction, idempotency and consistency machinery. Cloud Topics became generally available in Redpanda 26.1 in 2026 and can coexist with local and tiered storage modes in the same cluster.

Confluent Freight: direct write for throughput-oriented workloads

Confluent has independently moved in the same direction. Freight clusters use a direct-write mode in the Kora engine where payload data is written to object storage instead of first being durably replicated across broker-local storage. Confluent positions Freight for high-throughput workloads such as logs, observability and analytical ingestion, explicitly trading sub-100 ms latency for write paths that can take up to roughly one or two seconds.

Aiven Diskless: a separate Batch Coordinator

Aiven runs the Inkless implementation, whose contributors also participate in the upstream diskless KIPs. Its payload write path is leaderless: brokers upload batched records while a Batch Coordinator assigns offsets and tracks their locations. Aiven uses PostgreSQL for this coordination layer, unlike the internal Kafka topic and local SQLite state proposed upstream. See Aiven's architecture.

Classic and diskless topics can coexist in the same Standard Kafka service. Aiven documents limited availability on Aiven Cloud and support on selected -inkless BYOC plans. The implementation has important feature limits: transactions, compacted diskless topics, and Kafka Streams state stores are unsupported. A classic topic can be switched to diskless through the documented migration path, but the reverse switch is unsupported. See the overview, limitations, and migration guide.

What Apache Kafka proposes

Perhaps the clearest sign that this architecture matters is KIP-1150: Diskless Topics. The Apache Kafka community marked it Accepted on March 2, 2026. That acceptance is a decision about direction, not a shipped feature: Kafka should support topics whose user payload is durably stored in object storage instead of depending on broker block storage and direct payload replication.

The implementation is still being designed. As of August 25, 2026, KIP-1163: Diskless Core and KIP-1164: Diskless Coordinator are both Under Discussion. KIP-1165, covering object compaction for diskless data, was previously discarded but was re-opened in July 2026 and is again Under Discussion. KIP-1176, an alternative that kept more of the classic replication path, is Withdrawn; KIP-1183, a broader shared-storage abstraction proposed by AutoMQ contributors, remains a separate proposal Under Discussion. The community maintains an overview of this whole family of proposals on the Saving Cross-AZ Replication Costs KIPs wiki page.

That distinction matters for readers evaluating Apache Kafka itself: KIP-1150 is accepted, but native upstream Diskless Topics are not a generally available Kafka 4.3 feature today.

KIP-1163 describes a fascinating write path: brokers can buffer batches belonging to multiple diskless partitions, build a shared WAL segment, upload it to object storage, and then commit coordinates describing those batches to the coordinator, which assigns the global partition ordering and offsets. Conceptually:

There is a historical irony here. Kafka started from a wonderfully simple storage model: the log file is the log. In the proposed diskless path, objects contain the bytes, while metadata defines how those bytes become a Kafka log.

Why the proposed coordinator uses SQLite

KIP-1164 contains one of the most revealing details in the whole diskless Kafka design. The proposal introduces an internal Kafka topic, __diskless_metadata, as the source of truth for diskless metadata. The expected coordinator state can grow to hundreds of megabytes or even gigabytes, so keeping the whole thing in memory is unattractive. The proposal therefore materializes that metadata log into a local SQLite database for indexed access. Kafka remains the durable source of truth; SQLite is a rebuildable local state store.

So a possible future Kafka broker could look roughly like this:

One clarification the diagram deserves, because it is easy to assume otherwise: __diskless_metadata itself is a classic Kafka topic, replicated across broker disks the ordinary way - not a diskless topic. That is deliberate twice over. A diskless topic depends on the coordinator, so the coordinator's own storage cannot depend on diskless topics. Keeping metadata on the classic replication path avoids another object-storage write during commit; the producer still waits for the payload upload. The KIP's risk analysis is explicit about the consequence: if the metadata topic loses too many in-sync replicas, diskless writes stall. Object storage does appear in the metadata story, but in a supporting role - the proposal allows periodic snapshots of coordinator state to be stored in object storage, so the internal topic only has to retain the deltas since the last snapshot.

Aiven's production implementation currently makes a different choice: its Batch Coordinator is PostgreSQL-based, while the upstream proposal uses Kafka's own internal topic as the source of truth and SQLite as local materialized state. The details differ, but the direction is the same. Once payload storage becomes cheap and shared, the difficult engineering moves into metadata, ordering and recovery.

Reorganizing objects for reads

Cross-partition batching fixes the write path. Unfortunately, it damages the read path. Suppose we ingest data into a freshly written, level-zero object of the kind described earlier:

L0 object

P1 P7 P3 P18 P1 P9 P7 P2 P18 P1

This is excellent for writes - one reasonably large object-storage PUT carries data from many partitions. But when a consumer wants to replay partition P7, the system may need to find and range-read many different objects just to reconstruct one sequential partition history. So diskless systems often add another database-like concept: background reorganization. This is where the "level" in L0 pays off: in Redpanda, a background Reconciler reorganizes those level-zero objects into larger L1 files with data co-located and sorted by partition and offset. Conceptually:

WarpStream performs background file compaction for a similar reason: recently ingested multi-partition files are reorganized to improve locality for historical reads. Kafka's KIP-1163 includes a merging path, while the re-opened KIP-1165 explores object compaction for the same family of problems.

Anyone familiar with LSM trees will recognize the pattern. Fast writes produce one layout; efficient reads eventually want another. A diskless streaming broker therefore starts to look less like a directory full of append-only log files and more like a purpose-built storage engine.

Latency depends on the acknowledgement path

There is an obvious catch to putting object storage in the hot path: latency. A classic Kafka write can take advantage of local storage and tightly controlled replication between brokers. A diskless implementation may need to collect enough records to make an efficient object, upload that object, wait for storage durability, commit strongly consistent metadata, and only then acknowledge the producer. If you make the batching window very short, you create lots of small objects and increase object-store request costs; if you make it long, latency increases. That tension cannot be configured away - it is part of the architecture.

Different systems approach the trade-off differently. WarpStream uses a 250 ms default batch timeout and can trade more object-store requests for lower latency; its current low-latency configurations use shorter timeouts and, on AWS, S3 Express. AutoMQ can put a faster WAL in front of object storage, so acknowledgement can depend on the WAL instead of waiting for the slower long-term path. Redpanda documents noticeably higher end-to-end latency expectations for Cloud Topics backed by public cloud object stores, and Confluent describes Freight in similar terms. The current Kafka diskless proposal also states explicitly that remote storage will increase request and end-to-end latency compared with classic topics.

That is the architectural trade-off. Object storage buys elasticity and different economics by accepting a higher-latency durability primitive - or by adding a faster WAL in front of it.

What changes operationally

One argument for diskless architectures is that they make Kafka dramatically simpler. Operationally, there is truth to that: disk sizing, replica movement, partition data rebalancing, disk replacement, broker storage exhaustion and much of the cross-AZ payload replication either disappear or become less central.

But distributed systems rarely destroy complexity, they move it. Diskless systems have to care much more about object batching, metadata sequencing, object compaction, garbage collection, cache management, range-read efficiency, small-object economics, metadata recovery and tail-read behavior.

Classic Kafka spends a lot of engineering effort maintaining replicated physical logs. Diskless Kafka spends more engineering effort describing, caching and reorganizing data in shared storage. The architecture is simpler in some operational dimensions, not universally simpler.

Does diskless Kafka actually make Kafka cheaper?

Usually, this is the reason the architecture exists in the first place. The largest potential savings come from three areas. First, less cross-AZ replication - the broker can write once into a regional object-storage service whose durability is managed by the cloud provider. Second, cheaper storage: object storage is usually much cheaper per retained byte than large quantities of provisioned high-performance block storage. Third, compute becomes easier to scale: if brokers do not own durable data, adding and removing compute nodes no longer requires proportional movement of the retained dataset.

But "S3 is cheap" is not the end of the cost model. You still pay for PUT requests, GET requests, range reads, metadata infrastructure, caches, background compaction, compute, and potentially high-performance WAL storage. This is why vendor claims of "90% cheaper Kafka" need context: the architecture can absolutely change the economics for a high-throughput workload, but that does not mean it will reduce every Kafka bill by 90%.

Where it fits

Diskless Kafka looks especially attractive when the workload combines high throughput, long retention and some latency tolerance: logs and observability streams, telemetry, CDC, clickstreams, data-lake ingestion, ML/AI ingestion and other large analytical event streams.

Those workloads move enormous numbers of bytes and often care less about a few hundred milliseconds - or even a second - of additional latency than a service-to-service command path would. Classic local-storage Kafka still has a strong argument for latency-sensitive event processing, commands, very hot tailing workloads and smaller retained datasets. This is why the likely end state is not a single storage mode for every topic.

The future is probably multi-modal Kafka

This may be the most important part of KIP-1150: diskless is proposed as a topic-level choice, not necessarily a cluster-wide identity. A future Kafka deployment could look like:

The same Kafka API and administrative boundary can therefore hide very different storage engines underneath. Redpanda already allows local, tiered and cloud storage modes in one engine. Aiven lets classic and diskless topics coexist in the same Standard Kafka service; its current feature limits still exclude transactions, compacted diskless topics and Kafka Streams state stores. A one-way switch from classic to diskless is also being offered in early availability. Upstream Kafka's KIP-1163 explicitly envisions classic, tiered and diskless topics living in the same cluster, so applications can make the cost/latency trade-off per topic rather than per platform.

This is a more interesting future than simply asking whether "diskless Kafka will replace Kafka." Kafka itself may become multi-modal.

The Kafka protocol may outlive Kafka's original architecture

Kafka's API has become bigger than Kafka's original storage design. Applications understand topics, partitions, offsets, producers, consumers, consumer groups, transactions, and Kafka protocol semantics. They do not necessarily care whether the bytes underneath live in:

ext4
XFS
NVMe
EBS
S3
GCS
Azure Blob

That has allowed WarpStream, AutoMQ, Redpanda, and others to preserve Kafka compatibility while experimenting aggressively below the protocol boundary - and now Apache Kafka itself is starting to move in the same direction. Kafka was built around the idea that sequential disk I/O could make a persistent distributed log extremely fast. That insight was correct, and diskless Kafka does not prove otherwise. The newer question is whether operating Kafka in the cloud still requires the broker to own the durable log at all.

The answer is producing architectures that look less like the Kafka clusters of the 2010s and more like modern cloud databases:

stateless-ish compute
       +
shared object storage
       +
strongly consistent metadata
       +
caching
       +
background compaction

So, is diskless Kafka the future?

Not for every topic. For high-throughput cloud streaming, though, the direction is hard to ignore.

The traditional Kafka architecture still has excellent properties for latency-sensitive workloads. Local sequential storage is fast, and a tightly integrated leader with a replicated log remains a simple and powerful abstraction. The systems described here already offer several diskless designs. For an actual deployment, the useful comparison is acknowledgement latency, recovery behaviour, feature support and total cost for the workload.

Reviewed by: Adam Warski