Apache Pulsar vs Kafka: Architectural Benchmarks and Production Limits

When scaling streaming platforms beyond 100,000 writes per second, engineers hit a structural crossroad: Apache Kafka ties partition storage directly to physical broker disks, while Apache Pulsar divorces stateless compute brokers from an underlying storage tier built on Apache BookKeeper. In modern 2026 enterprise architectures, choosing between them is not an abstract framework debate; it dictates whether your tail latencies stay under five milliseconds when backfilling analytical pipelines, how many engineering hours vanish into partition rebalancing, and your ongoing cloud infrastructure bill.

Kafka dominates raw stream ingestion ecosystems with mature connectors, unified KRaft metadata quorums, and Linux kernel zero-copy optimizations. Yet, Kafka exposes operational friction when engineering teams demand tens of thousands of dynamic topics, unified message-queueing semantics, or multi-tenant hardware boundaries. Pulsar solves these edge cases through decoupled storage segments, ledger-level quorum replication, and native multi-tenancy, but introduces a distributed runtime with significantly higher operational complexity.

This technical breakdown contrasts the monolithic log approach of Kafka against Pulsar decoupled storage model. We evaluate real-world p99 tail latencies, segment management, consumption semantics, Kubernetes operational tooling, and day-2 total cost of ownership to establish an objective decision matrix for your distributed systems stack.

Executive Verdict: Workload-Specific Winners for Modern Streaming

Deciding between Apache Kafka and Apache Pulsar requires matching your workload characteristics to their low-level data structures. Kafka is engineered for append-only, high-throughput sequential data streams where consumers share similar read heads. Pulsar is a dual-engine architecture: it behaves as a distributed pub-sub messaging system and a distributed event log simultaneously, trading architectural simplicity for flexibility.

Core Architectural Takeaway: If your system processes sequential telemetry where compute and storage scale linearly, Kafka provides higher raw single-node throughput and operational simplicity. If your architecture demands native multi-tenancy, fine-grained worker queues, dynamic topic creation exceeding 50,000 topics, or geo-replication across cloud regions, Pulsar delivers structural advantages that Kafka cannot match without third-party tooling.

Below is a production checklist detailing workload attributes and their optimal engine pairings.

Core Architecture: Kafka KRaft Log vs Pulsar Decoupled BookKeeper

To understand the runtime characteristics of pulsar vs kafka, we must evaluate their internal storage models. Kafka implements a single-tier, broker-centric architecture. Pulsar implements a two-tier, decoupled architecture consisting of stateless serving nodes and stateful log-segment storage nodes.

+-----------------------------------------------------------------------+ Kafka (KRaft Unified Architecture) Producer/Consumer --> [ Broker 1 (KRaft) ] <--> [ Broker 2 (KRaft) ] (Compute + Log Segments on Local NVMe Disks) +-----------------------------------------------------------------------+ +-----------------------------------------------------------------------+ Apache Pulsar (Decoupled Compute & Storage Architecture) Producer/Consumer --> [ Pulsar Broker 1 ] <--> [ Pulsar Broker 2 ] (Stateless Serving Layer) | | v v [ BookKeeper Bookie 1 ] <---> [ Bookie 2 ] <---> [ Bookie 3 ] (Stateful Distributed Storage Layer via Ledger Segments) +-----------------------------------------------------------------------+

Kafka KRaft: Monolithic Log Broker

Modern Kafka eliminates ZooKeeper, utilizing an internal Raft metadata quorum (KRaft, KIP-500) where designated brokers run active and standby metadata controllers. In Kafka, compute and storage reside on the exact same node. A topic is split into partitions. Each partition maps directly to an append-only physical directory on the broker local filesystem (NVMe SSD). Within this directory, records write to segment files (typically 1 GB rolling segments).

Kafka relies on the Linux OS Page Cache rather than managing an extensive in-memory JVM cache. When writes arrive, Kafka writes sequentially to the page cache and calls sendfile() for consumers, executing a true zero-copy data transfer directly from the OS cache to the network socket, bypassing JVM heap allocations. The operational downside: when analytical consumers read historical data, cold disk blocks evict warm pages from the page cache. This produces memory thrashing that spikes write p99 latencies for real-time streaming producers.

Apache Pulsar: Broker + BookKeeper Segments

Pulsar decouples compute from storage completely. The serving layer consists of stateless Pulsar brokers. Brokers do not persist message data locally; they maintain ownership of topic partitions, handle consumer connections, enforce authentication, and orchestrate replication. Below the broker layer sits Apache BookKeeper, an optimized distributed append-only log storage service running daemons called Bookies.

In BookKeeper, topic partitions are subdivided into horizontal chunks called Ledgers. When a ledger reaches a configured size or time boundary, it seals permanently, and Pulsar creates a new active ledger. BookKeeper spreads these ledgers across a pool of bookies using three quorum variables: Ensemble Size (E), Write Quorum (Qw), and Ack Quorum (Qa). Data does not reside permanently on a single node; it stripes across the cluster. When historical consumers read data, Bookies read from their dedicated read cache or disk without polluting the broker compute memory or evicting real-time streaming caches.

Production Benchmark Matrix: Throughput, p99 Latency, and Partition Scaling

When comparing apache pulsar vs kafka in benchmark testing, raw marketing figures often hide critical hardware bottlenecks. Under controlled lab configurations with 10GbE network interfaces, identical NVMe drives, and synchronized write guarantees, the platforms exhibit stark differences in throughput and latency stability.

Sequential Throughput and the Zero-Copy Advantage

Kafka consistently achieves superior raw throughput per dollar of hardware when processing high-volume, continuous sequential writes. Because Kafka writes directly to the Linux page cache and streams bytes out over the network interface card via kernel-level sendfile(), CPU overhead per megabyte is minimal. Pulsar decoupled architecture requires a network hop between the stateless broker and the storage bookie. This hop introduces network serialization, packet processing overhead, and CPU context switching that limits Pulsar maximum single-node throughput to roughly 60% to 75% of Kafka capacity on identical server profiles.

p99 Tail Latency Under Load

While Kafka wins on raw continuous volume, Pulsar dominates in tail latency consistency (p99 and p99.9). BookKeeper segregates writes into two physical disk locations: a synchronous Journal on a dedicated fast NVMe disk (group-committing writes sequentially) and an asynchronous Entry Log on a secondary disk array for read retrieval.

In Kafka, if multiple consumers fall behind and begin pulling cold historical segments from disk, the Linux kernel must read physical blocks into the page cache, often stalling producers waiting for dirty pages to flush. Pulsar write path remains isolated on the BookKeeper journal disk, keeping write p99 latencies under 5 milliseconds even during massive backpressure events.

The 10,000 Partition Wall

Kafka performance drops when partition counts scale past tens of thousands. Each partition requires dedicated file handles, index files, and internal memory buffers inside the JVM. When partition counts increase, Kafka sequential disk access degrades into random disk I/O, causing high latency variance. Pulsar treats topics as logical namespaces; BookKeeper combines writes from thousands of topics into a single unified journal file, enabling Pulsar to sustain over one million active topics without degrading disk performance.

Message Consumption Patterns: Log Offsets vs Multi-Mode Subscriptions

Kafka and Pulsar handle consumption differently. Kafka offers a strict distributed log model based on monotonically increasing partition offsets. Pulsar combines event log semantics with full message queue subscription patterns.

Kafka Offset Commit Engine

In Kafka, a consumer group reads a topic by assigning each partition to an individual consumer thread. A partition can only be read by one consumer within a group at any given time. Consumption progress is tracked via a monotonically increasing 64-bit integer called an offset, saved to an internal compacted topic named __consumer_offsets. Because offsets are continuous markers, consumers cannot selectively acknowledge individual messages. If record 42 fails processing while records 43 to 50 succeed, the consumer must either pause the partition, crash, or implement an external retry framework.

Pulsar Multi-Mode Subscriptions

Pulsar decouples subscription logic from partition geometry by offering four distinct subscription types:

• Exclusive: A single consumer attaches to the topic subscription, preserving strict sequential processing identical to a single-threaded Kafka consumer.
• Failover: Multiple consumers connect, but only one receives messages. If the master consumer disconnects, Pulsar fails over to the next candidate instantly.
• Shared: Acts like an enterprise message queue (e.g. RabbitMQ). Messages distribute round-robin across connected worker pools. Individual consumers acknowledge or reject (Nack) specific messages independently.
• Key_Shared: Combines stream ordering with horizontal worker elasticity. Messages containing the same routing key land on the same consumer thread, but multiple consumers can dynamically process different keys within the same physical partition.

Production Code Comparison: Consumer Acknowledgment

The following implementations illustrate the practical programming difference between committing a sequential Kafka offset versus selectively acknowledging individual messages in Pulsar.

Kafka Consumer Offset Batch Implementation (Java):

import org.apache.kafka.clients.consumer.*; import org.apache.kafka.common.TopicPartition; import java.time.Duration; import java.util.*; public class ProductionKafkaConsumer { public static void main(String[] args) { Properties props = new Properties(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-cluster:9092"); props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-processing-group"); props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); consumer.subscribe(Collections.singletonList("financial-transactions")); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord<String, String> record: records) { // If this single record fails, offset tracking blocks the entire partition processTransaction(record.key(), record.value()); } // Kafka advances the continuous offset pointer across all read partitions consumer.commitSync(); } } finally { consumer.close(); } } private static void processTransaction(String key, String value) { /* Business Logic */ } }

Pulsar Consumer Selective Acknowledgment Implementation (Java):

import org.apache.pulsar.client.api.*; public class ProductionPulsarConsumer { public static void main(String[] args) throws PulsarClientException { PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://pulsar-broker:6650").build(); Consumer<byte[]> consumer = client.newConsumer().topic("financial-transactions").subscriptionName("payment-worker-sub").subscriptionType(SubscriptionType.Shared) // Elastic pool acknowledge.subscribe(); while (true) { Message<byte[]> msg = consumer.receive(); try { processTransaction(msg.getKey(), new String(msg.getData())); // Pulsar acknowledges ONLY this individual message (Individual Ack) consumer.acknowledge(msg); } catch (Exception ex) { // Selective failure: negative acknowledge sends message back to cluster consumer.negativeAcknowledge(msg); } } } private static void processTransaction(String key, String value) { /* Business Logic */ } }

Day-2 Operations: Partition Rebalancing vs Bookie Auto-Recovery

System design comparisons often ignore cluster maintenance operations. When expanding storage pools, replacing crashed nodes, or mitigating disk skew, the operational burden between these platforms diverges sharply.

Operational Risk Point: In Kafka, adding storage to a saturated cluster forces an administrator to repartition and physically stream gigabytes of historical data over the network interface. In Pulsar, scaling storage requires no existing data migration: new ledgers are simply allocated across newly provisioned bookies.

Kafka Partition Rebalancing Friction

When you expand a Kafka cluster by provisioning additional brokers, the new nodes sit idle until you manually reassign partition replicas. The steps to execute this operational rebalancing involve significant disk and network overhead:

Pulsar Segment-Based Auto-Recovery

Pulsar decoupled architecture bypasses partition-level data migration entirely. Because partitions are broken down into small, time-bounded ledgers, cluster expansion behaves differently:

• Instant Storage Scaling: When a new BookKeeper bookie joins the cluster, stateless Pulsar brokers immediately include it in the write ensemble for new ledgers. Existing sealed historical ledgers remain untouched on old bookies. Zero bytes transfer over the network to balance cluster capacity.
• Automated Self-Healing (Auto-Recovery): If a bookie crashes permanently, the BookKeeper Auditor daemon detects the quorum breach, queries ZooKeeper/Metadata store for all ledgers referencing the dead bookie, and commands surviving bookies in the ensemble to replicate missing ledger fragments concurrently.
• Zero Broker Disruption: Pulsar brokers never touch storage replication traffic, insulating real-time producer and consumer network connections from internal storage recovery overhead.

2026 Total Cost of Ownership: Infrastructure Footprint and Operational Overhead

Evaluating the total cost of ownership (TCO) between Kafka and Pulsar requires calculating infrastructure hardware footprints, cloud network egress costs, and the human engineering overhead required to maintain each distributed topology.

Infrastructure Footprint: Node Count Comparison

Kafka single-tier architecture requires fewer instances to run a production-ready, highly available cluster. Pulsar decoupled architecture, while modular, requires two separate clusters: a broker layer and a BookKeeper layer, plus an external metadata quorum.

Cloud Networking and Egress Costs

In public cloud environments (AWS, GCP, Azure), inter-availability-zone (cross-AZ) data transfer generates major infrastructure line items. In Kafka, data writes to the partition leader and replicates to two follower brokers across AZ boundaries. A producer write results in 1x ingress + 2x cross-AZ egress bytes.

In Pulsar, a producer writes to a stateless broker, which routes data across the internal VPC network to three separate BookKeeper bookies based on the write quorum configuration. If brokers and bookies reside in different AZs without strict rack-aware placement policies, every client write generates multiple inter-zone hops, drastically inflating monthly cloud egress bills.

Engineering Labor and Kubernetes Orchestration

Running distributed streaming systems on Kubernetes introduces ongoing human maintenance costs. The Kafka ecosystem benefits from Strimzi, a mature CNCF project that handles rolling updates, dynamic configuration changes, and topic provisioning via declarative Kubernetes Custom Resource Definitions (CRDs).

Deploying Pulsar on Kubernetes requires orchestrating four distinct stateful and stateless controllers: ZooKeeper StatefulSet, BookKeeper StatefulSet, Pulsar Broker Deployment, and Pulsar Proxy Deployment. Upgrading BookKeeper bookie storage classes, managing journal disk volume claims, and handling bookie recovery configurations demand specialized distributed systems engineers, raising the total organizational payroll cost for Pulsar deployments.

Engineering Decision Matrix: Choosing Kafka or Pulsar for Your Stack

To select the correct architecture for your system, review this decision matrix detailing specific organizational criteria, infrastructure trade-offs, and ecosystem realities.

Final Architectural Decision Checklist

Both Apache Kafka and Apache Pulsar are robust, battle-tested distributed streaming platforms. Kafka architectural choice to unify compute and storage around the OS page cache optimizes for raw sequential throughput, ecosystem integration, and operational simplicity. Pulsar two-tier separation between stateless compute brokers and stateful BookKeeper ledgers trades single-node raw throughput for tail latency predictability, massive topic scaling, and dynamic multi-tenancy.

Rather than adopting a platform based on vendor benchmarks, audit your underlying workload: if your streaming architecture mirrors a unified event log, Kafka remains the industry baseline. If your architecture requires an enterprise message bus that fuses distributed task queueing with long-term segment-offloaded event streaming, Pulsar modular foundation delivers unique structural benefits.