Apache Iggy™ is one of the ASF’s newest Top-Level Projects. In this Spotlight, the ASF spoke with PMC members Piotr Gankiewicz and Kranti Parisa about the engineering choices behind Iggy, the long road to clustering, and the community that has grown around it.
What Is Apache Iggy and Why It Exists
Can you tell us a bit about Apache Iggy?
Apache Iggy is a persistent message streaming platform written in Rust. It stores messages in streams, topics, and partitions as an append-only log, while consumers read independently or through consumer groups. The server is distributed as a single binary with no external broker or database dependency. It supports TCP, QUIC, WebSocket, and HTTP, and provides SDKs for Rust, C#, C++, Java, Go, Python, Node.js, and PHP. The project also includes a CLI, Web UI, benchmarking tooling, an MCP server, and a connector runtime with sources and sinks for systems including PostgreSQL, Elasticsearch, MongoDB, Apache Iceberg™, ClickHouse, Delta Lake, Apache Doris®, and more. Iggy became a Top-Level Project on 19 August 2026 and released 0.9.0, its first version with replication, on 21 September 2026.
What is changing in the streaming landscape?
The role of streaming is broadening. Real-time data increasingly crosses boundaries between services, interactive applications, operational databases, analytical systems, data lakes, and AI-oriented workloads. That creates pressure for streaming infrastructure to be efficient enough to sit on more data paths, accessible from more environments, and interoperable with a wider ecosystem. Iggy’s response is to combine a durable append-only log with modern systems engineering, multiple native protocols, broad language support, and connectors rather than optimizing only for a single server-to-server integration model.
Apache Kafka® and other streaming platforms already existed. Why build another one?
Previous generations of streaming systems proved the importance of durable, ordered event logs. But many were designed around the hardware, runtimes, and application architectures of an earlier era. Iggy began with a different question: if we designed a persistent streaming engine from scratch today, what would we do differently? The project uses Rust and modern systems techniques including thread-per-core execution, shared-nothing architecture, Linux io_uring, zero-copy data paths, and Direct I/O. The objective is not to reproduce Apache Kafka with a different implementation language. It is to rethink the streaming engine around predictable low tail latency, high throughput, resource efficiency, a smaller operational footprint, and a simpler path from application code to durable streams.
Streaming Beyond Server-to-Server
Why is multi-protocol support a core part of Iggy’s design?
Real-time systems are no longer composed only of backend services speaking one broker protocol. Producers and consumers increasingly include backend services, browsers, developer tools, edge applications, gateways, data platforms, and AI-oriented systems. Iggy therefore treats protocol choice as part of the developer experience. TCP provides a direct high-performance path, QUIC brings a modern transport option, WebSocket makes persistent bidirectional streaming natural for browser and interactive applications, and HTTP lowers the integration barrier for environments where a dedicated streaming client is impractical. These protocols expose the same durable streaming model rather than creating separate products for separate application surfaces.
How does that change the kinds of applications a streaming engine can serve?
A conventional broker is often deployed primarily as infrastructure between backend services. Iggy’s protocol surface is intended to make the durable log directly useful across a wider range of application boundaries. The same underlying stream can participate in service-to-service eventing, telemetry and observability, real-time user experiences, data ingestion, CDC, AI and agent event flows, video or IoT pipelines, and movement of data into databases, search systems, analytic engines, and data lakes. The architectural idea is simple: durable streaming should be a reusable systems primitive, not an integration pattern limited to one class of client.
Developer Experience as a Systems Feature
Why invest in native SDKs across so many languages?
A streaming platform is only as usable as the path from an application to the stream. Iggy maintains SDKs across major application ecosystems including Rust, C#, C++, Java, Go, Python, Node.js, and PHP. The goal is to let teams use the project from the language and runtime they already operate, with idiomatic APIs and runnable examples, rather than forcing every team through a single language binding or a bespoke integration layer. Community members helped drive this breadth early in the project by contributing SDKs for the languages they used themselves.
Where do connectors fit into that strategy?
Protocols and SDKs solve application connectivity; connectors solve system connectivity. Iggy’s connector runtime links streams to operational databases, search engines, analytics systems, and lakehouse storage. Together, the protocol, SDK, and connector layers are intended to reduce the amount of glue code required to move real-time data through a heterogeneous architecture.
Designed Around Modern Systems Engineering
What are the key architectural choices behind Iggy?
Iggy uses a thread-per-core, shared-nothing design. Each CPU core runs its own thread without shared hot-path state, and Linux io_uring is used for disk and network I/O. Because the server is written in Rust, it has no garbage collector. The project has also invested in zero-copy data paths and Direct I/O. These choices are aimed at reducing scheduling, synchronization, memory-management, and I/O overhead that can show up as latency variance under load.
Why focus so much on tail latency and efficiency?
Average latency can hide the behavior users actually experience under load. Streaming infrastructure sits on the path of many downstream systems, so slow outliers can propagate through an architecture. Iggy’s engineering work therefore emphasizes predictable tail behavior as well as throughput. The project’s published benchmark shows a single-node producer P99 latency of about 1 ms, and the project provides iggy-bench so users can measure performance on their own hardware rather than relying only on published numbers.
What changed with clustering?
Clustering was one of the project’s most requested capabilities. An early experiment used Raft, but the team later moved to Viewstamped Replication Revisited (VSR)-based consensus, chosen for the project’s high-throughput replication goals. The groundwork landed in 0.7.0, and 0.9.0 is the first release designed to run as a multi-node cluster. Single-node operation remains available for simpler deployments.
Real-World Use and Interoperability
What kinds of workloads is the community targeting?
Apache Iggy is designed for real-time, data-intensive workloads where predictable latency, throughput, and efficiency matter. Current use cases and integrations span event streaming, logs and observability, real-time analytics, CDC and data pipelines, AI and agent data streams, video and IoT ingestion, and high-volume movement between operational systems and downstream stores. Apache Camel™ includes a camel-iggy component, LakeSail pairs Iggy with its Sail compute engine, and AWeber’s Imbi platform engineering tool uses Iggy for message streaming. The project also maintains a Powered By page where adopters can share their use cases.
Does adopting Iggy require an all-or-nothing migration?
No. Interoperability is an important direction for the project. The community is developing a Kafka protocol gateway so existing Kafka applications can work with Iggy, while the connector ecosystem provides paths into databases, search engines, analytic systems, and lakehouse technologies. This lets teams evaluate Iggy at a specific boundary or workload rather than treating adoption as a wholesale replacement exercise.
Getting Started and Getting Involved
What is the best way to try Iggy?
Start with the official documentation and run the server with Docker or from source. Build a producer and consumer using the SDK that matches your application stack, then use iggy-bench to test the workload on your own hardware. The published benchmark site provides a reference point, while the project’s Discord and dev mailing list are the places to ask questions and discuss designs with the community.
How can people contribute beyond code?
Try Iggy on a real workload and report what you learn. Improve documentation and examples. Help answer questions. Share an integration or operational pattern. Add your organization and use case to the Powered By page. Present at a community meetup. For code contributions, the project’s good first issues provide an accessible starting point.
What does the future hold for Apache Iggy?
The immediate focus is hardening clustering on the path to 1.0. A Kafka protocol gateway is in progress, and the community is exploring multi-leader replication per partition, tiered storage, a message registry, embedded key-value storage, and additional connectors. The broader direction remains consistent with the project’s original question: how should durable streaming evolve when modern hardware, application architectures, and real-time workloads look very different from the environment in which the previous generation of streaming platforms was designed?
Resources
- Website: https://iggy.apache.org
- Documentation: https://iggy.apache.org/docs/
- GitHub: https://github.com/apache/iggy
- Benchmarks: https://benchmarks.iggy.apache.org
- Mailing lists: https://iggy.apache.org/community/mailing-lists/
- Discord: https://discord.gg/apache-iggy
- Powered By: https://iggy.apache.org/powered-by/
About the Interviewees
Piotr Gankiewicz is the original creator of Apache Iggy and a member of the Apache Iggy PMC. He previously worked as a software architect in fintech, trading systems, and cybersecurity, was a Microsoft MVP for his open source work, and co-founded DevMentors.io.
Kranti Parisa is Chair of the Apache Iggy PMC and led Iggy’s journey into the ASF. A longtime contributor to Apache Lucene®, Apache Solr™, and Apache Camel™, he co-authored the books Apache Solr Enterprise Search Server and O’Reilly’s award-winning Reliable Machine Learning.
