Sponsor Success at Apache: Google and Apache Iceberg

The Apache Software Foundation (ASF) is home to more than 350 open source projects and initiatives, all powered by a vibrant, global community of contributors. Among these projects, Apache Iceberg has emerged as a cornerstone of modern data architecture, revolutionizing how organizations manage massive analytical datasets with its high-performance, open-table format.

None of this innovation happens in a vacuum. Invaluable engineering talent contributing real-world expertise to the community accelerates Apache Iceberg’s rapid advancement and enhances its robust stability.

In this edition of Sponsor Success at Apache, we sit down with three key contributors from Google to discuss their journey with Apache Iceberg, why open source collaboration is vital to their engineering culture, and how their contributions are shaping the future of big data.

Perspectives from Google on Apache Iceberg ecosystem and community

Talat Uyayer

Q: You are an active presence in the Apache Iceberg community and delivered multiple talks at the 2026 Iceberg Summit. What excites you the most about the upcoming Iceberg V4 updates?

Talat: The Summit was incredible this year. I had three talks covering CDC, the Iceberg V4 specification, and security. What makes the V4 spec so exciting is how it improves Iceberg’s ability to work at enterprise scale in multiple ways by completely rethinking the metadata structure.

For example, the community is introducing Single-File Commits, which drastically reduces I/O overhead—a massive win for high-frequency streaming ingestion. Historically, the metadata tree was optimized for large, slow-moving tables. However, high-frequency streaming workloads introduce copy-on-write overhead and write amplification that can cause latency spikes. Shifting to single-file commits enables low-latency writes without sacrificing read performance. We are also moving metadata into Apache Parquet for faster columnar reads, and completely redesigning column statistics. Together, these V4 changes aren’t just incremental tweaks; they are re-tuning Iceberg to be faster, lighter, and vastly more flexible for enterprise operators.

Another critical aspect is the shift in focus toward Fine-Grained Access Control, using the Iceberg REST Catalog as a centralized security control plane. It ultimately comes down to your trust boundaries. You can either use an Access Decision Exchange Model, where the catalog pushes evaluated policies down to a trusted client engine to enforce, or a Materialized Scan Model for untrusted environments, where the catalog pre-applies row filters and column masks before returning file pointers. It’s complex, but it’s exactly what enterprises need to build compliant lakehouses at scale.

Q: You have a strong background in several different ASF communities, including Kafka, Beam, and Flink. How do your experiences in other communities influence your work on Apache Iceberg?

Talat: For a while, my core focus was building out streaming infrastructures utilizing Kafka, Beam, and Flink. Because I spent so much time in that streaming-native headspace, I bring the perspective of those users to Iceberg. Iceberg was initially designed for batch workloads, but now the community is addressing streaming needs with the V4 spec. My background helps me anticipate the friction points a streaming engineer will hit, and ensures my contributions feel like native extensions rather than bolted-on pieces and meet the high performance and reliability standards those streaming communities expect.

Q: You’ve managed massive-scale deployments of open source technologies. As an ASF Member, how does this inform the way you think about the project and contribute to its future?

Talat: When you manage massive-scale deployments on the enterprise side, you learn very quickly that “Day 2” operations are where a project succeeds or fails. Features are great, but at scale, what you really care about is observability, clean logging, and graceful degradation. As an ASF Member, I bring that operational empathy directly back to the project. When we are discussing the roadmap or reviewing a new feature, I’m usually the one asking: “How are we going to monitor this? What happens when it fails at scale? Is the upgrade path safe?” My goal is to ensure we aren’t just building innovative software, but software that enterprise operators can actually trust and maintain without burning out.

Q: You work closely with PMCs and committers in the Apache Iceberg community. What is it like collaborating on ideas and coordinating with so many different contributors, especially when developers from competing companies are often working together on shared proposals?

Talat: Working with developers from competing companies is actually one of the most fascinating parts of the Iceberg community. While our employers might compete fiercely on compute engines or managed services, everyone recognizes that a rock-solid open storage standard benefits the entire industry. Because Iceberg is an ASF project, we operate under the “Community over Code” ethos—meaning we contribute as individuals, not as corporate representatives. We all share the exact same goal: making Iceberg the best table format possible. When we collaborate on shared proposals, like the V4 spec, we leave our corporate badges at the door. It creates an environment where we can combine the best ideas from across the industry to solve massive data problems.

Q: We know you’ve been heavily involved in shaping the future of Iceberg. Could you tell us a bit about your recent Relative Paths proposal, or another current proposal that you are particularly passionate about right now?

Talat: I’m really passionate about the Relative Paths proposal that we are pushing for in the V4 spec. Right now, Iceberg stores absolute file paths in its metadata. That works fine until an enterprise needs to move a table. If you change a storage bucket or migrate to a new region, all those absolute paths instantly break, and you have to run expensive, distributed jobs just to rewrite the metadata.

The proposal changes this so that file references are stored relative to a base URI. It sounds like a simple string change, but the architectural impact is massive. It completely decouples the logical table structure from the physical storage layer. Suddenly, migrating a massive dataset, cloning it for testing, or setting up disaster recovery backups become trivial operations.

Ahmed Abualsaud

Q: You were one of the key developers of Apache Beam’s IcebergIO connector. What was the most rewarding part of building it?

Ahmed: One of our main goals for IcebergIO was to build a multi-language connector from the get-go. We wanted to write the core, feature-rich implementation once in Java, then package it to make it instantly usable in Python, SQL, or YAML pipelines without forcing users to worry about Java dependencies. It’s incredibly rewarding to complete a feature or bug fix in Java and know that users of other languages benefit immediately.

We took it a step further by capitalizing on Beam’s portability framework and wrapping the connector in the new ManagedIO layer. This gives it a unified API across different SDKs and allows the runner to dynamically upgrade the underlying connector—pulling in new features, optimizations, bug fixes, and security patches—without requiring the user to change or resubmit their pipeline.

Q: What were some of the standout technical challenges of implementing Managed I/O across Java, Python, and YAML?

Implementing a multi-language connector does come with challenges. First, we can’t use language-native objects to instantiate the connector. Normally, a Java connector’s configuration includes POJOs, whereas Python’s includes custom lambda expressions. When prioritizing portability, we have to design configurations that every language understands.

This forced us to think creatively when implementing support for writing to dynamic destinations. Usually, users pass a native function to determine the destination per record. In the absence of native functions, we turned to string interpolation as a language-agnostic expression. Users provide a template string with placeholders representing column names (e.g., “namespace.table_{year}”), which evaluates at runtime against each record’s column values. It’s a simple, clean approach accessible to all languages.

The second challenge we’re currently facing is that pipeline data types need to be language-agnostic and pre-defined. This creates an obstacle for multi-language connectors working with dynamic schemas. Neighboring open source projects have started adopting the new Variant data type to handle semi-structured data. It’s a promising option, and we’re considering bringing it into Beam to provide an elegant solution.

Q: When building connectors meant for massive data processing and working across multiple programming languages, what are your top tips for successful collaboration and integration?

Ahmed: Building scalable and resilient connectors takes time and requires a deep understanding of the target data store. While high-level documentation and blog posts are helpful early on, the real technical exploration comes when you start prototyping. I usually begin by writing a simple, one-class Java application to build a mental map of the order of operations, learn how data flows, and identify individual stages that can be mapped to separate transforms.

It also helps to investigate neighboring open source projects to discover relevant APIs and utilities that might not be obvious in the official documentation. When designing features for IcebergIO, I thoroughly analyzed how Spark and Flink implemented them. The amazing thing about open source is that all ideas, discussions, and implementations are free and publicly accessible. We can study existing strategies for write-coordination, maintenance procedures, and state management, and adapt them directly.

Finally, work with the developers of the target system to challenge your assumptions. If you need to comment on an older design document to dig deeper or drop a message on a public Slack channel or dev list, the maintainers are usually quick to help. Communicating openly ensures your connector aligns with how the target system was actually built to operate.

Q: How does bringing familiar tools like SQL into the Beam and Iceberg ecosystems empower users to tackle complex data challenges more efficiently?

Ahmed: These projects are powerful and sophisticated, but they’re also conceptually heavy. Beam manages complex distributed batch and streaming execution like windowing and stateful processing, while Iceberg handles atomic table states like snapshots, schema evolution, and ACID transactions. Often, the user doesn’t need to know all of these details. If we stopped at just the core libraries, we’d be forcing every data analyst to master low-level APIs and distributed systems concepts, which is too high a barrier to entry.

That’s why SQL is such a vital declarative interface. It allows users to express what data they want to process without worrying about how it’s handled by the underlying storage and execution engines. With SQL in Beam, a user can write a query to join a real-time Kafka stream with historical data in an Iceberg table, and write the results right back to Iceberg. The exact same SQL logic scales from a small local test to a multi-terabyte production pipeline, breaking down organizational silos so data scientists, analysts, and engineers can share a common language.

I’ll admit that, as a community, we focused so heavily on other core parts of Beam that the SQL component was inadvertently neglected for a few years. But we’ve recognized this gap and have put a lot of effort over the past year into catching up. We’ve rolled out support for standard DDL operations and cross-catalog queries, adding features designed to deliver a genuinely intuitive, SQL-native experience for data engineers.

Q: As the author of the official Iceberg Quickstart and a frequent community presenter, what advice do you have for newcomers seeking to make their very first contribution to the Iceberg or Beam ecosystems?

Ahmed: It’s normal to feel a little intimidated at first (I certainly did), but you’ll find that both communities are incredibly welcoming. It definitely helps to know the right entry points. Both projects do a great job of curating entry-level tasks using GitHub labels: “good first issue” for Beam, and “beginner” for Iceberg.

Your first contribution doesn’t have to be massive. You can build credibility and trust just by fixing outdated documentation or setup instructions. Because quickstarts are the front door to a project, improving them is highly impactful.

Before you get too deep into coding a solution, post your ideas on the project’s dev mailing list or Slack channels to get high-level feedback from maintainers and ensure your work aligns with the roadmap.

Finally, make your pull requests easy to review. As someone who regularly reviews PRs in Beam, I always appreciate clear, self-contained descriptions alongside well-commented code. It makes the entire process much smoother for everyone.

Parth Desai

Q: What do you see as the most vital future opportunities for the project?

Parth: The most vital future opportunities lie in fulfilling the promise of a truly open, differentiated lakehouse architecture. We are focusing our efforts on three strategic priorities to help customers bridge the gap between open source flexibility and enterprise performance:

1. Improving Interoperability and Open-Source Compatibility: We intend to further enhance the integration between BigQuery and the Apache Iceberg specification to establish it as a completely compliant query engine. We are working toward full specification compatibility, including support for advanced partitioning, custom table properties, and V3 features like variant and deletion vectors, while preparing for the upcoming V4 specification.
2. Maximizing Price-Performance and Storage Efficiency: Our ambition is to establish BigQuery as the premier industry platform for executing high-performance analytical workloads on Iceberg storage. We are directing substantial investments toward optimizations across the file I/O, query engine, and storage tiers to bring BigQuery’s performance on Iceberg to parity with native BigQuery storage, while refining automated background processes for table management.
   
3. Bringing BigQuery’s Unique Value to Iceberg: We intend to extend BigQuery’s specialized features to Iceberg storage, including Search/Vector indexes, Vector embeddings, BigQuery Graph, and multi-modal AI capabilities. This ensures that new advancements remain available to users who prioritize open-format interoperability without forcing them to compromise on functionality.

Q: Why is building interoperability, federation and openness amongst data systems so crucial for developers, and how does an open-source project like Apache Iceberg make it possible?

Parth: Data gravity is one of the most powerful forces in technology. Moving data from one system to another is expensive and introduces a lot of risk, including security, timeliness, and lineage tracing. Developers don’t want to move data; they want to bring the best compute engine for the job directly to it. They want to use BigQuery for data science and AI, Apache Spark for heavy distributed transformations, and Apache Flink for stream processing—all on the exact same dataset.

Storage layers that support interoperability and openness protect developers from being trapped in a walled garden by serving the common language for data. Because Iceberg defines a clear, open specification for table metadata, schema evolution, and transactional guarantees, it allows different, independent compute engines to concurrently read and write to the same Iceberg table in object storage without needing to move, copy, or risk corrupting the data. Iceberg has effectively commoditized the storage layer, shifting the competitive focus to where it belongs: engine performance, usability, and advanced analytics.

Q: Your talks have demonstrated how to leverage an open lakehouse foundation to power GenAI with Gemini. How do you see the intersection of open source data formats and artificial intelligence evolving over the next few years?

Parth: We are entering a phase where AI strategy is data strategy. Foundation models are incredibly powerful, but their true enterprise value is unlocked only when they are grounded in a company’s proprietary, trusted data. Open source data formats are rapidly becoming the bedrock of this grounding process.

Over the next few years, I expect this intersection to evolve in a few key ways. First, we will see RAG execute natively within the lakehouse. Instead of moving data into specialized vector databases, organizations will leverage Iceberg’s scale and vector search capabilities to ground LLMs directly on the open data lake.

Second, data versioning and lineage will become critical AI safety mechanisms. Because Iceberg inherently supports time-travel and snapshots, developers can point an AI model to an exact state of data from a specific point in time. This is critical for model reproducibility, auditing, and debugging hallucination issues.

Ultimately, by standardizing on an open foundation like Apache Iceberg, organizations ensure that their data remains agile, governed, and ready to fuel whatever the next generation of AI brings, seamlessly bridging the gap between vast data stores and the transformative reasoning power of models.

Driving Innovation Together

The contributions from Google highlight a core tenet of The Apache Way: that community-driven software thrives when diverse voices and real-world use cases come together. By actively investing in Apache Iceberg, Google is ensuring that the project remains scalable, reliable, and accessible for the entire global data ecosystem.

We are incredibly grateful for Google’s continued sponsorship and engineering leadership.

To learn more about Apache Iceberg and how to get involved, visit the Apache Iceberg Project Page. To find out how your organization can support the ASF, visit our Sponsorship Page.