
Apache Iceberg vs Proprietary Metadata Layers: How Do They Play With Agentic AI?
As organizations extend their data platforms to support agentic AI, the metadata layer sitting between raw storage and every query engine becomes an important choice. Openness at the table-format and catalog layer directly determines how easily AI agents can autonomously query, reason over, and take action on data safely and efficiently. While proprietary metadata layers can be excellent within their own walls, the lock-in can become a constraint.
A data lakehouse stores raw data (typically Parquet files) in object storage such as Amazon Simple Storage Service (S3), Azure Data Lake Storage (ADLS), or Google Cloud Storage (GCS). On its own, that storage has no concept of transactions, schema, or version history. The table format and its catalog sit above the files and add the structure that engines and applications rely on, including atomicity, consistency, isolation, and durability (ACID) transactions, schema and partition evolution, time travel, and effective query planning.
Broadly speaking, there are two ways to implement this layer:
- Open table formats, such as Apache Iceberg and Delta Lake, which are open source and allow any compliant engine to read and write, typically through an open catalog such as a REST catalog.
- Proprietary metadata layers, which are built into platforms such as a single-vendor warehouse, can only be accessed by other engines through vendor-provided bridges, if at all.
Debates about vendor lock-in are nothing new, but the distinctions matter more for agentic AI. While human data analysts may run scheduled dashboards and occasional ad hoc queries using one or two tools, AI agents query continuously and unpredictably, across many different frameworks and tools at once.
The bottom line? The more open and standards-based the table format and catalog are, the less custom integration work is required every time a new agent framework, model provider, or internal tool is adopted.
Apache Iceberg vs proprietary metadata layers
Standardizing on Iceberg as the table format, accessed through a Representation State Transfer (REST) catalog, doesn’t preclude using proprietary-platform compute engines where they add value. Most major vendors, including Snowflake and Databricks, now expose Iceberg-compatible REST catalog APIs. The key is to confirm that any vendor under consideration allows Iceberg tables to be written and read natively, not just imported/exported, and that an agent-facing access path, such as Model Context Protocol (MCP), is available or on a committed roadmap.
Below are some factors to consider when comparing Iceberg to proprietary metadata layers.
| Capability | Apache Iceberg open table format | Proprietary metadata layer |
|---|---|---|
| Format ownership | Open source via the Apache Software Foundation; implemented by many engines | Owned and controlled by a single vendor |
| Engine access | Any compliant engine (Spark, Trino, Flink, DuckDB, Snowflake, Databricks, etc.) reads/writes the same table | Typically limited to the vendor’s own engine, or read-only access for others |
| Catalog interoperability | REST Catalog standard (e.g., Apache Polaris) enables multi-engine governance on one copy of data | Proprietary catalog; interoperability depends on vendor-provided bridges or exports |
| Data portability / lock-in risk | Low; data stays in Parquet/ORC in your object storage; metadata is swappable | Higher; migrating off the platform often requires re-ingesting or exporting data |
| AI/agent tool ecosystem | Broad and growing MCP/agent tool support because many vendors build against the same open standard | Narrower; agent tooling limited to what that vendor ships or partners on |
| Governance | Emerging standard via REST catalog | Mature within the vendor’s walled garden, but not portable to other engines |
| Pace of innovation | Community-driven; features such as Iceberg v3 ship across many vendors simultaneously | Vendor-driven; features arrive on the vendor’s roadmap and timeline |
Five questions to ask before choosing a data catalog
It’s worth piloting agentic workloads against at least one open Iceberg/REST catalog and one proprietary option before committing to a long-term platform. Below are five questions to ask about the table format and catalog layer when evaluating vendors:
- Can every tool or AI agent we use actually read the same data? If you bring in a new AI tool next year, can the solution read your data on day one, or do you need the vendor’s permission and custom engineering first?
- Can AI agents connect the way modern AI tools expect to connect? If the vendor doesn’t support MCP, every new AI project may need custom, one-off engineering work to get the agent talking to your data.
- Can you limit exactly what each AI agent is allowed to see? An agent built for customer support should not be able to see payroll or financial data, even by accident. Check how granular a vendor’s permissions are for AI agents specifically, and whether access can be limited to exactly what’s needed and automatically expire.
- Can the system get smarter over time, or does it forget everything? As AI agents use your data, they can learn useful things along the way, such as which reports are actually used or how data relates across departments, which could help the next agent work faster. Look for a platform that allows the accumulated knowledge to be saved and reused.
- Will performance hold up once AI agents are barraging the database with questions? A human analyst might run a handful of reports a day. AI agents can generate hundreds or thousands of small requests in rapid succession. This can slow a system down in ways that aren’t obvious from a normal demo.
Open doesn’t mean unmanaged
One common misconception is that choosing an open architecture means giving up governance.
It doesn’t.
An open table format can coexist with strong identity controls, catalog-level permissions, lineage, monitoring, encryption, and policy enforcement. The challenge is that those capabilities may come from several components rather than a single vendor platform.
That creates a different responsibility for the CTO: defining which capabilities should remain portable and which are worth tying to a specific platform.
A useful architecture might keep data in an open table format and object storage while using proprietary engines for workloads where they provide a meaningful advantage. Another organization may prefer a more integrated platform because operational simplicity matters more than portability.
The right choice depends on the organization’s workload, governance requirements, skills, and tolerance for platform dependency.
In a recent Cisco and Omdia study, The Race to Agentic AI: Why Infrastructure Will Make or Break Workforce Transformation, 87% of surveyed executives said their strategic priorities have been fundamentally reshaped by agentic AI. On average, they estimate that 55% of their workforce will regularly work alongside agents within the next 24 months. Is your data architecture ready?