Open Table Formats in 2026: Apache Iceberg v2 vs Delta Lake 3.0 on Cloud Object Storage

The data lakehouse architecture, leveraging cloud object storage (S3, GCS, Azure Blob Storage) as its foundation, has become the de facto standard for analytical workloads. Central to this evolution are open table formats: Apache Iceberg and Delta Lake. These formats imbue object storage with ACID transactionality, snapshot isolation, and time-travel capabilities, traditionally associated with relational databases. This guide dissects Apache Iceberg (v2 specification) and Delta Lake (3.0 with UniForm) in 2026, providing a technical comparison and architectural insights.

Core Principles: ACID on Object Storage

Both Iceberg and Delta Lake achieve ACID properties by maintaining a transactional log or metadata tree that tracks data file changes. Data files themselves are immutable. Updates, deletes, and inserts are handled by writing new data files and updating the metadata to reflect the new state, while marking old files as logically deleted. This copy-on-write (CoW) approach is fundamental to their operation on append-only object storage.

Apache Iceberg v2: Hierarchical Metadata and Evolved Partitioning

Iceberg's architecture is built around a hierarchical metadata structure:

1. Catalog: Stores the current metadata pointer for a table. This can be a Hive Metastore, Nessie, AWS Glue, or a custom catalog.
2. Metadata File: Points to a list of manifest lists. Each commit generates a new metadata file.
3. Manifest List: Points to a list of manifest files. Each manifest list represents a snapshot of the table.
4. Manifest File: Contains entries for data files, including their paths, partition values, and statistics (min/max values, null counts).
5. Data Files: Parquet, ORC, Avro files containing the actual data.

This structure allows for efficient pruning during query planning. Iceberg v2 significantly enhances capabilities, particularly around schema and partition evolution, and row-level operations.

Partition Evolution

Iceberg's partition evolution allows changing partitioning schemes over time without rewriting existing data. This is critical for long-lived tables where data access patterns evolve.

-- Example: Evolving a table's partitioning in Iceberg
-- Initial table creation
CREATE TABLE my_iceberg_table (
    id BIGINT,
    event_time TIMESTAMP,
    value STRING
)
USING iceberg
PARTITIONED BY (days(event_time));

-- Later, evolve partitioning to hourly
ALTER TABLE my_iceberg_table
ADD PARTITION FIELD hours(event_time);

-- Existing data remains partitioned by days, new data by hours.
-- Queries automatically handle both partition schemes.

Row-Level Deletes and Updates (Copy-on-Write vs. Merge-on-Read)

Iceberg v2 introduces two primary approaches for row-level operations:

1. Copy-on-Write (CoW): The traditional method. For a delete or update, the affected data files are rewritten, excluding or modifying the target rows. This can be I/O intensive for large files with few changes.
2. Merge-on-Read (MoR) with Delete Files: Iceberg v2 supports "delete files" (positional or equality deletes).
   • Positional Delete Files: Store the file path and position (row number) of rows to be deleted within a data file.
   • Equality Delete Files: Store the values of columns that uniquely identify rows to be deleted. During query time, the engine reads the data files and applies the delete files to filter out rows. This defers the rewrite cost to read time.

// Example: Writing an Iceberg table with Spark and performing a delete
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder()
  .appName("IcebergDeleteExample")
  .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkSessionCatalog")
  .config("spark.sql.catalog.local.type", "hadoop")
  .config("spark.sql.catalog.local.warehouse", "/tmp/iceberg_warehouse")
  .getOrCreate()

spark.sql("CREATE NAMESPACE IF NOT EXISTS local.db")
spark.sql("""
  CREATE TABLE local.db.my_iceberg_table (
    id BIGINT,
    data STRING,
    ts TIMESTAMP
  ) USING iceberg
  PARTITIONED BY (days(ts))
""")

// Insert initial data
spark.sql("""
  INSERT INTO local.db.my_iceberg_table VALUES
  (1, 'value_a', TIMESTAMP '2026-01-01 10:00:00'),
  (2, 'value_b', TIMESTAMP '2026-01-01 11:00:00'),
  (3, 'value_c', TIMESTAMP '2026-01-02 12:00:00')
""")

// Perform a delete operation (CoW by default, but can be configured for MoR with delete files)
// For MoR, ensure the table property 'format-version' is 2 and 'write.delete.mode' is 'merge-on-read'
spark.sql("ALTER TABLE local.db.my_iceberg_table SET TBLPROPERTIES ('format-version'='2', 'write.delete.mode'='merge-on-read')")

spark.sql("DELETE FROM local.db.my_iceberg_table WHERE id = 2")

// Verify the delete
spark.sql("SELECT * FROM local.db.my_iceberg_table").show()
// Expected output: rows with id 1 and 3

Delta Lake 3.0: Transaction Log and UniForm

Delta Lake's core is its transaction log, a series of JSON files (and Parquet checkpoints for efficiency) stored alongside the data files in the _delta_log directory. Each entry in the log represents an atomic commit, detailing additions and removals of data files.

UniForm (Universal Format)

Delta Lake 3.0 introduces UniForm, a critical feature for interoperability. UniForm generates Iceberg and Hudi metadata alongside Delta Lake's native transaction log. This allows engines that natively understand Iceberg or Hudi to read Delta tables without requiring Delta Lake connectors. This is a significant step towards true open table format interoperability.

// Example: Creating a Delta table with UniForm enabled in Spark
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("DeltaUniFormExample")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .getOrCreate()

// Create a Delta table with UniForm enabled
spark.sql("""
  CREATE TABLE my_delta_table (
    id BIGINT,
    data STRING,
    ts TIMESTAMP
  ) USING DELTA
  TBLPROPERTIES (
    'delta.enableIcebergCompatV2' = 'true',
    'delta.universalFormat.enabledFeatures' = 'iceberg'
  )
  PARTITIONED BY (CAST(ts AS DATE))
""")

// Insert data
spark.sql("""
  INSERT INTO my_delta_table VALUES
  (1, 'delta_val_a', TIMESTAMP '2026-01-01 10:00:00'),
  (2, 'delta_val_b', TIMESTAMP '2026-01-01 11:00:00')
""")

// Now, this Delta table can be read by Iceberg-compatible engines
// (e.g., Trino, DuckDB) directly, without Delta Lake specific connectors.
// The Iceberg metadata is automatically generated and kept in sync.

Partitioning and Schema Evolution

Delta Lake also supports schema evolution (adding/dropping columns, changing types) and partition evolution. Partitioning is typically defined at table creation and can be changed by rewriting the table.

-- Example: Delta Lake schema evolution
ALTER TABLE my_delta_table ADD COLUMNS (new_col INT);

Row-Level Deletes and Updates

Delta Lake primarily uses a CoW approach for DELETE, UPDATE, and MERGE operations. It identifies affected files, rewrites them with changes, and records the new files in the transaction log. For large tables with frequent small updates, this can lead to file amplification and performance overhead. Databricks has introduced "liquid clustering" and "photon-accelerated writes" to mitigate some of these issues, but the core mechanism remains CoW.

Engine Interoperability

The ability to query tables from various engines is a cornerstone of the lakehouse.

Trino

Trino (formerly PrestoSQL) has robust connectors for both Iceberg and Delta Lake.

-- Trino: Querying an Iceberg table
-- Catalog configuration (e.g., in etc/catalog/iceberg.properties)
-- connector.name=iceberg
-- iceberg.catalog.type=rest # or hive, glue, hadoop

SELECT * FROM iceberg.db.my_iceberg_table WHERE id = 1;

-- Trino: Querying a Delta Lake table (with UniForm, via Iceberg connector)
-- Assuming UniForm is enabled on the Delta table, Trino's Iceberg connector
-- can read it directly.
SELECT * FROM iceberg.db.my_delta_table WHERE id = 1;

-- Trino: Querying a Delta Lake table (via Delta Lake connector)
-- Catalog configuration (e.g., in etc/catalog/delta.properties)
-- connector.name=delta_lake
-- delta.catalog.type=glue # or hive, hadoop

SELECT * FROM delta.db.my_delta_table WHERE id = 1;

Apache Spark

Spark is the native environment for both Iceberg and Delta Lake.

// Spark: Reading Iceberg
spark.read.format("iceberg").load("local.db.my_iceberg_table").show()

// Spark: Reading Delta Lake
spark.read.format("delta").load("/path/to/my_delta_table").show()

DuckDB

DuckDB, an in-process OLAP database, offers excellent local analytical capabilities.

# Python with DuckDB: Reading Iceberg
import duckdb

con = duckdb.connect(database=':memory:', read_only=False)
con.execute("INSTALL iceberg;")
con.execute("LOAD iceberg;")

# Point to the Iceberg table's metadata file or catalog
# For local Iceberg table created by Spark in /tmp/iceberg_warehouse/db/my_iceberg_table
con.execute("CREATE OR REPLACE TABLE my_iceberg_table AS SELECT * FROM iceberg_scan('/tmp/iceberg_warehouse/db/my_iceberg_table/metadata/v*.metadata.json');")
con.execute("SELECT * FROM my_iceberg_table;").fetchdf()

# Python with DuckDB: Reading Delta Lake (with UniForm, via Iceberg extension)
# Assuming UniForm is enabled on the Delta table
con.execute("INSTALL delta;") # Delta extension is also available
con.execute("LOAD delta;")
con.execute("CREATE OR REPLACE TABLE my_delta_table AS SELECT * FROM delta_scan('/path/to/my_delta_table');")
con.execute("SELECT * FROM my_delta_table;").fetchdf()

# If UniForm is enabled, you might be able to use iceberg_scan directly on the Delta path
# (This depends on the exact implementation of DuckDB's Iceberg connector and UniForm's metadata sync)
# con.execute("CREATE OR REPLACE TABLE my_delta_table_as_iceberg AS SELECT * FROM iceberg_scan('/path/to/my_delta_table');")

Architectural Comparison and Tradeoffs

Tradeoffs

• Metadata Overhead: Iceberg's hierarchical metadata can lead to more small files (manifests) for very large tables with many partitions and frequent commits. However, its pruning capabilities are highly efficient. Delta Lake's transaction log is simpler but can grow very large, necessitating checkpointing.
• Row-Level Operations: Iceberg's MoR with delete files offers a compelling alternative to CoW, especially for tables with high update/delete frequency on a small percentage of rows, reducing write amplification. Delta Lake's CoW can be very performant for batch operations but might incur higher rewrite costs for fine-grained changes.
• Interoperability: Iceberg's design is inherently more open and engine-agnostic from its inception. Delta Lake's UniForm is a strategic move to achieve similar broad interoperability by generating Iceberg-compatible metadata, effectively making Delta tables readable by Iceberg engines. This is a significant win for the ecosystem.
• Community & Ecosystem: Both have strong, active communities. Iceberg benefits from Apache Foundation governance, fostering a diverse ecosystem. Delta Lake, while open-sourced, has strong ties to Databricks, which drives much of its development and feature set.

Production Gotchas & Troubleshooting

1. Small File Problem (Both Iceberg & Delta Lake): Frequent small appends or updates can lead to an explosion of small data files. This degrades query performance due to increased metadata processing and I/O overhead.

   • Fix: Implement compaction jobs (e.g., using Spark OPTIMIZE for Delta, or Iceberg's rewrite_data_files action) to merge small files into larger ones. Schedule these regularly.
   scalaCopy

   // Delta Lake compaction
   spark.sql("OPTIMIZE my_delta_table ZORDER BY (event_time)")
   
   // Iceberg compaction (Spark example)
   import org.apache.iceberg.spark.actions.SparkActions
   SparkActions.get(spark).rewriteDataFiles(spark.table("local.db.my_iceberg_table")).execute()
   

2. Metadata Bloat (Both Iceberg & Delta Lake): Over time, the transaction log (_delta_log) or Iceberg's metadata files can become very large, impacting commit times and query planning.

   • Fix: Configure retention policies for old snapshots. For Delta Lake, use VACUUM with a retention period. For Iceberg, expire_snapshots action.
   scalaCopy

   // Delta Lake vacuum (retains 7 days by default, can be configured)
   spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled = false") // Disable safety check for testing
   spark.sql("VACUUM my_delta_table RETAIN 0 HOURS") // DANGER: Deletes files not referenced by current snapshot
   spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled = true")
   
   // Iceberg expire snapshots (retains last N snapshots or snapshots newer than a timestamp)
   SparkActions.get(spark).expireSnapshots(spark.table("local.db.my_iceberg_table"))
     .retainLast(10) // Retain last 10 snapshots
     .execute()
   

3. Schema Evolution Conflicts: While both support schema evolution, unexpected data types or column renames can cause issues.

   • Fix: Validate incoming data against the table schema before writing. Use spark.sql.legacy.allowUntypedScalaUDF for Spark if encountering UDF issues with schema evolution. For Iceberg, ensure schema.auto.merge is not blindly enabled in production without careful consideration.

4. Object Storage Rate Limiting/Throttling: High concurrency or frequent small operations can hit object storage API limits (e.g., S3 5000 PUT/GET requests per second per prefix).

   • Fix: Optimize writes for larger file sizes. Implement exponential backoff and retry logic in data ingestion pipelines. Distribute data across more S3 prefixes if possible. Monitor cloud provider metrics for throttling.

5. Inconsistent Reads (Time Travel): If not explicitly using time travel, queries might pick up stale data if a concurrent write commits just after the query starts planning but before it reads data files.

   • Fix: For critical analytical queries requiring strict consistency, use time travel to query a specific snapshot ID or timestamp.
   sqlCopy

   -- Delta Lake time travel
   SELECT * FROM my_delta_table VERSION AS OF 5;
   SELECT * FROM my_delta_table TIMESTAMP AS OF '2026-01-01 12:00:00';
   
   -- Iceberg time travel
   SELECT * FROM local.db.my_iceberg_table FOR SYSTEM_VERSION AS OF 1234567890123456L; -- Snapshot ID
   SELECT * FROM local.db.my_iceberg_table FOR SYSTEM_TIME AS OF '2026-01-01 12:00:00';
   

Frequently Asked Questions

1. When should I choose Iceberg over Delta Lake, or vice-versa? Choose Iceberg if: you prioritize a truly open, vendor-agnostic ecosystem, require advanced partition evolution, or need efficient MoR for frequent row-level updates/deletes. Choose Delta Lake if: you are heavily invested in the Databricks ecosystem, prefer a simpler transaction log model, or benefit from Databricks-specific performance optimizations (e.g., Photon, Liquid Clustering). With UniForm, Delta Lake's interoperability gap with Iceberg is significantly reduced.

2. How does UniForm impact my decision between Iceberg and Delta Lake? UniForm makes Delta Lake tables readable by Iceberg-compatible engines. This significantly reduces the "vendor lock-in" concern for Delta Lake. If your primary concern was Iceberg's broader engine support, UniForm largely mitigates that. However, UniForm currently only provides read-only Iceberg compatibility; writes still require Delta Lake.

3. What are the performance implications of Iceberg's MoR vs. Delta Lake's CoW for row-level operations? Iceberg's MoR can offer better write performance for frequent, small updates/deletes by deferring the rewrite cost to read time. This is beneficial when write amplification is a concern. However, MoR can introduce read overhead as the engine must apply delete files. Delta Lake's CoW ensures reads are always clean but can lead to higher write amplification and I/O for fine-grained changes. The optimal choice depends on your read/write patterns and latency requirements.

4. Can I migrate an existing Hive table to Iceberg or Delta Lake? Yes, both formats provide tools for migration. For Iceberg, you can use ALTER TABLE ... SET TBLPROPERTIES ('format'='iceberg') in Spark for unmanaged tables, or use the migrate action. For Delta Lake, CONVERT TO DELTA command can convert Parquet or Hive tables in place. Always test migrations thoroughly on a subset of data first.

5. What is the role of a catalog in these table formats? The catalog is crucial. It acts as the central registry for tables, storing the current metadata pointer (for Iceberg) or the base path (for Delta Lake). Without a catalog, engines cannot locate and interpret the table's state. Common catalogs include Hive Metastore, AWS Glue Data Catalog, Project Nessie (for Iceberg), and Databricks Unity Catalog (for Delta Lake).