
The data lakehouse architecture, leveraging cloud object storage (S3, GCS, Azure Blob Storage) as its foundation, has become the de facto standard for analytical workloads. Central to this evolution are open table formats: Apache Iceberg and Delta Lake. These formats imbue object storage with ACID transactionality, snapshot isolation, and time-travel capabilities, traditionally associated with relational databases. This guide dissects Apache Iceberg (v2 specification) and Delta Lake (3.0 with UniForm) in 2026, providing a technical comparison and architectural insights.
Both Iceberg and Delta Lake achieve ACID properties by maintaining a transactional log or metadata tree that tracks data file changes. Data files themselves are immutable. Updates, deletes, and inserts are handled by writing new data files and updating the metadata to reflect the new state, while marking old files as logically deleted. This copy-on-write (CoW) approach is fundamental to their operation on append-only object storage.
Iceberg's architecture is built around a hierarchical metadata structure:
This structure allows for efficient pruning during query planning. Iceberg v2 significantly enhances capabilities, particularly around schema and partition evolution, and row-level operations.
Iceberg's partition evolution allows changing partitioning schemes over time without rewriting existing data. This is critical for long-lived tables where data access patterns evolve.
-- Example: Evolving a table's partitioning in Iceberg
-- Initial table creation
CREATE TABLE my_iceberg_table (
id BIGINT,
event_time TIMESTAMP,
value STRING
)
USING iceberg
PARTITIONED BY (days(event_time));
-- Later, evolve partitioning to hourly
ALTER TABLE my_iceberg_table
ADD PARTITION FIELD hours(event_time);
-- Existing data remains partitioned by days, new data by hours.
-- Queries automatically handle both partition schemes.
Iceberg v2 introduces two primary approaches for row-level operations:
// Example: Writing an Iceberg table with Spark and performing a delete
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder()
.appName("IcebergDeleteExample")
.config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.local.type", "hadoop")
.config("spark.sql.catalog.local.warehouse", "/tmp/iceberg_warehouse")
.getOrCreate()
spark.sql("CREATE NAMESPACE IF NOT EXISTS local.db")
spark.sql("""
CREATE TABLE local.db.my_iceberg_table (
id BIGINT,
data STRING,
ts TIMESTAMP
) USING iceberg
PARTITIONED BY (days(ts))
""")
// Insert initial data
spark.sql("""
INSERT INTO local.db.my_iceberg_table VALUES
(1, 'value_a', TIMESTAMP '2026-01-01 10:00:00'),
(2, 'value_b', TIMESTAMP '2026-01-01 11:00:00'),
(3, 'value_c', TIMESTAMP '2026-01-02 12:00:00')
""")
// Perform a delete operation (CoW by default, but can be configured for MoR with delete files)
// For MoR, ensure the table property 'format-version' is 2 and 'write.delete.mode' is 'merge-on-read'
spark.sql("ALTER TABLE local.db.my_iceberg_table SET TBLPROPERTIES ('format-version'='2', 'write.delete.mode'='merge-on-read')")
spark.sql("DELETE FROM local.db.my_iceberg_table WHERE id = 2")
// Verify the delete
spark.sql("SELECT * FROM local.db.my_iceberg_table").show()
// Expected output: rows with id 1 and 3
Delta Lake's core is its transaction log, a series of JSON files (and Parquet checkpoints for efficiency) stored alongside the data files in the _delta_log directory. Each entry in the log represents an atomic commit, detailing additions and removals of data files.
Delta Lake 3.0 introduces UniForm, a critical feature for interoperability. UniForm generates Iceberg and Hudi metadata alongside Delta Lake's native transaction log. This allows engines that natively understand Iceberg or Hudi to read Delta tables without requiring Delta Lake connectors. This is a significant step towards true open table format interoperability.
// Example: Creating a Delta table with UniForm enabled in Spark
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("DeltaUniFormExample")
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
.getOrCreate()
// Create a Delta table with UniForm enabled
spark.sql("""
CREATE TABLE my_delta_table (
id BIGINT,
data STRING,
ts TIMESTAMP
) USING DELTA
TBLPROPERTIES (
'delta.enableIcebergCompatV2' = 'true',
'delta.universalFormat.enabledFeatures' = 'iceberg'
)
PARTITIONED BY (CAST(ts AS DATE))
""")
// Insert data
spark.sql("""
INSERT INTO my_delta_table VALUES
(1, 'delta_val_a', TIMESTAMP '2026-01-01 10:00:00'),
(2, 'delta_val_b', TIMESTAMP '2026-01-01 11:00:00')
""")
// Now, this Delta table can be read by Iceberg-compatible engines
// (e.g., Trino, DuckDB) directly, without Delta Lake specific connectors.
// The Iceberg metadata is automatically generated and kept in sync.
Delta Lake also supports schema evolution (adding/dropping columns, changing types) and partition evolution. Partitioning is typically defined at table creation and can be changed by rewriting the table.
-- Example: Delta Lake schema evolution
ALTER TABLE my_delta_table ADD COLUMNS (new_col INT);
Delta Lake primarily uses a CoW approach for DELETE, UPDATE, and MERGE operations. It identifies affected files, rewrites them with changes, and records the new files in the transaction log. For large tables with frequent small updates, this can lead to file amplification and performance overhead. Databricks has introduced "liquid clustering" and "photon-accelerated writes" to mitigate some of these issues, but the core mechanism remains CoW.
The ability to query tables from various engines is a cornerstone of the lakehouse.
Trino (formerly PrestoSQL) has robust connectors for both Iceberg and Delta Lake.
-- Trino: Querying an Iceberg table
-- Catalog configuration (e.g., in etc/catalog/iceberg.properties)
-- connector.name=iceberg
-- iceberg.catalog.type=rest # or hive, glue, hadoop
SELECT * FROM iceberg.db.my_iceberg_table WHERE id = 1;
-- Trino: Querying a Delta Lake table (with UniForm, via Iceberg connector)
-- Assuming UniForm is enabled on the Delta table, Trino's Iceberg connector
-- can read it directly.
SELECT * FROM iceberg.db.my_delta_table WHERE id = 1;
-- Trino: Querying a Delta Lake table (via Delta Lake connector)
-- Catalog configuration (e.g., in etc/catalog/delta.properties)
-- connector.name=delta_lake
-- delta.catalog.type=glue # or hive, hadoop
SELECT * FROM delta.db.my_delta_table WHERE id = 1;
Spark is the native environment for both Iceberg and Delta Lake.
// Spark: Reading Iceberg
spark.read.format("iceberg").load("local.db.my_iceberg_table").show()
// Spark: Reading Delta Lake
spark.read.format("delta").load("/path/to/my_delta_table").show()
DuckDB, an in-process OLAP database, offers excellent local analytical capabilities.
# Python with DuckDB: Reading Iceberg
import duckdb
con = duckdb.connect(database=':memory:', read_only=False)
con.execute("INSTALL iceberg;")
con.execute("LOAD iceberg;")
# Point to the Iceberg table's metadata file or catalog
# For local Iceberg table created by Spark in /tmp/iceberg_warehouse/db/my_iceberg_table
con.execute("CREATE OR REPLACE TABLE my_iceberg_table AS SELECT * FROM iceberg_scan('/tmp/iceberg_warehouse/db/my_iceberg_table/metadata/v*.metadata.json');")
con.execute("SELECT * FROM my_iceberg_table;").fetchdf()
# Python with DuckDB: Reading Delta Lake (with UniForm, via Iceberg extension)
# Assuming UniForm is enabled on the Delta table
con.execute("INSTALL delta;") # Delta extension is also available
con.execute("LOAD delta;")
con.execute("CREATE OR REPLACE TABLE my_delta_table AS SELECT * FROM delta_scan('/path/to/my_delta_table');")
con.execute("SELECT * FROM my_delta_table;").fetchdf()
# If UniForm is enabled, you might be able to use iceberg_scan directly on the Delta path
# (This depends on the exact implementation of DuckDB's Iceberg connector and UniForm's metadata sync)
# con.execute("CREATE OR REPLACE TABLE my_delta_table_as_iceberg AS SELECT * FROM iceberg_scan('/path/to/my_delta_table');")
| Feature | Apache Iceberg v2 | Delta Lake 3.0 (UniForm) |
|---|---|---|
| Metadata Structure | Hierarchical (Catalog -> Metadata File -> Manifest List -> Manifest File -> Data Files) | Transaction Log (JSON files, Parquet checkpoints) |
| Row-Level Ops | CoW, MoR (Positional/Equality Delete Files) | Primarily CoW (rewrites files) |
| Partition Evolution | Native, seamless (add/drop fields, change transforms) | Requires table rewrite for significant changes |
| Schema Evolution | Add/drop columns, type changes | Add/drop columns, type changes |
| Interoperability | Open spec, multiple language implementations (Java, Python, Rust, C++) | Open spec, but native engine support often requires Delta Lake connector. UniForm bridges to Iceberg/Hudi. |
| Catalog Options | Hive Metastore, Nessie, AWS Glue, custom REST | Hive Metastore, AWS Glue, Databricks Unity Catalog |
| Open Source Governance | Apache Foundation | Linux Foundation, heavily influenced by Databricks |
| File Format | Parquet, ORC, Avro | Parquet |
| Use Case Strength | Complex data pipelines, evolving schemas/partitions, diverse engine ecosystem, MoR for frequent small updates | Databricks ecosystem, strong CoW performance tuning, UniForm for broader reach |
Small File Problem (Both Iceberg & Delta Lake): Frequent small appends or updates can lead to an explosion of small data files. This degrades query performance due to increased metadata processing and I/O overhead.
OPTIMIZE for Delta, or Iceberg's rewrite_data_files action) to merge small files into larger ones. Schedule these regularly.// Delta Lake compaction
spark.sql("OPTIMIZE my_delta_table ZORDER BY (event_time)")
// Iceberg compaction (Spark example)
import org.apache.iceberg.spark.actions.SparkActions
SparkActions.get(spark).rewriteDataFiles(spark.table("local.db.my_iceberg_table")).execute()
Metadata Bloat (Both Iceberg & Delta Lake): Over time, the transaction log (_delta_log) or Iceberg's metadata files can become very large, impacting commit times and query planning.
VACUUM with a retention period. For Iceberg, expire_snapshots action.// Delta Lake vacuum (retains 7 days by default, can be configured)
spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled = false") // Disable safety check for testing
spark.sql("VACUUM my_delta_table RETAIN 0 HOURS") // DANGER: Deletes files not referenced by current snapshot
spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled = true")
// Iceberg expire snapshots (retains last N snapshots or snapshots newer than a timestamp)
SparkActions.get(spark).expireSnapshots(spark.table("local.db.my_iceberg_table"))
.retainLast(10) // Retain last 10 snapshots
.execute()
Schema Evolution Conflicts: While both support schema evolution, unexpected data types or column renames can cause issues.
spark.sql.legacy.allowUntypedScalaUDF for Spark if encountering UDF issues with schema evolution. For Iceberg, ensure schema.auto.merge is not blindly enabled in production without careful consideration.Object Storage Rate Limiting/Throttling: High concurrency or frequent small operations can hit object storage API limits (e.g., S3 5000 PUT/GET requests per second per prefix).
Inconsistent Reads (Time Travel): If not explicitly using time travel, queries might pick up stale data if a concurrent write commits just after the query starts planning but before it reads data files.
-- Delta Lake time travel
SELECT * FROM my_delta_table VERSION AS OF 5;
SELECT * FROM my_delta_table TIMESTAMP AS OF '2026-01-01 12:00:00';
-- Iceberg time travel
SELECT * FROM local.db.my_iceberg_table FOR SYSTEM_VERSION AS OF 1234567890123456L; -- Snapshot ID
SELECT * FROM local.db.my_iceberg_table FOR SYSTEM_TIME AS OF '2026-01-01 12:00:00';
A data engineering team is managing a large Iceberg table with petabytes of historical data, partitioned by `days(event_time)`. Due to evolving query patterns, they decide to `ALTER TABLE ADD PARTITION FIELD hours(event_time)`. What is the most significant operational advantage of Iceberg's partition evolution in this scenario?
When should I choose Iceberg over Delta Lake, or vice-versa? Choose Iceberg if: you prioritize a truly open, vendor-agnostic ecosystem, require advanced partition evolution, or need efficient MoR for frequent row-level updates/deletes. Choose Delta Lake if: you are heavily invested in the Databricks ecosystem, prefer a simpler transaction log model, or benefit from Databricks-specific performance optimizations (e.g., Photon, Liquid Clustering). With UniForm, Delta Lake's interoperability gap with Iceberg is significantly reduced.
How does UniForm impact my decision between Iceberg and Delta Lake? UniForm makes Delta Lake tables readable by Iceberg-compatible engines. This significantly reduces the "vendor lock-in" concern for Delta Lake. If your primary concern was Iceberg's broader engine support, UniForm largely mitigates that. However, UniForm currently only provides read-only Iceberg compatibility; writes still require Delta Lake.
What are the performance implications of Iceberg's MoR vs. Delta Lake's CoW for row-level operations? Iceberg's MoR can offer better write performance for frequent, small updates/deletes by deferring the rewrite cost to read time. This is beneficial when write amplification is a concern. However, MoR can introduce read overhead as the engine must apply delete files. Delta Lake's CoW ensures reads are always clean but can lead to higher write amplification and I/O for fine-grained changes. The optimal choice depends on your read/write patterns and latency requirements.
Can I migrate an existing Hive table to Iceberg or Delta Lake?
Yes, both formats provide tools for migration. For Iceberg, you can use ALTER TABLE ... SET TBLPROPERTIES ('format'='iceberg') in Spark for unmanaged tables, or use the migrate action. For Delta Lake, CONVERT TO DELTA command can convert Parquet or Hive tables in place. Always test migrations thoroughly on a subset of data first.
What is the role of a catalog in these table formats? The catalog is crucial. It acts as the central registry for tables, storing the current metadata pointer (for Iceberg) or the base path (for Delta Lake). Without a catalog, engines cannot locate and interpret the table's state. Common catalogs include Hive Metastore, AWS Glue Data Catalog, Project Nessie (for Iceberg), and Databricks Unity Catalog (for Delta Lake).
Clean AI CLI logs, build cron expressions, decode JWTs, and calculate chmod permissions offline.

Columnar data formats benchmarked: disk compression with Parquet, in-memory zero-copy with Arrow, and vectorized query execution with DuckDB.
Read more
Slashing Kafka storage bills by 80% with object storage tiering: S3 and GCS segment offloading, fetch latency benchmarks, and disaster recovery architectures.
Read more
A battle-tested production guide to Google Cloud Platform. Learn why Cloud Run beats GKE for 90% of workloads, how to leverage BigQuery and Secret Manager, and the 5 hidden cost traps that drain cloud budgets.
Read more