a) Lakehouse formats like Iceberg work well for analytical workloads where data is mostly appended and queries read large amounts of historical data. But there are so many usecases such as customer profiles and inventory and order status which receive millions of updates and deletes every day and at this rate Iceberg has to rewrite many data files or accumulate many delete files which is super expensive. Apache Paimon is a table format which tries to solve this problem.
b) Lets first look at LSM trees or Log Structured Merge Trees as they are the foundation of Paimon. Instead of modifying records in place an LSM tree buffers new records in memory and writes them into new sorted files so multiple versions of the same record can exist in different files. A reader can return the latest version or read previous available versions.
A background compaction periodically merges the files and removes obsolete versions.
c) Paimon brings this LSM architecture to lakehouse. A table is divided into partitions and buckets and each record within a partition is assigned to a bucket by hashing its primary key. All versions of same record hence land in same bucket.
Each bucket is its own LSM tree with files organized into levels and this allows frequent updates without rewriting the entire dataset.
d) Paimon provides Iceberg-like snapshots and manifests. Streaming engines like Flink continuously writes updates and commits a new snapshot at each checkpoint and analytical engines can query a consistent snapshot while writes continue. Paimon also produces a changelog so downstream streaming jobs can process incremental updates rather than repeatedly scanning the entire table.
e) Paimon supports three table modes that help choose between write throughput and read performance. Merge on Read is the default and keeps multiple versions in files and merges them during reads. Merge on Write uses deletion vectors to mark obsolete records so reads can skip them without merging. Copy on Write runs a full compaction on every commit so reads see fully merged files.
f) There are a few important tradeoffs. Paimon makes frequent updates cheaper but it moves that cost into background compaction which runs continuously and needs tuning to keep reads fast and data fresh. We also need to be careful about large bucket skew because a few oversized buckets can slow down both compaction and reads. Paimon also has a smaller ecosystem of query engines than Iceberg.