I see a clear trend across open-source analytical stacks. They are trying to close the gap between the lakehouse and real-time analytical systems.
The lakehouse gives us open time-travel, snapshots and interoperability through formats like Iceberg. Real-time analytical systems give us low-latency queries and data that stays up to date as changes arrive.
1. Flink SQL with Paimon takes the streaming-compute approach. Flink continuously processes changelogs and maintains updated state over Paimon’s primary-key and LSM-style storage.
2. Doris combines direct lakehouse queries with local caching and materialized views. Changed partitions are quickly refreshed into materialized views.
3. StarRocks follows a similar direction with caching and materialized views. It goes a step further and its newer incremental materialized views can process only newly appended rows for supported queries instead of rebuilding an entire changed partition.
The architecture and workflow will start to to look different from the traditional model where lakehouse data is copied into a separate serving database through another pipeline.