I’ve been down with the flu, so keeping this one very short.
DataFusion 55 was released in late August, and it is a good example of how a query engine gets faster by doing less work.
One change is runtime pruning for TopK queries. If a query asks for the top 10 values, DataFusion can use the results it has already found to skip Parquet row groups that cannot possibly contain a better value. On one TPC-H TopK query this made execution 4.2x faster.
The same idea applies to how Parquet data is read. If a query only needs a few fields inside a large nested column, DataFusion can now avoid reading the fields it does not need. In one production Comet query, this reduced data read from 1.35 TB to 30.9 GB. It can also recognize when data is already organized by ranges such as date or time and avoid repartitioning it again.