Iceberg 1.12 introduces server-side scan planning work in Iceberg REST Catalog. I have mixed feeling not specifically about this change but moving more work towards a central control plans from engines.
1. What happens today
a. Spark, Trino, Flink, and other engines use Iceberg client library for Iceberg-specific scan planning.
b. The engine determines the SQL predicate and required columns, then passes them to Iceberg.
c. The Iceberg client library runs inside the engine process and reads the snapshot, manifest list, manifests, file statistics, and delete-file metadata to produce FileScanTasks.
d. The engine then converts those Iceberg tasks into Spark partitions, Trino splits, or the equivalent execution unit.
2. Server side planning
a. The engine can send the snapshot, projection, filter, and other scan parameters to the REST Catalog.
b. The catalog service performs the Iceberg scan planning and returns the resulting FileScanTasks.
c. The API also supports asynchronous planning, polling, fetching large sets of scan tasks incrementally, cancellation, and returning temporary storage credentials.
d. SQL optimization, join planning, split scheduling, Parquet execution, and other engine-specific work remain in Spark, Trino, or Flink.
3. Why this can be useful
a. A centralized service can may be reuse decoded manifests, snapshot metadata, statistics, and indexes across many queries and engines.
b. The REST service combines scan planning with authorization and credential vending, so clients can receive short-lived access only to the files required for a particular scan.
c. Newer clients in lets say python etc also benefit because they no longer need the full Iceberg manifest-planning implementation locally.
4. Tradeoffs in my mind
a. If we continue this trend clients ( e.g execute query) can now become concentrated in a shared service. Spark, Flink etc are heavily distributed systems and designed to scale.
These distributed engines are also being made performant via projects like Comet/DataFusion.
b. A failure or overload in the planning service could affect multiple engines at the same time.
c. Small or already-cached tables may be faster to plan locally.
d. Client and server versions must agree on expression semantics. Backward compatibility is important.
To me, the important change is not that Iceberg is taking planning away from Spark or Trino. Much of the Iceberg-specific logic was already in the Iceberg client library. I am more concerned about what else we continue to move from engines to centralized control plane and impact of that on capacity management and optimizations that have existed and matured in distributed engines.