Table formats and catalogs have worked around gaps in object store features for years with extra jobs and extra services and following capabilities in the object store would remove a lot of that work.
a) Object stores should support a multi-object atomic commit so that the data files and the manifests and the metadata pointer swap happen as one transaction. Then a failed commit would not leave orphan files behind for a cleanup job to find later.
b) An ordered and complete log of object creates and deletes for each bucket or prefix would do more for lakehouse operations than a faster LIST. Orphan cleanup and cache invalidation could then read only what changed instead of listing millions of objects on every run.
c) Lets look at the read path as this is where query engines send the most requests. A multi-range GET that returns all the column chunks of a Parquet file in one request would cut the request count and the first-byte latency on every query.
d) The object store could accept credentials scoped to the files in a specific table snapshot instead of a whole prefix. The audit logs could then record the table and the snapshot and not just the object key.
e) As far as I know cross-region replication on the major object stores is asynchronous per object and a replica can receive a new metadata file before the data files it points to. If replication copied the data files first and the metadata pointer last then every replica would always be a consistent snapshot of the table.
