I read the OmniTable paper from Ant Group, which won the Best Industry Paper award at VLDB this year. The core architecture is built around a simple idea that I think is useful for large-scale LLM data curation: Logical Unification, Physical Separation.
Researchers usually think about adding a quality score, domain label, PPL score, dedup signal or another feature to a corpus. The platform underneath often sees many physical tables, joins and separate CPU/GPU pipelines. The paper mentions one case where adding a feature required coordinating work across 106 tables.
OmniTable changes the abstraction. A data entity becomes a row, and derived properties become columns in one logical wide table. That table does not physically exist as one table. A catalog maps the logical columns to a family of physical tables and can split, merge, index or materialize them without changing what the researcher sees. In one production web dataset, 800+ logical columns are spread across six physical tables.
Compute is also abstracted. Different feature operations can run on Spark, SQL or GPU inference. Compatible feature jobs can be fused so the same PB-scale input is not scanned repeatedly.
The production numbers are meaningful. The system manages more than 35 PB of training data and reduces one human-in-the-loop curation cycle from about 14 days to 2.5 days, a 5.6x speedup.
A few things I'd flag as tradeoffs.
The wide-table model gets harder for multi-modal AI data, in my view. A stable ID with many features works well for documents, but multimodal data, chunks, multiple tokenizations, embeddings and agent trajectories create more complex relationships.
Consistency becomes a catalog problem. Physical tables, indexes, ClickHouse copies and materialized views all sit behind one logical table, so freshness, lineage and metadata correctness now live or die with the catalog.
Physical separation hides complexity rather than removing it. The paper's own lessons-learned section admits this: they initially treated compaction and splitting as nice-to-have, and within three months small-file accumulation degraded query latency 3-5x and column growth hit engine limits. The abstraction only holds if the governance layer runs continuously.
The part I like most is still the change in abstraction.