Infino vs Iceberg / Hudi
Infino complements Iceberg and Hudi. The table managers remain responsible for schemas, snapshots, deletes, and the active file set; Infino reduces cost by adding an indexed read path for fast SQL, BM25, vector, and hybrid retrieval.
diff --stat iceberg hudi infino
Complementary layers
Iceberg and Hudi define the table. Infino can serve selective reads over the table without taking over its write model or catalog.
- The catalog remains the ACID authority for snapshots, schema, the active file set, and deletes.
- Infino follows that state and maintains the retrieval metadata needed for file pruning, BM25, and vector search.
- Spark, Trino, Flink, and other table-format clients continue to use their existing read and write paths.
cat SPLIT.md
Table state versus retrieval
Two metadata layers over the active Parquet file set.
- Iceberg metadata or the Hudi timeline identifies the active file set and the table-format state that must be applied to it.
- Infino derives a retrieval manifest from that state, adding scalar bounds, term summaries, vector centroids, and index offsets used to prune reads.
- A hydrated data file remains valid Parquet. Standard readers use the columns; Infino also uses the embedded index regions.
infino hydrate --from iceberg
Indexed Parquet in the active table
Infino reads an active data file, writes a valid Parquet replacement with embedded retrieval indexes, and registers the file change through the table format.
- The replacement preserves rows, partition values, and column statistics. Standard readers continue to see ordinary Parquet columns.
- Infino uses the embedded FTS and vector regions; standard readers ignore those regions and read the columns.
- The file replacement is committed through the table format so readers see a consistent active file set.
infino explain pruning
Why selective reads can use less compute
The table format selects the correct snapshot. Infino then uses retrieval metadata to reduce the files, byte ranges, rows, and columns that reach DataFusion.
- Scalar bounds, term summaries, and vector centroids in the retrieval manifest can eliminate files before they are opened.
- Within a surviving file, the FTS or vector index identifies candidate rows before the query decodes projected columns.
- Search results are DataFusion relations, so filters, joins, and aggregates run directly over the candidate set in one query plan.
Read compute carries the query cost for Iceberg and Hudi tables. Selective retrieval served by Infino can reduce that cost by 10× or more compared with general-purpose lakehouse compute or a separate search service. Model the read workload →
infino diff --features
Table management and retrieval
| capability | iceberg / hudi | infino |
|---|---|---|
| job | table state, snapshots, schema, and deletes | optional indexed read path for SQL, BM25, vector, and hybrid |
| authority | catalog decides which rows and files are live | follows the selected snapshot; catalog authority remains intact |
| files | Parquet data files plus delete files | hydrated Parquet with retrieval indexes inside |
| writers | Spark, Trino, Flink, the warehouse | hydration registers a data-preserving file replace |
| readers | any engine that speaks the table format | Infino can serve indexed reads; standard readers still read the columns |
| deletes | table-format update and delete semantics | applied from the selected snapshot before results are returned |
| schema | field-id evolution, partition specs | search config on those columns; schema follows the table |
| where it runs | your catalog and your bucket | alongside the catalog and active Parquet files |
elasticsearch / opensearch · qdrant / pinecone · postgres / clickhouse · snowflake / databricks · iceberg / hudi
cat MIRRORS.md
Keeping the two manifests current
The table format owns table state; Infino owns retrieval metadata. The integration keeps the two views aligned.
- Hydration coordinates the retrieval-manifest update with the table-format file replacement.
- When another writer appends, deletes, or rewrites data, Infino derives a new retrieval generation from the updated table state. Unchanged files reuse their existing retrieval metadata.
- External changes become visible after synchronization. Each query remains pinned to one Infino generation while it runs.
The same boundary applies to Delta Lake: table state remains in the log, while Infino maintains retrieval metadata over the active Parquet file set.
cat FAQ.md
Does Infino replace Iceberg or Hudi?
Iceberg and Hudi remain responsible for table state: catalogs, snapshots, schema and partition evolution, and the active data and delete files. Infino adds an optional indexed read path over that state.
How can Infino reduce the cost of reading an Iceberg or Hudi table?
Query cost comes from the compute engine reading the Iceberg or Hudi table. Infino can skip files with manifest summaries, use embedded FTS or vector indexes to identify candidate rows, and let DataFusion decode only the columns needed by the remaining plan. Selective retrieval benefits most; broad scans have less to prune.
How are they similar?
Both work with Parquet in your bucket. A hydrated file remains valid Parquet, so Spark, Trino, DuckDB, and other compatible readers continue reading its columns while Infino uses embedded retrieval indexes.
How are they different?
Iceberg and Hudi remain the authority for table state and decide which rows are live. Infino keeps retrieval metadata (scalar bounds, BM25 blooms, vector centroids, and index offsets) so it can prune and search the active files.
What is hydration?
Infino reads an existing Parquet data file, adds BM25 and vector index regions, and writes a new object that remains valid Parquet. The table format then replaces the original file with the hydrated one in one snapshot. Rows, partition values, and column statistics stay the same.
Does Infino make every read cheaper?
The advantage comes from pruning and indexed candidate generation. Selective retrieval benefits most; a broad scan or large aggregate that touches most rows may remain more efficient on Spark, Trino, or a warehouse engine.
What about Delta Lake?
The same boundary applies. Delta keeps table state; Infino can hydrate the Parquet and serve retrieval against the selected table version.
How do deletes work?
Iceberg or Hudi authors row-level deletes and table rewrites. Infino follows the selected table state and applies its update and delete semantics before returning results.
Go deeper: integration · vs snowflake / databricks · architecture