Inside a Parquet superfile
An Infino superfile is both a Parquet table and a search artifact. A standard Parquet reader sees columns and rows. Infino reads those same bytes plus an inverted index for BM25 and a vector index for nearest-neighbor search.
There is no sidecar to coordinate. The data and both indexes become visible together when the file is committed.
Why use Parquet at all?
Parquet is built for column scans, not posting-list jumps or vector probes. We considered three places to put the search index:
sidecar parquet data ──sync──▶ search cluster / vector store two copies · two consistency models new format columns + search indexes in a search-first file one copy · dedicated readers index inside parquet row groups + search regions + parquet footer one copy · standard parquet columns
A sidecar gives each engine its preferred layout, but every write creates a synchronization boundary. A new format keeps one copy, but makes that engine responsible for every reader and connector the data needs elsewhere.
Embedding the index accepts Parquet’s access-pattern constraints in exchange for keeping one immutable file and its existing columnar contract. The rest of the article is how that trade works at the byte level, and where it still hurts.
The byte layout
Parquet row groups come first and the Parquet footer remains last. The search regions occupy the space between them:
byte 0
┌────────┬────────────────────┬────────────────┬──────────────┬────────┬────────┐
│ PAR1 │ parquet row groups │ full-text index│ vector index │ footer │ PAR1 │
└────────┴────────────────────┴────────────────┴──────────────┴────────┴────────┘
end of file
footer metadata:
full-text offset + length + column config
vector offset + length + column config The row groups are written by the normal Arrow Parquet writer. Their byte offsets never change. The footer is an ordinary Parquet footer with additional namespaced metadata that tells Infino where each optional search region begins and ends.
Build the body, then rewrite the footer
The index offsets are unknown until the row groups have been encoded, so the builder cannot provide them as ordinary writer properties up front.
The build runs in three steps:
- Write a complete Parquet file with the user columns and row groups.
- Remove the original footer and append the full-text and vector regions.
- Emit a new standard footer carrying the original row-group metadata plus the index offsets.
standard parquet
│
├── keep row-group bytes and metadata
├── append full-text region
├── append vector region
└── write footer with final offsets
result: one immutable file Appending the regions does not move the row groups, because they were already before the old footer. The replacement footer continues to point at the same column bytes.
Why a normal reader still works
A Parquet reader starts from the trailing PAR1, reads the footer, and follows the row-group offsets named there. The search regions sit outside those ranges. Metadata keys the reader does not recognize are ignored.
That is enough for a normal reader to project columns, apply predicates, and run SQL without knowing that BM25 or vector structures are present. Infino opens the same footer, recognizes its metadata namespace, and also addresses the embedded regions.
The compatibility guarantee covers reads. A generic writer emits the columns it understood and a new footer, so the search regions disappear. The result remains valid Parquet and requires reindexing before Infino can search it.
Parquet’s point-lookup tax
A Parquet reader decodes a page to retrieve one value. That is efficient for adjacent rows and wasteful when search needs to resolve one result by id.
Most columns use 1 MiB pages. On the corpus we measured, one page covered roughly 65,000 rows. The id column instead uses 8 KiB pages, roughly 512 rows, so resolving a shortlist does not decode a megabyte per hit.
Making every column use small pages was worse. On 320,000-row segments at k=10, full-row reads stayed flat while the id-plus-score path became about 8× slower. The reader had many more pages to plan over. Small pages belong on the point-lookup column, not everywhere.
Inside the full-text region
The full-text index contains a term dictionary, posting lists, document-length data, and per-block score bounds.
The dictionary is a finite-state transducer that maps each term to its posting list. Document ids are delta- and bit-packed in fixed-size blocks together with term frequencies. Each block records the maximum BM25 contribution any document in that block can make, allowing the query engine to skip blocks that cannot enter the current top-k.
Ranked BM25 and boolean token matching share those posting lists. Prefix lookup starts from the ordered term dictionary. Exact phrase matching first intersects the member tokens, then verifies the candidate against the stored string value; it does not require a second text copy.
Inside the vector region
The durable object-storage path uses OPANN with Sq16 data. Vectors are organized into contiguous regions so a query can select relevant clusters, fetch bounded ranges, score their quantized vectors, and rerank a shortlist without downloading the full column.
When a working set is pinned in RAM, Infino can use HNSW over the resident vectors instead. That is a serving choice above the durable file: the Parquet superfile remains the source of truth, and a cold worker can fall back to the object-storage search path.
This split follows the hardware. HNSW is effective when graph hops resolve in memory. Object storage favors a small number of planned, contiguous reads, which is what the clustered OPANN layout provides.
Checksums and range reads
Each index region is checksummed, so corruption is detected before a query trusts its contents. The Parquet footer locates the top-level full-text and vector regions; the headers and directories inside those regions locate the term dictionary, posting blocks, vector clusters, and rerank data.
An eager reader can materialize the complete file. An object-store reader instead opens a byte source and fetches the footer and region directories first. A BM25 query then reads the posting lists for its terms. A vector query reads the selected cluster ranges. The rest of the object stays in the bucket.
From one file to one table
A table is a manifest over many immutable superfiles. Before opening any file, the manifest uses scalar ranges, term-presence summaries, and vector routing data to reject files that cannot contribute to the query. The surviving files execute their local search, and the table layer merges the results.
The file and table layers therefore solve different problems: the superfile keeps data and indexes atomic; the manifest makes many superfiles behave as one snapshot.
The format is intentionally simple at its boundary: Parquet row groups, opaque search regions, and a standard footer describing both. Infino adds the search access paths Parquet lacks while keeping data and indexes in one immutable file.