agents / object-storage

Search the bucket

Loading object storage into both a search cluster and a warehouse creates three copies of the same rows. Infino searches the copy in your bucket.

infino explain superfile

The index lives inside the file

  • Both indexes, keyword and vector, live inside the Parquet file next to the columns they index.
  • The engine reads byte ranges from the bucket and caches the hot parts locally.
  • The same files remain standard Parquet.
  • There is no window where the search cluster and the warehouse return different answers for the same rows.
SEPARATE SYSTEMS WITH INFINO object storage · the rows pipeline · embed · load search cluster copy two warehouse copy three two systems to keep in step a join across them is an export your application · one SQL query infino engine keyword · vector · SQL, one pass byte-range reads object storage · the same rows superfiles: columns + both indexes one copy · nothing to reconcile

infino explain latency

Measured warm latency

Per-file summaries in the manifest prune files that cannot match before a byte is read, and workers cache the hot byte ranges on local disk. Over 10M documents, warm full-text p50 is 2 ms. Warm vector top-10 is 5 ms at p50 and 12 ms at p99.

How the engine is put together →  ·  Opening a file on S3 used to take two round trips →