A table is a schema plus an index spec: which columns are full-text- and
vector-indexed (see Indexing). Create it once, then append rows.
Create and append
import infino
import pyarrow as pa
# Durable storage (a path or s3:// URI), required if you'll update/delete later.
db = infino.connect("./data")
schema = pa.schema([
pa.field("doc_id", pa.large_utf8(), nullable=False),
pa.field("body", pa.large_utf8(), nullable=False),
pa.field("embedding", pa.list_(pa.float32(), 384), nullable=False),
])
docs = db.create_table(
"docs", schema,
infino.IndexSpec().fts("body").vector("embedding", 384, 64, "cosine"),
)
# One append is one atomic commit, durable when the call returns.
docs.append([
{"doc_id": "1", "body": "To cancel, open Settings then Billing.", "embedding": embed("...")},
])
import { connect, IndexSpec } from "@infino-ai/infino";
// Durable storage (a path or s3:// URI), required if you'll update/delete later.
const db = connect("./data");
const docs = db.createTable(
"docs",
{ doc_id: "large_utf8", body: "large_utf8", embedding: { vector: 384 } },
new IndexSpec().fts("body").vector("embedding", 384, 64, "cosine"),
);
// One append is one atomic commit, durable when the call returns.
docs.append([
{ doc_id: "1", body: "To cancel, open Settings then Billing.", embedding: embed("...") },
]);
use std::sync::Arc;
use arrow_schema::{DataType, Field, Schema};
use infino::{connect, IndexSpec, Metric};
// Durable storage (a path or s3:// URI), required if you'll update/delete later.
let db = connect("./data")?;
let item = Arc::new(Field::new("item", DataType::Float32, true));
let schema = Arc::new(Schema::new(vec![
Field::new("doc_id", DataType::LargeUtf8, false),
Field::new("body", DataType::LargeUtf8, false),
Field::new("embedding", DataType::FixedSizeList(item, 384), false),
]));
let docs = db.create_table(
"docs", schema,
IndexSpec::new().fts("body").vector("embedding", 384, 64, Metric::Cosine),
)?;
// One append is one atomic commit. Build `batch` as an Arrow RecordBatch
// matching the schema (see the Quickstart).
docs.append(&batch)?;
One append == one atomic commit, durable on return. Append in batches rather than
row-by-row for throughput.
Open an existing table
A table persists, so you create it once and open it on later runs with open_table.
docs = db.open_table("docs")
const docs = db.openTable("docs");
let docs = db.open_table("docs")?;
Update and delete
In Python and Node.js, update and delete match rows by a SQL predicate
string. In Rust, they take a DataFusion Expr (col(...).eq(lit(...))), where col
and lit come from the datafusion-expr crate (see the Quickstart for the
companion crates Rust needs). update replaces matched rows 1:1 with the new rows you
supply.
# Replace the row(s) matching the predicate (1:1 with the new rows):
docs.update("doc_id = '1'", [
{"doc_id": "1", "body": "Refunds go to the original payment method.", "embedding": embed("...")},
])
# Delete rows matching a predicate:
stats = docs.delete("doc_id = '1'")
print(stats.matched, stats.n_tombstoned)
// Replace the row(s) matching the predicate (1:1 with the new rows):
docs.update("doc_id = '1'", [
{ doc_id: "1", body: "Refunds go to the original payment method.", embedding: embed("...") },
]);
// Delete rows matching a predicate:
const stats = docs.delete("doc_id = '1'");
console.log(stats.matched, stats.nTombstoned);
use datafusion_expr::{col, lit}; // build predicates as DataFusion expressions
// Replace the row(s) matching the predicate (1:1 with the new rows in `batch`):
docs.update(col("doc_id").eq(lit("1")), &batch)?;
// Delete rows matching a predicate:
let stats = docs.delete(col("doc_id").eq(lit("1")))?;
println!("{} matched, {} tombstoned", stats.matched, stats.n_tombstoned);
update and delete require durable storage: a path or s3:// URI, not
memory://. update is 1:1, so the matched row count must equal the replacement count.
Compact, list, and drop
docs.optimize() # merge small / underfilled superfiles
db.list_tables() # ["docs"]
db.drop_table("docs", purge=True) # purge=True also deletes the table's storage
docs.optimize(); // merge small / underfilled superfiles
db.listTables(); // ["docs"]
db.dropTable("docs", true); // purge=true also deletes the table's storage
use infino::OptimizeOptions;
docs.optimize(&OptimizeOptions::default())?; // merge small / underfilled superfiles
db.list_tables()?; // Vec<String>
db.drop_table("docs", true)?; // purge = true also deletes storage
optimize accepts options (omit for engine defaults):
| Option (Python / Node.js / Rust) | Description |
|---|
max_memory_mb / maxMemoryMb | build-time memory budget, in MB |
min_fill_percent / minFillPercent | only compact superfiles below this fill percent (0 to 100) |
target_superfile_size_mb / targetSuperfileSizeMb | target merged-superfile size, in MB |
Reclaim storage (gc)
Compaction and interrupted writes can leave behind storage objects that the
table no longer references. gc deletes them and reports what it reclaimed. It
only removes objects older than a grace window (in seconds), so it never races a
concurrent reader or writer; requires durable storage.
report = docs.gc(3600.0) # delete orphans older than 1 hour
print(report.bytes_freed, report.objects_deleted)
const report = docs.gc(3600); // delete orphans older than 1 hour
console.log(report.bytesFreed, report.objectsDeleted);
use std::time::Duration;
let report = docs.gc(Duration::from_secs(3600))?; // delete orphans older than 1 hour
println!("{} {}", report.bytes_freed, report.objects_deleted);
optimize already runs a best-effort gc after compacting, so calling gc
directly is mainly for reclaiming on a schedule or after bulk deletes.
Limitations
- One writer per table at a time. Appends/updates/deletes are serialized through a
single atomic commit (concurrent writers from other processes retry).
update is 1:1. The matched row count must equal the replacement-row count.update / delete need durable storage, not memory://.- Schema is fixed at creation. Adding or changing columns means a new table.
See also
