Connect & storage: memory, disk, S3, and Azure

Where your data lives is chosen by the connect URI:

db = infino.connect("memory://")              # in-process, ephemeral
db = infino.connect("./data")                 # local disk (durable)
db = infino.connect("s3://bucket/prefix")     # object storage (durable)

const db = connect("memory://");              // in-process, ephemeral
const db = connect("./data");                 // local disk (durable)
const db = connect("s3://bucket/prefix");     // object storage (durable)

use infino::connect;

let db = connect("memory://")?;               // in-process, ephemeral
let db = connect("./data")?;                  // local disk (durable)
let db = connect("s3://bucket/prefix")?;      // object storage (durable)

Data is stored as standard Apache Parquet at that location, so anything that reads Parquet can read it.

memory:// is in-process and ephemeral. update and delete require durable storage (a path or s3:// URI).

Object-storage configuration

Credentials go in storage_options (storageOptions in Node.js), keyed by the standard object_store config strings — aws_* for S3, azure_* for Azure Blob (the same names the AWS and Azure SDKs use). Infino reads no credentials from the environment; omit them to use ambient cloud identity (an IAM instance role or Azure managed identity).

# S3
db = infino.connect(
    "s3://bucket/prefix",
    storage_options={
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
        "aws_region": "us-east-1",
    },
    cache_dir="/var/cache/infino",   # local disk cache for remote-backed tables
    cache_budget_bytes=2_000_000_000,
    validate=True,                   # probe the store now, not on first query
)

# Azure Blob
db = infino.connect(
    "az://container/prefix",
    storage_options={
        "azure_storage_account_name": "...",
        "azure_storage_account_key": "...",
    },
)

// S3
const db = connect("s3://bucket/prefix", {
  storageOptions: {
    aws_access_key_id: "...",
    aws_secret_access_key: "...",
    aws_region: "us-east-1",
  },
  cacheDir: "/var/cache/infino",   // local disk cache for remote-backed tables
  cacheBudgetBytes: 2_000_000_000,
  validate: true,                  // probe the store now, not on first query
});

// Azure Blob
const db = connect("az://container/prefix", {
  storageOptions: {
    azure_storage_account_name: "...",
    azure_storage_account_key: "...",
  },
});

use infino::{connect_with, ConnectOptions};

// S3
let db = connect_with(
    "s3://bucket/prefix",
    ConnectOptions::new()
        .with_storage_option("aws_access_key_id", "...")
        .with_storage_option("aws_secret_access_key", "...")
        .with_storage_option("aws_region", "us-east-1")
        .with_cache_dir("/var/cache/infino")     // local disk cache for remote-backed tables
        .with_cache_budget_bytes(2_000_000_000)
        .with_validate(true),                    // probe the store now, not on first query
)?;

// Azure Blob
let db = connect_with(
    "az://container/prefix",
    ConnectOptions::new()
        .with_storage_option("azure_storage_account_name", "...")
        .with_storage_option("azure_storage_account_key", "..."),
)?;

Common credential keys:

Backend	Keys
S3	`aws_access_key_id`, `aws_secret_access_key`, `aws_region`, `aws_session_token`, `aws_endpoint`
Azure	`azure_storage_account_name`, `azure_storage_account_key`, `azure_storage_sas_key`, `azure_storage_client_id`, `azure_storage_client_secret`, `azure_storage_tenant_id`

The full set is whatever object_store accepts for the backend; an unknown key is rejected at connect. For an S3-compatible endpoint (MinIO / R2 / Ceph), set aws_endpoint (add aws_allow_http: "true" for plain HTTP) alongside the credentials.

Pass validate (with_validate(true) in Rust) to probe the backend during connect, so wrong credentials or an unreachable bucket fail there instead of on the first query. The local cache keeps hot superfiles on disk so warm queries don’t re-fetch from object storage.

Connect options

Option (Python / Node.js / Rust)	Description
`storage_options` / `storageOptions` / `with_storage_option(k, v)`	cloud credentials & tuning, keyed by `object_store`’s `aws_` / `azure_` config strings
`validate` / `validate` / `with_validate(bool)`	probe the object store at `connect` to fail fast on bad credentials (default off)
`cache_dir` / `cacheDir` / `with_cache_dir`	local disk-cache directory for remote-backed tables
`cache_budget_bytes` / `cacheBudgetBytes` / `with_cache_budget_bytes`	disk-cache budget, in bytes
`connection_memory_budget_bytes` / `connectionMemoryBudgetBytes` / `with_connection_memory_budget_bytes`	per-connection heap budget, in bytes (`0` / omitted = measure only)
`cold_fetch_mode` / `coldFetchMode` / `with_cold_fetch_mode`	how cold-cache misses are serviced

Connection memory budget

connection_memory_budget_bytes caps the RAM (heap) a connection may hold while it works, the transient memory used to ingest data and run queries over it (keyword, vector, hybrid, or SQL). It is per connection: one ceiling shared by every table on the connection and by all its work, tracked as a single running total. By default it is unset, usage is measured but never enforced; pass a byte value to enforce it. When a request would cross the limit, Infino tries to stay within budget before it fails:

SQL that can spill (sorts, aggregations, joins) writes intermediate state to disk and keeps going, so the query still completes, just slower.
Work that can’t spill (a vector search’s cold fetch, an ingest build) is refused before it allocates, returning a recoverable error rather than risking an out-of-memory crash.

db = infino.connect("./data", connection_memory_budget_bytes=512_000_000)

const db = connect("./data", { connectionMemoryBudgetBytes: 512_000_000 });

let db = connect_with(
    "./data",
    ConnectOptions::new().with_connection_memory_budget_bytes(512_000_000),
)?;

A refusal surfaces as each language’s idiomatic recoverable error: ConnectionMemoryBudgetError in Python, a thrown Error in Node, and InfinoError::OverBudget in Rust. Catch it, then narrow the query, split the ingest, or raise the budget.

This budget is RAM only. It is separate from cache_budget_bytes, which bounds the local disk cache, and it does not cap memory-mapped reads (the OS reclaims those on its own). Give each independent workload its own connection so their budgets stay isolated rather than sharing one ceiling. Omit it (or pass 0) to only measure usage, never enforce.

Limitations

memory:// is ephemeral and can’t be mutated. update/delete need a path or s3:// URI.
Object-storage reads are cached locally. Cold queries fetch from the store; size the cache (cache_budget_bytes) for your working set.

Get started

Guides

Reference

Connect & storage: memory, disk, S3, and Azure

Object-storage configuration

Connect options

Connection memory budget

Limitations

See also

​Object-storage configuration

​Connect options

​Connection memory budget

​Limitations

​See also

Object-storage configuration

Connect options

Connection memory budget

Limitations

See also