> ## Documentation Index
> Fetch the complete documentation index at: https://infino.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect & storage: memory, disk, S3, and Azure

> Run Infino against in-memory, local disk, or object storage backends like S3 and Azure Blob, picked at runtime by the connect URI scheme.

Where your data lives is chosen by the `connect` URI:

<CodeGroup>
  ```python Python icon="python" theme={null}
  db = infino.connect("memory://")              # in-process, ephemeral
  db = infino.connect("./data")                 # local disk (durable)
  db = infino.connect("s3://bucket/prefix")     # object storage (durable)
  ```

  ```typescript Node.js icon="node-js" theme={null}
  const db = connect("memory://");              // in-process, ephemeral
  const db = connect("./data");                 // local disk (durable)
  const db = connect("s3://bucket/prefix");     // object storage (durable)
  ```

  ```rust Rust icon="rust" theme={null}
  use infino::connect;

  let db = connect("memory://")?;               // in-process, ephemeral
  let db = connect("./data")?;                  // local disk (durable)
  let db = connect("s3://bucket/prefix")?;      // object storage (durable)
  ```
</CodeGroup>

Data is stored as standard Apache Parquet at that location, so anything that reads Parquet
can read it.

<Warning>
  `memory://` is in-process and ephemeral. `update` and `delete` require **durable**
  storage (a path or `s3://` URI).
</Warning>

## Object-storage configuration

Credentials go in `storage_options` (`storageOptions` in Node.js), keyed by the
standard [`object_store`](https://docs.rs/object_store) config strings — `aws_*`
for S3, `azure_*` for Azure Blob (the same names the AWS and Azure SDKs use).
Infino reads no credentials from the environment; omit them to use ambient cloud
identity (an IAM instance role or Azure managed identity).

<CodeGroup>
  ```python Python icon="python" theme={null}
  # S3
  db = infino.connect(
      "s3://bucket/prefix",
      storage_options={
          "aws_access_key_id": "...",
          "aws_secret_access_key": "...",
          "aws_region": "us-east-1",
      },
      cache_dir="/var/cache/infino",   # local disk cache for remote-backed tables
      cache_budget_bytes=2_000_000_000,
      validate=True,                   # probe the store now, not on first query
  )

  # Azure Blob
  db = infino.connect(
      "az://container/prefix",
      storage_options={
          "azure_storage_account_name": "...",
          "azure_storage_account_key": "...",
      },
  )
  ```

  ```typescript Node.js icon="node-js" theme={null}
  // S3
  const db = connect("s3://bucket/prefix", {
    storageOptions: {
      aws_access_key_id: "...",
      aws_secret_access_key: "...",
      aws_region: "us-east-1",
    },
    cacheDir: "/var/cache/infino",   // local disk cache for remote-backed tables
    cacheBudgetBytes: 2_000_000_000,
    validate: true,                  // probe the store now, not on first query
  });

  // Azure Blob
  const db = connect("az://container/prefix", {
    storageOptions: {
      azure_storage_account_name: "...",
      azure_storage_account_key: "...",
    },
  });
  ```

  ```rust Rust icon="rust" theme={null}
  use infino::{connect_with, ConnectOptions};

  // S3
  let db = connect_with(
      "s3://bucket/prefix",
      ConnectOptions::new()
          .with_storage_option("aws_access_key_id", "...")
          .with_storage_option("aws_secret_access_key", "...")
          .with_storage_option("aws_region", "us-east-1")
          .with_cache_dir("/var/cache/infino")     // local disk cache for remote-backed tables
          .with_cache_budget_bytes(2_000_000_000)
          .with_validate(true),                    // probe the store now, not on first query
  )?;

  // Azure Blob
  let db = connect_with(
      "az://container/prefix",
      ConnectOptions::new()
          .with_storage_option("azure_storage_account_name", "...")
          .with_storage_option("azure_storage_account_key", "..."),
  )?;
  ```
</CodeGroup>

Common credential keys:

| Backend | Keys                                                                                                                                                                    |
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| S3      | `aws_access_key_id`, `aws_secret_access_key`, `aws_region`, `aws_session_token`, `aws_endpoint`                                                                         |
| Azure   | `azure_storage_account_name`, `azure_storage_account_key`, `azure_storage_sas_key`, `azure_storage_client_id`, `azure_storage_client_secret`, `azure_storage_tenant_id` |

The full set is whatever `object_store` accepts for the backend; an unknown key
is rejected at `connect`. For an S3-compatible endpoint (MinIO / R2 / Ceph), set
`aws_endpoint` (add `aws_allow_http: "true"` for plain HTTP) alongside the
credentials.

<Note>
  Pass `validate` (`with_validate(true)` in Rust) to probe the backend during
  `connect`, so wrong credentials or an unreachable bucket fail there instead of
  on the first query. The local cache keeps hot superfiles on disk so warm queries
  don't re-fetch from object storage.
</Note>

## Connect options

| Option (Python / Node.js / Rust)                                                                         | Description                                                                              |
| -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `storage_options` / `storageOptions` / `with_storage_option(k, v)`                                       | cloud credentials & tuning, keyed by `object_store`'s `aws_*` / `azure_*` config strings |
| `validate` / `validate` / `with_validate(bool)`                                                          | probe the object store at `connect` to fail fast on bad credentials (default off)        |
| `cache_dir` / `cacheDir` / `with_cache_dir`                                                              | local disk-cache directory for remote-backed tables                                      |
| `cache_budget_bytes` / `cacheBudgetBytes` / `with_cache_budget_bytes`                                    | disk-cache budget, in bytes                                                              |
| `connection_memory_budget_bytes` / `connectionMemoryBudgetBytes` / `with_connection_memory_budget_bytes` | per-connection heap budget, in bytes (`0` / omitted = measure only)                      |
| `cold_fetch_mode` / `coldFetchMode` / `with_cold_fetch_mode`                                             | how cold-cache misses are serviced                                                       |

## Connection memory budget

`connection_memory_budget_bytes` caps the **RAM (heap)** a connection may hold while it
works, the transient memory used to ingest data and run queries over it (keyword, vector,
hybrid, or SQL). It is **per connection**: one ceiling shared by every table on the
connection and by all its work, tracked as a single running total. By default it is unset,
usage is measured but never enforced; pass a byte value to enforce it.

When a request would cross the limit, Infino tries to stay within budget before it fails:

* **SQL that can spill** (sorts, aggregations, joins) writes intermediate state to disk and
  keeps going, so the query still completes, just slower.
* **Work that can't spill** (a vector search's cold fetch, an ingest build) is refused
  before it allocates, returning a recoverable error rather than risking an out-of-memory
  crash.

<CodeGroup>
  ```python Python icon="python" theme={null}
  db = infino.connect("./data", connection_memory_budget_bytes=512_000_000)
  ```

  ```typescript Node.js icon="node-js" theme={null}
  const db = connect("./data", { connectionMemoryBudgetBytes: 512_000_000 });
  ```

  ```rust Rust icon="rust" theme={null}
  let db = connect_with(
      "./data",
      ConnectOptions::new().with_connection_memory_budget_bytes(512_000_000),
  )?;
  ```
</CodeGroup>

A refusal surfaces as each language's idiomatic recoverable error: `ConnectionMemoryBudgetError`
in Python, a thrown `Error` in Node, and `InfinoError::OverBudget` in Rust. Catch it, then
narrow the query, split the ingest, or raise the budget.

<Note>
  This budget is RAM only. It is separate from `cache_budget_bytes`, which bounds the local
  disk cache, and it does not cap memory-mapped reads (the OS reclaims those on its own).
  Give each independent workload its own connection so their budgets stay isolated rather
  than sharing one ceiling. Omit it (or pass `0`) to only measure usage, never enforce.
</Note>

## Limitations

* **`memory://` is ephemeral** and can't be mutated. `update`/`delete` need a path or `s3://` URI.
* **Object-storage reads are cached locally.** Cold queries fetch from the store; size the cache (`cache_budget_bytes`) for your working set.

## See also

* [Tables](/guides/tables)
* [Open format](/guides/parquet-interop)
* [FAQ](/faq)