Infino
talk to an engineer sign up

The 3× to 10× regression from running decode on tokio’s I/O pool

· engineering

Infino’s public API is synchronous, while its storage I/O is asynchronous. A query may look like one blocking function call to the caller, but underneath it drives object-store reads through tokio and CPU-heavy decode through a sync-to-async bridge.

The wrong pool

Cold vector search had been about 1.1 s. After decode moved onto the tokio runtime that issued the storage reads, the same work took 3.7 s to 11 s: a 3× to 10× regression.

The cause was scheduler starvation. Tokio tasks yield at await points; decode has none. Once a worker began decoding a fetched block, it held that thread until the CPU work finished. Other tasks assigned to the same worker could not issue their next object-store reads, so CPU work serialized the I/O it was supposed to consume.

Tokio for I/O, rayon for CPU

The bridge now hands decode to rayon and awaits the result through a oneshot channel. Tokio’s workers remain available to schedule reads, while rayon’s bounded pool handles the data-parallel work. No thread acts as both an async-runtime driver and a rayon worker.

spawn_blocking would move work off the async workers, but onto tokio’s separate blocking pool. Decode is short, bounded, and data-parallel, so rayon gives it the pool and work-stealing policy we want without creating a second async runtime.

TOKIO (I/O) async task read from object store RAYON (CPU) decode data-parallel work hand off oneshot the task awaits; I/O threads stay free
An async task hands CPU work to rayon and awaits a oneshot channel, so the tokio I/O threads never block on decode.

The range-fetch bridge has three caller contexts

Lazy object-store range reads can be requested from a multi-thread tokio runtime, a current-thread test runtime, or a plain rayon/CLI thread with no runtime. That Send + 'static bridge takes a different path in each case:

  • With an ambient multi-thread runtime, block_in_place tells the scheduler that this thread will block, allowing it to move other async work elsewhere.
  • With no ambient runtime, a current-thread runtime polls the storage futures inline. There is no worker-thread handoff to coordinate.
  • With an ambient current-thread runtime, block_in_place is illegal. The bridge runs the operation on a dedicated worker thread instead of blocking the only runtime thread.

Other sync-to-async entry points use a narrower contract. The main query path receives Infino’s shared multi-thread I/O runtime and always drives the future there; an ambient runtime only determines whether block_in_place is needed. The three-way bridge above is reserved for range-fetch futures that can safely move to a dedicated thread.

sync-to-async bridge multi-thread rt block_in_place warns the scheduler no runtime current-thread rt built on the fly current-thread rt worker thread no block_in_place
The range-fetch bridge picks one of three paths depending on which tokio runtime, if any, is already running.

The second regression

An earlier version of the generic no-runtime path had another avoidable handoff. It reused one multi-thread runtime pinned to a single worker. Reuse sounded cheaper than building a runtime per call, but every block_on poll had to coordinate with that worker. A current-thread runtime polls the same future inline.

The shared runtime added 6 to 17 percent to multi-term full-text queries at 10 million documents, with the penalty increasing as the query fanned out. Single-term queries were flat. Building a current-thread runtime for the no-runtime case removed that coordination cost.

The rule is now explicit: tokio schedules object-store I/O, rayon performs decode, and the bridge adapts to the caller without mixing those responsibilities.