---
title: "The 3× to 10× regression from running decode on tokio’s I/O pool"
description: "Cold vector search went from 1.1 s to 3.7-11 s when CPU-heavy decode ran on the tokio I/O pool. The fix was a strict tokio-for-I/O, rayon-for-CPU boundary."
canonical: https://infino.ai/blog/lessons-building-search-in-rust/
published: 2026-05-19
---

# The 3× to 10× regression from running decode on tokio’s I/O pool

May 19, 2026 · engineering

Infino’s public API is synchronous, while its storage I/O is asynchronous. A query may look like one blocking function call to the caller, but underneath it drives object-store reads through tokio and CPU-heavy decode through a sync-to-async bridge.

## The wrong pool

Cold vector search had been about **1.1 s**. After decode moved onto the tokio runtime that issued the storage reads, the same work took **3.7 s to 11 s**: a **3× to 10×** regression.

The cause was scheduler starvation. Tokio tasks yield at await points; decode has none. Once a worker began decoding a fetched block, it held that thread until the CPU work finished. Other tasks assigned to the same worker could not issue their next object-store reads, so CPU work serialized the I/O it was supposed to consume.

## Tokio for I/O, rayon for CPU

The bridge now hands decode to rayon and awaits the result through a oneshot channel. Tokio’s workers remain available to schedule reads, while rayon’s bounded pool handles the data-parallel work. No thread acts as both an async-runtime driver and a rayon worker.

`spawn_blocking` would move work off the async workers, but onto tokio’s separate blocking pool. Decode is short, bounded, and data-parallel, so rayon gives it the pool and work-stealing policy we want without creating a second async runtime.

*Figure: An async task on the tokio I/O pool hands CPU work to the rayon pool and awaits a oneshot channel for the result An async task hands CPU work to rayon and awaits a oneshot channel, so the tokio I/O threads never block on decode.*

## The range-fetch bridge has three caller contexts

Lazy object-store range reads can be requested from a multi-thread tokio runtime, a current-thread test runtime, or a plain rayon/CLI thread with no runtime. That `Send + 'static` bridge takes a different path in each case:

- With an ambient multi-thread runtime, `block_in_place` tells the scheduler that this thread will block, allowing it to move other async work elsewhere.
- With no ambient runtime, a current-thread runtime polls the storage futures inline. There is no worker-thread handoff to coordinate.
- With an ambient current-thread runtime, `block_in_place` is illegal. The bridge runs the operation on a dedicated worker thread instead of blocking the only runtime thread.

Other sync-to-async entry points use a narrower contract. The main query path receives Infino’s shared multi-thread I/O runtime and always drives the future there; an ambient runtime only determines whether `block_in_place` is needed. The three-way bridge above is reserved for range-fetch futures that can safely move to a dedicated thread.

*Figure: Three branches of the sync-to-async bridge depending on which tokio runtime is ambient The range-fetch bridge picks one of three paths depending on which tokio runtime, if any, is already running.*

## The second regression

An earlier version of the generic no-runtime path had another avoidable handoff. It reused one multi-thread runtime pinned to a single worker. Reuse sounded cheaper than building a runtime per call, but every `block_on` poll had to coordinate with that worker. A current-thread runtime polls the same future inline.

The shared runtime added **6 to 17 percent** to multi-term full-text queries at 10 million documents, with the penalty increasing as the query fanned out. Single-term queries were flat. Building a current-thread runtime for the no-runtime case removed that coordination cost.

The rule is now explicit: tokio schedules object-store I/O, rayon performs decode, and the bridge adapts to the caller without mixing those responsibilities.