How Fish Audio Keeps a GPU Busy: Continuous Batching for Streaming Speech

A voice API gets a short sentence and a long paragraph at the same time. They start at different moments, produce different amounts of audio, and finish at different moments. Something has to decide which of them the GPU works on next.

That something is the scheduler, and for a streaming product it's basically part of the user experience. It decides how long you wait before your audio starts and whether a long request can hold everyone else up.

Fish Audio published a concurrency sweep for their S2.1 model on an H200 (FP8) in their July 2026 engineering post:

That's roughly 52x the throughput for 36.6 ms more time to first byte. One caveat up front: this sweep increases concurrency, it doesn't isolate any single optimization. Their stack also has FP8 and custom kernels. But the reason a GPU can absorb 64x the work with barely any extra wait is mostly about how requests get batched.

01Why batching matters for generation

Generation has two phases. Prefill processes the input once. Decode then produces output one token at a time, running the model again for every token.

Decode at small batch sizes is usually limited by memory bandwidth, not compute. Most of the time goes into moving model weights around, not doing math with them. If you run one decode step for 64 requests instead of one, you move the weights once and use them 64 times. That's where the throughput comes from.

So there are really two separate problems:

1. Make each model execution fast (kernels, quantization)
2. Decide which requests go into the next execution (scheduling)

Faster kernels help the first. Only the scheduler helps the second. And spinning up a goroutine per HTTP request doesn't solve it either, something still has to combine those requests into shared model steps.

02The batch outlives its members

Take a decoder with room for two active requests, and three requests waiting:

• A: short sentence, 2 decode steps
• B: long paragraph, 6 decode steps
• C: another short sentence, 2 decode steps

Assume prefill is done and every step costs the same. These are step counts, not GPU timings.

Fixed batching starts A and B together. A finishes after step 2, but its slot stays empty until B finishes at step 6. C only starts at step 7 and finishes at step 8.

Continuous batching also starts A and B. When A finishes after step 2, C takes its slot at step 3. C finishes at step 4, and everything is done by step 6.

The only thing that changed is when the scheduler is allowed to make decisions. Fixed batching decides once per batch. Continuous batching decides after every step. The ORCA paper (OSDI 2022) called this iteration-level scheduling, and it's now the default in vLLM and SGLang.

Worth noting: in the fixed case the GPU isn't idle, B is still running. It's one slot that's wasted, and C sitting in a queue for no reason.

03Where speech makes it harder

Fish serves S2.1 on SGLang. Their public SGLang-Omni integration shows the pipeline has more stages than a text model: input preprocessing, an autoregressive model (a Slow AR predicts a semantic code, a smaller Fast AR fills in the remaining nine codebook tokens), and then a DAC vocoder that turns those codes into an actual waveform.

So the scheduler is juggling three kinds of work at once. New requests need prefill. Active requests need another decode step. Generated codes need vocoding before anyone can hear them.

Their March S2 technical report describes two things that help here: reusing the reference voice's prefix state with RadixAttention so the same voice prompt isn't processed again, and using NVIDIA MPS to run vocoder work alongside decoding. Those are from the earlier S2 release, separate from the July numbers above, but they show the shape of the problem. Batching handles who's in the decode step. Prefix caching cuts repeated prompt work. Vocoder scheduling gets the audio out.

04Building one

To actually see this work, I built a small inference server with a fake model inside: continuous-batching-lab. No GPU, no real model. Each decode step sleeps for a base cost plus a small per-request cost, so batching genuinely pays off the same way it does on hardware.

You can curl it and watch tokens stream back.

The structure:

client ── POST /generate ──> handler goroutine
                                 │ submit
                                 v
                    ┌── scheduler goroutine ──┐
                    │ waiting → ready → active │ ── one decode step for everyone
                    └─────────────────────────┘
                                 │ tokens
                                 v
              per-request bounded channel ──> SSE back to client

The key decision is that one goroutine owns all the batch state. HTTP handlers submit requests and read tokens from their own channel. No locks around the batch, because only one thing ever modifies it.

The scheduler loop is roughly:

for {
    drain new submissions into waiting
    drop cancelled requests, free their slots
    prefill some waiting requests, within a budget
    admit ready requests into active, if there's room
    run one decode step for all active requests
    push each token into that request's channel
    retire finished requests
}

Switching between fixed and continuous is a single condition: whether admission runs every iteration or only when the active set is empty. Same code, same model, same traffic.

A few things turned out to matter more than I expected.

Capacity isn't just a request count. A limit of 64 active requests says nothing about memory if some of them have long prompts and long outputs. The server checks both a slot limit and a KV budget, and reserves the worst case (prompt + max tokens) at admission, so anything that gets admitted is guaranteed to finish. SGLang's tuning docs describe the same tradeoff: admit too aggressively and you run out of KV cache and have to pull requests back out.

New work can't hog prefill. A request can't enter decode until its prompt is processed. If a burst of long prompts arrives and the scheduler prefills all of them at once, every active stream stalls. So prefill gets a per-iteration budget.

A slow client can't block the batch. Each request writes into a bounded channel with a non-blocking send. If a client stops reading, its channel fills up and only that request pauses. If it stays stalled, it gets cancelled. Everyone else keeps decoding. When a client disconnects, its context cancels and the scheduler frees the slot on the next loop.

05What I measured

The load generator sends a mix of short and long requests at increasing rates, against both policies, and records time to first token, gaps between tokens, and total completion time.

The A/B/C example above finishes at step 8 with fixed batching and step 6 with continuous batching.

Pending: time to first token (p50 and p99) versus request rate, throughput versus request rate for both policies, and analysis of the load-test results.

All of these timings come from the sleep-based cost model, so they show scheduling behavior, not GPU performance. Reproducing Fish's numbers would need the real model, real hardware and the vocoder stage.

06Throughput isn't the whole story

A server can push a lot of tokens per second while users still wait too long to hear the first word. For a voice service I'd want, from the client's point of view:

• queue wait before work starts
• time to first playable audio, p50 / p95 / p99
• gaps between audio chunks, p95 / p99
• total response time
• audio tokens per second
• real-time factor (generation time divided by audio duration)

First byte and first playable audio aren't the same thing either. A response can start before there's enough audio to actually play. And audio token rates depend on the codec, so they don't compare directly with text tokens from an LLM benchmark.

The way I'd pick a concurrency limit in production: the highest throughput that still meets your first-audio and streaming-gap targets, under realistic traffic. Medians won't tell you that, the p99 will.