OpenAI's September 11, 2026 Habitat write-up reports more than 70 million storage requests per second. That is storage-platform traffic, not model-inference traffic. OpenAI's write-up is worth reading for the scale alone.
But one incident is more interesting than the scale number. Overloaded processes kept attracting work even after the client that triggered the overload had stopped. The team traced the behavior to LIFO connection reuse. Switching to FIFO broke the feedback loop.
A connection pool sounds like a boring place to find a load-balancing problem. Its job is to keep connections around so the client does not have to open new ones constantly.
Except choosing an existing connection can also mean choosing where the next request goes.
01What exactly is the pool reusing?
An HTTP client can keep a connection open after a request finishes and use it again later. Node's HTTP agent, for example, maintains reusable sockets instead of establishing a fresh TCP connection for every request. Its documentation calls out both FIFO and LIFO scheduling options.
For this example, assume three backend processes serve the same logical service. The client has already established one connection to each. Each connection stays attached to its backend, and we allow one in-flight request per connection.
Client connection pool
connection A ──> backend A
connection B ──> backend B
connection C ──> backend CWhen a request borrows connection A, that connection is busy. After the response is consumed and the connection is reusable, it goes back into the idle pool.
“Idle” describes that particular connection. It does not tell us whether backend A is busy handling requests from a hundred other clients.
Now a new request arrives and several connections are idle. Which one should it use? That small decision is what we are investigating.
02LIFO and FIFO choose connections, not requests
LIFO means last in, first out: pick the connection most recently returned to the idle pool. FIFO means first in, first out: pick the connection that has been waiting there longest.
Suppose the idle pool looks like this:
Oldest return Newest return
A B CFIFO picks A. LIFO picks C.
The important detail is what “newest” means here. It does not mean the newest TCP connection, the fastest server, or the server with the most spare CPU. It means the connection that became available most recently.
We are also not rearranging a queue of incoming user requests. We are choosing a reusable connection for the next request that needs one. Those are different queues, and mixing them up makes the rest of the explanation confusing.
03A burst changes the order of the pool
Give the example some made-up timings. At time zero, the client sends three concurrent requests, one over each connection. A and B respond quickly; C takes longer.
Time Event Idle pool
0 ms Requests sent to A, B, C []
8 ms A finishes [A]
10 ms B finishes [A, B]
60 ms C finishes [A, B, C]These are illustrative timings, not OpenAI measurements.
At 61 milliseconds, another request arrives. Under LIFO, it takes C. Not because C is healthy or has capacity, but because its previous request finished last.
Now suppose the client sends a sequence of follow-up requests, waiting for each response before sending the next. There are no other borrowers from this particular pool during that sequence. C completes the first follow-up and returns to the end of the pool. The next request picks C again.
Before request: [A, B, C]
Borrow C: [A, B]
C finishes: [A, B, C]
Next request: borrow C againA and B remain available. This client simply is not choosing them.
The client has no instruction saying “prefer the slow server.” Its connection-selection rule produces that preference under this timing pattern. Completion order has become a routing preference.
04You can see the selection bias without a network
This small TypeScript trace models the post-burst behavior. Each iteration represents one request being borrowed, completed, and returned before the next request begins.
1type Backend = "A" | "B" | "C";
2type Policy = "lifo" | "fifo";
3
4function traceReuse(policy: Policy): Backend[] {
5 const idle: Backend[] = ["A", "B", "C"];
6 const selected: Backend[] = [];
7
8 for (let request = 0; request < 6; request++) {
9 const connection = policy === "lifo" ? idle.pop() : idle.shift();
10
11 if (connection === undefined) {
12 throw new Error("No idle connection available");
13 }
14
15 selected.push(connection);
16 idle.push(connection);
17 }
18
19 return selected;
20}
21
22console.log("LIFO:", traceReuse("lifo").join(" "));
23console.log("FIFO:", traceReuse("fifo").join(" "));The output is:
LIFO: C C C C C C
FIFO: A B C A B CWith FIFO, returning C puts it behind the connections that were already waiting. In this particular model, follow-up work rotates across A, B, and C.
This is a policy trace, not a performance benchmark. There are no sockets, server queues, or timeouts in the code. It demonstrates the selection rule; it does not measure latency or reproduce a production incident.
In a real pool, an efficient FIFO implementation normally uses a queue or deque rather than repeatedly shifting a JavaScript array. Current aiohttp connector source uses a deque with append and popleft to implement FIFO reuse.
05Why stopping the burst might not restore the system
The trace explains a preference. A self-sustaining overload needs another ingredient: enough clients directing work toward the same struggling part of the system.
Imagine the toy fleet has spare capacity overall, but many clients develop the same preference for C. Baseline traffic continues after the initial burst ends. If the share reaching C exceeds what C can process, removing the burst has not removed the overload.
There are two separate questions:
- What pushed the system into trouble?
- What is keeping it there?
They do not have to have the same answer.
This broader failure pattern is called a metastable failure: the system enters a degraded condition that helps sustain itself. Meta documented a related connection-pool problem in 2014, where traffic concentrated on congested network links. Its account is a useful companion to the OpenAI story, not evidence that the two deployments were identical.
C becomes overloaded
↓
C's responses finish later
↓
Its connections return later in a burst
↓
Clients preferentially reuse those connections
↓
Enough baseline traffic keeps C overloadedThe traffic did not have to remain unusually high across the whole fleet. It only had to remain too high for the part still receiving it.
This is why a test that merely adds a fixed delay to one server is incomplete. That can demonstrate preference for a slow connection. To demonstrate self-sustaining overload, model how server delay changes as work accumulates, and what happens after the original disturbance ends.
06FIFO is not a health check
FIFO rotates through eligible connections; it does not divide work into equal portions of server capacity. If the pool contains eight connections to C and one each to A and B, a complete rotation still sends eight of ten requests to C.
Neither policy can choose a busy connection in this model. When only one connection is idle, there is no meaningful choice. When none are idle, the client has to wait or establish another connection, subject to its limits.
FIFO also does not know that C is unhealthy. It will still choose C when C's connection reaches the front.
LIFO has legitimate benefits. Node documents that it can reduce the chance of choosing a connection closed by the server after a long idle period, and can keep the active socket set smaller. Its HTTP agent defaults to LIFO. See the scheduling option in Node's HTTP agent documentation.
The useful question is not “which data structure is good?” It is “what behavior does this policy encourage in this request pattern, with this routing topology?”
There is a version detail worth keeping straight. OpenAI's account describes LIFO behavior in its deployment. Current aiohttp source implements FIFO reuse. Do not turn a historical incident into advice claiming every current aiohttp installation has the same default: check the version you actually run.
07Where does the decision actually happen?
The model above assumes selecting a connection also selects a backend process. Do not assume that about every architecture.
A client connection can terminate at a proxy that makes a separate upstream routing decision. Envoy documents that a load balancer picks an endpoint when a request arrives, then the selected endpoint's connection pool supplies a connection. Its request lifecycle documentation describes those two steps.
Connection-affine model:
client ── existing connection ──> one backend
Request-routing proxy:
client ── existing connection ──> proxy
├──> backend A
├──> backend B
└──> backend CA load-aware policy can use a more relevant signal than connection recency. For equally weighted endpoints, Envoy's least-request policy samples two available hosts by default and picks the one with fewer active requests. That count is a routing signal, not a complete measurement of backend health. Envoy's load-balancer documentation describes the policy.
OpenAI says it now mostly relies on Istio and Envoy for pooling and more load-aware balancing.
HTTP/2 is another separate decision. It allows multiple requests to share a connection; it does not mean one connection to a particular backend automatically spreads those requests across other backends. Envoy supports multiplexing while maintaining upstream pools.
Before changing a client setting, find the layer where a request becomes attached to a particular process. That is where the architecture starts to matter more than the name of the pool algorithm.
08The experiment I would run next
The small trace is enough to explain the mechanism. It is not enough to choose a production setting.
For a real experiment, keep topology and workload identical and change only the reuse policy. Start with a baseline, inject a temporary burst or slowdown, remove that disturbance, and keep sending baseline traffic while watching recovery.
Record which backend each request actually reaches. A fleet-level throughput number cannot tell you whether one process received nearly all the work.
Also separate time spent waiting for a connection from time spent after acquiring one. Otherwise, a latency increase in the client can look like a slower backend even when the backend has not received that request yet.
For each backend, track incoming attempts, successful completions, in-flight work, and request latency. Make retries identifiable rather than quietly counting them as new user demand. For the pool, track idle and busy connections, connection creation, and reuse history.
Then vary the timing: bursty and continuous traffic, one client and many clients, a pool with spare idle connections and one that is continuously busy.
A result that disappears when you remove the pauses is still useful. It tells you the failure depends on the pauses. It does not mean the experiment failed.
Report the topology and workload beside any p99 chart. “FIFO improved p99” is not a complete result without explaining what was actually tested.
09The detail to take into a design review
A connection pool is not necessarily passive plumbing.
In the model we walked through, it makes a repeated choice about where work goes. That choice depends on when previous work finished. Under the wrong conditions, those two facts create a preference for the part of the system that needs less traffic, not more.
The practical lesson is not to replace every stack with a queue. It is to inspect the decisions hidden behind reuse.
When one process is struggling while its neighbors have room, do not stop at “we have a load balancer.” Follow an actual request. Find out who selected its destination, whether that decision was made for the request or an older connection, and why the same destination keeps being selected.
Sometimes the next useful thing to inspect is not a query plan or a CPU profile. It is which end of a queue your client takes a connection from.
10References
- OpenAI: Rapidly scaling online storage to serve over 1 billion ChatGPT users
- Node.js HTTP Agent documentation
- aiohttp connector source
- Meta: Solving the Mystery of Link Imbalance
- Envoy: Life of a Request
- Envoy: Supported load balancers
