Max, I owe you a correction, and it’s a good one.
Before starting on the layer I went to develop to see what it would sit on, and found the orphan buffer that came in with #7057. On 1.21.5-rc.3, which is what we pin, a Data frame can reach the far side before the Open that registers its stream. It goes into send_to_stream, misses the map, and an else branch drops it without logging. That is the failure I have been describing for three weeks: the stream opens, accept() fires, the first read never completes, and nothing anywhere errors.
So I measured it this afternoon. Same rig as every number I have posted here, two clients in one process, 64 byte echo, no gRPC. One build on develop at ece291d, one on the pinned release.
400 trials on develop: zero failures. Same afternoon, same rig on the pinned release: 21%, twice. Over the first 200 trials of a long run, and again over 80 trials on a fresh client started 14 seconds after that run was stopped.
develop counts the mechanism itself, which is what makes this more than two rates in two windows. It logs each rescued frame at trace, and in those 400 trials it logged 90, every one seq 0, in 90 distinct streams. 22.5%. It logged zero streams whose Open never arrived, so nothing was going missing. The payload was arriving first, and the pinned release discards what arrives before the stream exists.
The two curves track each other:
| reply surbs | frames rescued on develop |
failures on the pinned release |
|---|---|---|
| 1 | 38% | 40% |
| 20 | 24% | 25% |
| 100 | 27% | 20% |
| 400 | 1% | 0% |
Which points at the budget curve I posted on the 4th and could not account for. Every message carries the budget, and both go into the same lane in that order, so a bigger budget puts a bigger Open ahead of a bigger Data. That buys separation somewhere between enqueueing, transmission and reassembly, and at 400 it buys enough that the race is over before it starts. Which of the three does the work I can’t tell you from this run: there are no packet departure times in it.
Write-up, raw logs and the run that fell over halfway (something went wrong in that client and I can’t tell you what, kept because it is the kind of thing that gets mistaken for network weather): docs/measurements/2026-08-24-reordering-not-loss.md.
What this does to my last post:
- The ack I asked for loses most of its case. I proposed an
Opencarrying the first payload and an ack for those bytes, on the strength of a residue that turns out to be this race. In an afternoon where the race is handled there is no residue left to point at. I would rather withdraw the wire question than keep you talking about it. If a run without this race still shows something unexplained, I will bring it back. - The layer survives, smaller. Probing a stream and discarding it when nothing answers is the right shape for a per-stream race with independent retries, which is why it worked before I knew what it was compensating for. That part I would still like to write, and it needs nothing from the wire.
- The other two stand. The reconnection knobs are unreachable from the SDK on
developtoo, andsend_to_streamstill warns and skips ahead when the reorder buffer passes 256.
I spent three weeks pointing at your transport. In that run it delivered the first payload of 90 streams that would have died on the release I pin, and you had already fixed that ten days before I noticed.
LaDale, this may be what your 55-60% was. Worth a rerun on develop if you have an evening. Your gateway and your machine are the only way to get the number I cannot produce from here: whether the rate is the same somewhere that is not my box.