Max, answers to all three, and the SURB numbers I owed you.
The handshake
Yes, that is the whole request. We need to know that a stream we just opened has reached something alive, before the caller commits a request to it.
The cheap version is enough: an ack that says the peer’s SDK registered the stream. That already rules out the two failures that cost us the most, an address whose gateway is gone and a process that is not running. I would not ask for more than that.
The gateway that leaves mid-conversation
You are right, and it was never a request from me. Deadlines cover it on our side. I listed it in the design note so I was not overstating what your branch fixes.
The orphan buffer bounds
The warning is worth it for whoever is debugging, which is most of what this thread has been. But it is a log line, not something a consumer can act on, and I would not want it to be the trigger for anything. We give up on a stream through our own deadline, not by reading logs. If you want it actionable it has to arrive as an error, the way the reorder overflow does on your branch.
The retry I would not do either, and I would not want a consumer doing it blindly. Once a stream has lost frames the far side’s state is unknown, and resending a request that may already have been delivered is worse than failing cleanly.
The SURB thresholds I owed you since post 19
DEFAULT_NUMBER_OF_SURBS and minimum_reply_surb_storage_threshold are both 10, and the overlap costs nothing measurable.
I built the worst case I could: budget 1, so the receiver holds two reply blocks against a threshold of ten, and a fresh dialling client for every trial so nothing carries over and each exchange is a first exchange. 100 trials per arm, alternating blocks of 50 so that neither arm sits entirely in one window of the afternoon.
|
threshold 0 |
default 10 |
| failures |
30% |
33% |
| p50 |
4,188 ms |
4,662 ms |
| p90 |
8,305 ms |
8,750 ms |
A forced round trip before the first reply would cost roughly a second and a half at this budget, going by what a budget 1 exchange usually takes. The gap is 474 ms, and it does not even hold its sign: the default arm was 490 ms slower in the first pair of blocks and 380 ms faster in the second. Between its own two blocks the threshold 0 arm moved 850 ms and the default arm 1,720 ms.
So what I can say is that this run found no consistent difference, not that the collision is free. It is worth adding that in the default arm the replenishment definitely happens: two stored blocks never exceed a reserve of ten, so every first reply there goes through a request for more. Whatever that costs, it is smaller than an afternoon’s drift.
One thing fell out of it that I did not go looking for. A fresh client’s first exchange at budget 1 fails about 30% of the time in both arms, and 62 of those 63 failures are on the way back, with the reply already written by the far side. My guess is that the receiver is simply short of reply blocks: with two of them, or with ten requested and granted, there is very little left to retransmit with once the first attempt goes missing. An established client carries thousands and never notices.
I did look at other budgets, and they do not back me up. An earlier fresh-client run across budgets 1, 20, 100 and 400 failed 3, 3, 1 and 2 out of 20. If reply blocks were the whole story, 400 should have been far safer than 1 and it was not. Twenty trials a row is too few to conclude anything, but it is the evidence I have and it does not point where I was pointing.
The reconnection PR
Open as #7109, the one I offered in post 19: two settings that exist in the gateway client but cannot be reached from DebugConfig. Defaults unchanged.
It grew one thing while I wrote it. The settings have to be applied after the two construction paths in start_gateway_client converge, because a client upgraded from the registration handshake keeps the config it registered with. That path is what an ephemeral SDK client takes every time. Doing it there also fixes gateway_response_timeout, which was set on the other path only.
LaDale
Yes, it moved, by 35. Every one of them carried 202 bytes up and 259 back, so all 35 of your calls reached the upstream and came back. That figure is a count from the container log rather than a scrape of the counter, so it will not line up against your deltas the way your own three do. The interesting one is the last: it closed at 22:49:49Z, nearly two minutes past the end of your window. That is almost certainly the transport error you counted, finishing here long after your client had given up on it.
The pin you asked about: the serving half is on 1.21.5-rc.3, not 1.21.5.
Before that comparison settles in the thread: src/mixnet/stream/ is identical between 1.21.5-rc.3 and 1.21.5, so neither your new pin nor my old one carries the orphan buffer. The 16.67% to 2.86% is real, but it cannot be that fix.
That rules out the buffer, not the pin. The two crates do differ in three files, all of them around nym-api URL handling, so the pin stays on your list alongside the gateway and the load, which went from TCP-only on 15 Aug to real gRPC now.
