Great, looking forward to seeing the final results.
I’m just now seen this let me wire up now.
1 Like
Hey @maxnym tried max/lwd-stream-patch @ 362e736 on our dialling half wait_established before our probe. Local matched-pin run: 35/35 established, 0 unestablished. Covers our need. This check out good @Joaco public serving half is still on the old pin without that handshake, so dialing him correctly timed out. Which was expected hopefully this all work ready to try Joaco when he ready.
Max, measured. Three arms on 693201bf, three trials each, all of them starting from an established stream that nobody writes to again.
| arm | what I did after the acknowledgement | reported unresponsive | time to report |
|---|---|---|---|
| both alive | nothing | 0 of 3, all survived 420 s | |
| peer gone | disconnected the accepting client | 3 of 3 | 252, 258, 260 s |
| stream gone | kept the client, dropped the accepted stream | 3 of 3 | 257, 258, 262 s |
Seven ping intervals of silence did not cost the live arm its stream in any of the three trials.
The third arm is the one I did not expect anything from. We have a decision written down here saying that a stream one side discards is invisible to the other until its idle reaper fires half an hour later, and both halves of this proxy discard streams on their failure path. Keepalive is not a close, but it puts minutes where there were thirty of them. I had that case filed as permanently unreadable and it is not.
On the number itself, three unanswered pings cost four intervals, since the first sweep after 60 seconds of silence sends a ping without counting a miss. So 240 s is the floor. Each of those waits ends on a cleanup tick rather than on the second, and the six failures landed at 252 to 262. I will budget above 280 s rather than against what I saw, since four ticks of slip fit under the same model.
One thing I could not see from outside: a ping. Neither ping nor pong is logged at any level, so the live arm measures an outcome and the two dying arms are what show the machinery running at all.
What I have not tested is where a false positive would cost the most: a dialler on this branch against a peer that has never heard of keepalive. Arming should make it impossible by construction, and I cannot check that from a rig that runs both ends in one process, so I am naming it rather than claiming it.
Write-up and raw logs: docs/measurements/2026-09-02-keepalive.md.
1 Like
All I exposed was MixnetStream::last_peer_activity() so that you wouldn’t have to parse logs for the pings/pongs yourself - so you have a situation in which:
- peer gone / stream gone: the clock freezes at the last pong, and
.elapsed()grows until the stream reports unresponsive - both alive: poll it,
.elapsed()stays under the ping interval the whole time, it keeps refreshing off the pongs
If you want to watch the countdown yourself for whatever reason / for the false-positive case you raised where a dialler against an old-pin peer should never arm (so it should never ping) I can
modifty the getter() though, if that’s helpful. Just let me know 
1 Like
Max, you were right and I had the tool the whole time. Polled every five seconds, it shows what you said it would.
| arm | trials | oldest the last inbound frame got | times that age fell back |
|---|---|---|---|
| alive | 2 | 71.4 s, 76.9 s | 6, 5 |
| stream gone | 1 | 265.1 s | 0 |
The getter reports that the peer was heard from, not what it said, so this rests on the setup: after the acknowledgement neither side writes and an acceptor never pings, which leaves a pong as the only frame that can arrive. The live peak sits a little above the 60 second interval, the interval plus the tick plus the trip out and back. The dying arm climbs to its failure without one refresh.
No change needed for the false-positive case either, as far as I can tell. Against a peer on an old pin the stream should never arm, so nothing arrives: the age would climb without a ceiling, nothing would fail, and the getter as it stands would show that. I am reading the code there, not a run.
LaDale, your timeout was the right outcome and not a fault at either end: the public serving half was on 1.21.5-rc.3, which has no OpenAck to answer your open with. It is on 693201bf now, same address, so it is worth another run.
One thing to watch, since you put wait_established in front of your probe. A timeout there means unknown, not dead, and the stream stays usable afterwards. Against anything not on the branch it will always run out, so a probe that treats it as a failure will refuse every peer that has not upgraded.
That upgrade also put a number on one direction of what I said I could not test. Our own dialling half is still on the release pin, and twelve trials of it against the upgraded serving half all answered on the first round, p50 1,418 ms. Twelve trials cannot tell a design property from a quiet afternoon, and the bench says so itself. But an old dialler talking to a new acceptor is what the code predicts: the OpenAck it cannot parse is dropped on the unknown discriminant, and only the dialling side pings.
1 Like
Ok got it, Joaco going to test this one out soon.
I tested this from the extension wallet stack of Nozy here the data I collected.
Independent dialling-half tryout of max/lwd-stream-patch, including a rerun against Joaco’s public serving half after 
#46.
Dialling pin: nymtech/nym rev 362e736 (branch tip; Joaco’s measured f5e46b7d plus review nits). Our dial path calls MixnetStream::wait_established before the LWMP probe. One long-lived Docker client; sequential gRPC via :9068 (verify_lwd), not TCP-only.
- Local matched-pin (2026-09-02) — server + client both on that rev. 35 dials:
connections_total Δ 35
connections_unestablished_total Δ 0
first_round_failures_total Δ 0
establishment_seconds_count Δ 35
Logs: mixnet stream establishment acknowledged, then probe answer. Upstream that day was not lightwalletd (app-layer 415 after establish) — scoring the mixnet path only.
- Joaco public half (2026-09-03) — serving pin
693201bf, same address / gateway6PkVkJ8nq882V1C95uCHUUoBWgo1ZzWHVpDtrhSjwDVn. Window ~22:27–22:29 UTC. Smoke + 35 sequentialGetLightdInfo:
connections_total 36 (1 smoke + 35 load)
connections_unestablished_total 0
first_round_failures_total 0
establishment_seconds_count 36
gRPC GetLightdInfo 35/35 OK
latency p50 ~3.2 s (min 2.4 / p95 6.2 / max 6.5)
Real LWD behind the pipe (Zebra:6.3.0, test). Same class of counters as our Aug 26 load (#35), where first-round/unestablished were 2.86% on 1.21.5 vs rc.3 here both are 0/36 with OpenAck on both halves.
Requirements (dialling half): the establishment handshake covers what we need to know the peer registered before committing request bytes. Keepalive / PeerUnresponsive mid-stream we have not measured yet (max/stream-keepalive). Happy for review/merge on the handshake path from our side; we can follow with a mid-stream kill harness when useful.
Operator measurement only, Nozy does not default the product to :9068.
@Lowo88 @Joaco are you both happy with the state of that branch then? If you let me know you don’t require further changes I can start getting this reviewed etc.
2 Likes
I’m happy with the results learn a lot and collected a lot of notes for my paper on this matter.
1 Like
Max, yes from me, and nothing further needed on my side.
The one case I said I was reading from the code rather than running, a dialler on this branch against a peer that has never heard of keepalive, I ran last night. Two images: an acceptor on the released =1.21.5-rc.3 that accepts streams and holds them without ever writing, and a dialler on 693201bf that opens one and watches.
| trials | establishment | outcome | oldest the last inbound frame got | times that age fell back |
|---|---|---|---|---|
| 3 | timed out, 3 of 3 | survived 420 s, 3 of 3 | 450.2 s | 0 |
Nothing arrived and nothing failed. A stream that had armed would have been pinged and failed about four and a half minutes after the last thing it heard, and these sat silent for 450 seconds. Three trials carry that because one survivor already contradicts the failure.
The same run puts a number on half of what I had only read in the source: against a peer with no acknowledgement to send, wait_established runs out, three times out of three, and the stream is still there 420 seconds later with nothing reported. I did not read or write after the timeout, so the usable part is still your documentation and not my run.
1 Like
Hmmm. Hard to add backwards compatibility here off the bat, since ‘this peer is dead’ and ‘this peer doesn’t speak in the connection handshake’ unless we also add a versioning response, but that’s (a) potentially identifying info, and (b) adds yet another handshake before you start a stream, which I’m a bit hesitant to do… Let me have a think. Glad to hear overall you’re happy though.
2 Likes
Max, I do not think we need it here. A timeout that cannot tell a dead peer from an old one costs us nothing: the layer treats it as unknown and falls back to its own deadline, which it needed anyway. The consumer it would hurt is the one that makes establishment a condition. That is what I warned LaDale about when he put the wait in front of his probe.
On the extra exchange, in case a number helps: the acknowledgement itself measured p50 1,551 ms and p90 1,932 ms over 500 trials, from the open going out to the ack coming back. What a second one would add on top of that I have not measured.
Thanks for how you have handled all of this.
3 Likes
Agree, it might be unnecessary. Glad to have been able to help
1 Like
Just a little update: that code has been merged @Joaco @Lowo88 - it will be going through QA, and I will let you know when we publish it to crates.io
2 Likes
Thanks for this update @maxnym
Thanks for the update, Max, and good to know it is in.
One reminder while you are in release mode: #7109 is still waiting on a review. It is the small one: gateway_reconnection_attempts and gateway_reconnection_backoff through DebugConfig, plus the wasm mirror. If it can ride the same release, all the better.
1 Like