Lwd-mixnet-proxy: light-wallet gRPC over the Nym mixnet, and what three days of measuring it found

Zcash Apps
1

hi zcashers,

i’ve published lwd-mixnet-proxy, two small rust programs that carry a light wallet’s grpc connection over the nym mixnet. the wallet doesn’t change, the server doesn’t change, and neither one knows the mixnet is there.

  wallet  --tcp-->  [lwd-mixnet-client]  --mixnet-->  [lwd-mixnet-server]  --tcp-->  light-client server

each half just moves bytes. neither parses grpc. a mixnet stream is an ordinary read/write stream, so the connection passes through as is. the wallet points at a local port. the serving half reads its upstream from config, so it works in front of any light-client server, not only lightwalletd-rs.

MIT, and setup is in the readme.

an address you can point at

i’m running the serving half on testnet:

HDfv77FgcWJTf4NFrxCvHRzZx4Zppc2pLiFUx6g2Lt3v.6VqPTWwZdmdFKEAyGW93hsPULp5XwEiVZh3GwHoXoqoK@3xLD3rpA5XWqdzfbTNs2E38tGp9K11QkU9RfQwxJGyC4

you still run the client half yourself, on your own machine. that’s the part that matters: your traffic enters the mixnet before it leaves your box, so the address above is the only piece i can host for you. testnet only for now. it’s an anonymous endpoint with nothing but a stream cap in front of it, so treat it as an experiment i might have to take down, not something to depend on.

one thing to know before you use it: it forwards to my public testnet lightwalletd-rs, the same one you can already reach directly. if you sync against that and submit through this, i see both sides and the mixnet bought you nothing. see below.

where it came from

while building lightwalletd-rs i kept running into the same thing. the calls that use the most bandwidth are the ones that leak the least. downloading blocks is the heavy part and tells the server almost nothing, since the wallet fetches everything and decrypts locally. the cheap calls are the ones that hurt. sending a transaction ties it to your ip. asking about a transparent address hands the address over.

that’s a good match for a mixnet, which delays and shuffles packets, so correlating by timing gets much harder. this has come up here before, most visibly in the 2024 nym grant thread.

so i put a mixnet listener inside lightwalletd-rs, measured it for three days, and then took it back out. the sdk pulls in 756 crates, more than the server itself, and there’s no way to trim it, so everyone building the server would carry it for the few operators who want it. reasoning in adr 0029. a separate process made more sense. this repo is that process.

what i found while measuring

this is the part i’d most like other people to check.

a stream can open, get accepted on the far side, and quietly lose its first payload. the sender’s write returns Ok. the receiver’s accept() fires and its read never returns. no error on either side, no timeout, both ends hang forever.

i took grpc and my own code out of the picture and reproduced it in about 200 lines using nothing but the sdk. the rate moved between 2% and 51% across sessions with the same settings, on two sdk versions. it isn’t steady either: in one 400-trial run, failures nearly tripled from the first half to the second. more reply blocks cut the failure rate (51% with one, 26% with four hundred) but cost 5.3x the latency. nothing i tried got to zero.

grpc libraries handle errors all day. they don’t handle silence. anything built here has to fix that first, and i suspect it’s why earlier attempts stopped at the demo stage.

the reproduction lives in the lightwalletd-rs repo, in contrib/nym, with the raw output of every run next to it. if you know the sdk better than i do, it’s yours.

what the proxy does about it

three things:

  • probe first. before the wallet sends anything, the client half runs its own small round trip on the fresh stream and drops the stream if nothing comes back in time. it’s the same round trip the transport loses, so the probe gets eaten instead of the wallet’s first request.
  • open three, keep the first that answers. you only learn about a silent failure when the deadline expires, so retrying one at a time pays that deadline every single time. measured side by side in the same window: p99 of 31.3 s opening one at a time, 6.3 s opening three.
  • a watchdog once bytes are moving. if the other end goes quiet, close the connection. the wallet gets a closed socket, and its grpc library already knows what to do with that.

in the run that settled the design, one afternoon on one network path, streams were failing 34.67% of the time. what a wallet would have seen: 0 failures in 300 connections, 1.2 s median to connect, 6.3 s at p99. every run is written up in docs/measurements/, including the two i had to throw out and why.

what it’s good for

good: sending transactions, transparent address queries, single transaction lookups. these leak a lot, cost few bytes, and latency barely matters when a block takes 75 seconds anyway.

bad: syncing. at this throughput, a full sync from an old birthday moves tens of gigabytes at seconds per round trip. that’s days. it’s not a drop-in replacement for a normal connection, and i’ve tried to keep the readme from suggesting it is.

one rule matters more than the software: don’t sync and submit through the same server. if an operator watches an address sync and then receives an anonymous transaction a minute later, timing joins the two. with few users there’s no crowd to hide in. sync through one instance, submit through another. it’s free.

before you leave it running

a connected mixnet client sends cover traffic the whole time it’s up, around 2 mbps. that’s what the traffic analysis resistance is made of, and it’s also a real bill on a metered connection.

both halves export two failure rates taken from the same connection attempts: what the transport does, and what a wallet sees. read both. the first swings by an order of magnitude between afternoons, so on its own it mostly tells you which afternoon it is.

where i’d like help

  1. has anyone kept a nym client up for days? mine logged Not enough bandwidth six minutes into a run, and no test of mine lasted longer than two hours. that’s thin evidence for something meant to run for weeks. the instance above is my attempt at an answer, and i’ll post what it does over the next few weeks, but one machine on one network path is still one data point.
  2. wallet and sdk devs: can a wallet point at two endpoints? the proxy moves bytes, so it can’t pick calls by method. sending only the sensitive ones over the mixnet means sync goes to one endpoint and submission to another. what does that cost on your side?
  3. anyone who knows nym well: does the silent stream failure ring a bell? a confirmation or a correction would both help.

this is early software. v0.1.0 is tagged, it hasn’t been audited, and the testnet instance above is the only one i know of running anywhere.

repo: GitHub - jpgonzalezra/lwd-mixnet-proxy: Carry a light-wallet gRPC connection over the Nym mixnet, transparently to both ends. · GitHub

if you try it, tell me what breaks. questions welcome.

5 Likes
Lowo Life as a ZEC Member
2

I like this project, I’m LaDale, I created Nozy wallet with it’s own smaller Nym wire for remote sendrawtransaction over the IP tx. Important work here you doing keep going @Joaco

Directly on your help asks:

  1. Long-lived Nym client we haven’t kept a smolmix session up for days either. Our runs are short evidence passes (IP relocate, RPC probe, sendraw). So I can’t contradict the “Not enough bandwidth” / <2h story; more multi-day operator data would help everyone.

  2. Two endpoints from a wallet yes, and it’s cheap asf if you treat it as policy, not as something the proxy has to parse. In Nozy we already keep local/LAN sync direct and only push remote sendrawtransaction through a mixnet helper when enabled. Pointing compact sync at one LWD and submit/lookups at another (or at your mixnet local port) is doable without teaching the tunnel about methods the wallet just has two base URLs. The hard part isn’t plumbing it’s making that split the default UX so people don’t “sync then submit a minute later” through the same hosted instance.

  3. Silent first-payload loss haven’t reproduced it ourselves yet, but your “probe gets eaten instead of the wallet’s first request” fix matches how we’d want any long-lived stream client to behave. Happy to try the testnet serving address from our side and report what we see (failure rates + whether a wallet multi-endpoint setup stays usable).

Glad you are working on this

1 Like
3

hey LaDale, thanks for this. good to know Nozy already runs sendrawtransaction over its own Nym wire; that’s exactly the neighbor data point i was hoping existed.

on your points:

  1. that matches my fear: everyone’s runs are short, so the <2h ceiling could be my client, my gateway, or the network, and nobody can tell yet. my serving half has been up since aug 13, but that’s days, not weeks, so too early to claim anything. i’ll report back when it has real mileage. if you ever leave a client running overnight, even one data point helps.

  2. agreed on all of it. the proxy is deliberately a byte pipe that never looks at gRPC, so the split has to live in the wallet as policy, and two base URLs is the right shape for it. and yes, the UX is the actual problem: if the user syncs direct and submits through the mixnet a minute later against the same operator, the timing alone can re-link them. making the split the default is wallet work too, and Nozy having already done it is encouraging.

  3. please do. the serving address is in the post, and the readme has the client-half setup (it runs in a container). what would help most from your side: your per-stream failure rate, whether you see the first-payload loss (the client logs every discarded probe), how long a session lives before bandwidth errors, and rough UTC timestamps so i can line your runs up with my server-side metrics. the rate here drifted between 2% and 51% over three days, so timestamps matter more than any absolute number.

if anything breaks, an issue on the repo is perfect. thanks again.

1 Like
4

Hello Joaco, will do.

I’ll spin up the client half against your testnet serving address and report back with:

  • per-stream failure rate
  • discarded-probe / first-payload loss (if we see it)
  • how long the session lasts before bandwidth errors
  • rough UTC start/stop so you can line it up with your metrics

we haven’t left a mixnet client overnight yet either, so if we get even one overnight run you’ll get that data point. anything that breaks to issue on the repo.

on the wallet side: yes, split stays policy in the wallet sync one endpoint, submit another. that’s how Nozy’s sendraw-over-Nym wire is shaped already; making that the default UX so people don’t re-link themselves on timing is the real work.

I’m setting up the client half against your testnet serving address and capture the metrics you asked for. lets see what we get out of this.

report back soon.

1 Like
5

thanks LaDale. before you start, here’s what the serving half has done since my last post, so you have something to compare your run against.

the bandwidth question has a partial answer, and it isn’t the one i expected. the serving half has been up 51 hours with no restart. it logged Not enough bandwidth 22 times in that window and recovered from every one of them by itself. both times it actually emptied, it emptied at exactly 00:00:00 UTC:

00:00:00.078 run out of bandwidth … remaining bandwidth: 0.00 B
00:00:00.112 Failed to send sphinx packet(s) (x11)
00:00:00.298 managed to claim testnet bandwidth

220 ms on aug 14, 279 ms on aug 15, around ten sphinx packets dropped inside each window. so the “under two hours” ceiling in my post was my laptop or my session, not a property of the network. a client that re-claims rides through it.

that’s the thing to watch in your overnight run, since it will cross 00:00 UTC. a dropped sphinx packet isn’t an error anywhere above it. it’s silence, the same shape as the failure the probe exists for. i’m not claiming it explains a rate that swings from 2% to 51% between afternoons: those windows are minutes long and this one is a quarter of a second. but it’s one confirmed way to produce the silence, and it happens on a schedule.

server-side counters, process started 13 aug 14:04 UTC:

  • 150 streams arrived from the mixnet. 67 of them, 45%, never delivered a handshake inside the 30 s deadline (lwd_mixnet_server_streams_rejected_total{reason=“unanswered”}).
  • of the 83 that did, 82 carried no request (lwd_mixnet_server_streams_without_request_total). a hedged dial drops two of every three streams by design, but 82 of 83 is well past that: most dials never got as far as a request.
  • 1 reached the upstream.

small sample, and most of it is my own smoke testing, so read 45% as an observation rather than a rate. what’s useful is that it’s the same failure seen from the receiving end.

two practical notes for your run:

  • the dialling half exits non-zero after five consecutive dials in which the sdk refused every open (adr 0010), so run it with restart: unless-stopped. an exit is a data point, write down the time.
  • 256 concurrent streams is where the serving half stops: past that it drops connections before reading the handshake. you’re nowhere near it, but an over-capacity drop looks exactly like an unanswered probe from your side. if you ever see a wall of them, tell me and i’ll read the rejected counter.

if you can send the gateway your client registered with and the sdk version alongside the timestamps, that helps a lot. both moved the rate when i measured, so two runs missing them aren’t comparable.

the full write-up of the midnight window, with the raw log, is in the repo: lwd-mixnet-proxy/docs/measurements/2026-08-15-daily-bandwidth-cliff.md at main · jpgonzalezra/lwd-mixnet-proxy · GitHub

i’ll post my side of whatever window you run.

1 Like
6

hello Joaco,

thanks for the midnight write-up; our overnight client hit the same cliff.

client half (still up, RestartCount 0):

  • up: 2026-08-15T16:07:56Z
  • still serving as of ~2026-08-16T16:16Z (~24h+)
  • nym-sdk = 1.21.5-rc.3 (image from your repo pin)
  • gateway registered: DuMkz6bVpKnZnbWf5DYtKHFhaLUUzB4vNSqdoiVR4j8X (node_id 2762, ws://147.79.68.95:9000/)
  • compose restart: unless-stopped no exit in this window

bandwidth / 00:00 UTC: at 2026-08-16T00:00:01Z we got a burst of Not enough bandwidth WARNs (14 in the log scrape), then managed to claim testnet bandwidth and kept going. no container restart. so same shape as your serving-half reclaim — a client that re-claims rides through it. that lines up with your revised read: the <2h ceiling wasn’t a hard network property.

short load window (for your server-side compare):

  • UTC: health 16:11:22Z, metrics snapshot 16:18:05Z on Aug 15
  • client metrics: first-round 5/30 = 16.67%, wallet-visible unestablished 2/30 = 6.67%, unanswered discards 17
  • bench 20 trials: wallet-visible 0%; first-round 60% (r1) vs 10% (r3)

overnight connection counters stayed flat (idle after that load) this useful mainly as the longevity / midnight reclaim data point.

happy to keep it up longer or file anything odd on the repo. your server-side 45% unanswered observation matches the silence/probe story from the other end. I’m check to see if it’s ok to push this to GitHub this good data we came up on so far.

7

LaDale, this is the run i couldn’t do myself. thanks for leaving it up.

your traffic reached me, and i can point at it. the server’s nym client logged 13 duplicate fragment received warnings in 96 hours. all 13 land between 16:11:29Z and 16:16:43Z on 15 aug, seven seconds after your health check and inside your load window. nothing else in the log is inbound traffic. my write-up calls that sample “mostly my own smoke testing”. that’s wrong and i’ll fix it: my testing was 13 aug, and the single stream that ever reached the upstream is dated 13 aug 15:03:11. a good chunk of those 150 streams and that 45% unanswered figure is your run, seen from the receiving end.

the midnight window isn’t my gateway. four events now, one a night, always in the first second of 00:00 UTC. yours on 16 aug lines up with mine: you logged 14 Not enough bandwidth warnings, my server logged 16 claim attempts and 15 refused sphinx packets, first at 00:00:00.232Z and back at 00:00:00.756Z. that’s 525 ms, the longest of the four, on the same night on both sides. (i quoted 00:00:00.078Z earlier, which is the 14 aug event, not yours.) the other three ran 220 ms, 279 ms and 202 ms with 11, 9 and 5 packets refused, so the size varies more than i thought. two clients, two gateways, two machines, same second: testnet accounting, not my gateway. i’ll update the doc and credit the run.

the two hour ceiling is gone. your 24h with RestartCount 0, and my serving half is now at 96 hours on one process and one registration. my early runs ended because the laptop suspended. pmset showed it afterwards.

your numbers reconstruct exactly. 17 unanswered discards over 30 connections is 5 x 3 + 2 x 1. the client defaults to --probe-attempts 4 --probe-concurrency 3, so the ladder is a round of three then a round of one: five connections lost the whole first round, three recovered on the single retry, two didn’t. that only closes if the sdk opened every stream asked of it, so your client was healthy throughout. it’s the check that invalidated my own first run.

so the per-stream rate backs out three ways: 5/30 as p cubed gives 0.55, your bench measured 0.60 directly, r3 gives 0.46. that’s the worst afternoon on record. my range over three days was 2% to 51%, and you found 60% on a gateway that isn’t mine, so the drift isn’t my box either.

which makes your 2/30 the attempt budget rather than a defect. ADR 0005 puts the design on p^k, and at p=0.55 with four attempts it predicts 9.2% wallet-visible against your 6.67%. first time that model has held on someone else’s machine. four attempts is calibrated for p near 0.35, where it buys 1.5%; at 0.6 it buys 13%. i’d rather raise the default than pretend four covers a bad afternoon.

the question still standing: your overnight was idle and so was mine, so nobody has caught a dial inside the reclaim window. five to fifteen sphinx packets get refused per event, across 200 to 525 ms. a heartbeat through the client half, one connection a minute for ten minutes either side of midnight, would settle it. if streams sail straight through, that’s just as useful and i stop pointing at the cliff.

three asks. the raw /metrics text rather than the derived rates: if the ladder ran clean it reads rounds_total 35 and streams_opened_total 95. anything under 95 means the sdk was refusing opens, so part of that 16.67% is my counter rather than the network. second, what was behind your local port. my upstream_connections_total is still 1 and it’s from 13 aug, so none of your 30 connections sent a request that reached the node, which makes your wallet-visible number a measure of dialling rather than of a call completing. third, confirmation that the load window was 16:11 to 16:18. the client registered at 16:07:56 and i’ve never measured whether a fresh client is at its steady rate or still settling.

and yes, push it. a PR to docs/measurements/ would be ideal, same shape as the others: method, what happened, limitations, raw logs under raw/. it’d be the first entry there from a machine that isn’t mine.

2 Likes
8

Good day Joaco glad it lined up on your side. The PR is open now but your repo is not loading correct never seen this before.

1 Like
9

Wow! This is great feedback @Lowo88. It’s great you got this working, and I’ll try to see if @maxnym can step in to answer SDK specific questions. Two parts of feedback:

  1. Use the mixnet for sending but do not use the mixnet for sync’ing. But @maxnym and the rest of the Nym team are getting the decentralized “fast mode” in the SDK (see Recommended — per-app hybrid: dVPN for block sync + mixnet for broadcast — Zcash × Nym ) - would love to see if you can try that for sync.

  2. That packet loss rate is not nearly as good as it should be. We’ll look into it. Usually that means that there’s some poor quality mix nodes still in the network. I think a reasonable packet loss rate should be more 10% for Nym, although our internal measurements tend to have it more to 20%, @maxnym can explain.

Also, a bit worried on “a dropped sphinx packet isn’t an error anywhere above it” - a dropped sphinx packet should not return an “ack” (acknowledgement), but it should not be silent when it fails, and if you don’t get an acknowledgement within a certain window of time, it should be sent again. We’ll see if that can be documented.

One problem we have seen is that some gRPC endpoints will close before the Sphinx packet gets through the mixnet. You can estimate and tinker with Sphinx packet getting through the net (even in NymVPN GUI right now under “mixnet parameterization.”). Here is a good video walking through the GUI, and all these options are in CLI: https://www.youtube.com/watch?v=Nmai02ThyBE

4 Likes
10

harry, thanks for stepping in, and for offering to pull maxnym into it. credit where it’s due:
most of what’s in this thread now is LaDale’s run, on his machine and his gateway, which is the
only reason any of it counts for more than one data point.

LaDale, the PR landed on my end, whatever github was showing you. thanks for writing it up.

on sync, we landed in the same place. the original post says don’t sync over the mixnet, for a
bandwidth reason. your 10-20% gives it a second one.

the hybrid: happy to try it, with one note about where it has to live. lwd-mixnet-proxy is
deliberately a byte pipe. it never parses gRPC, so it can’t route by method, and there’s nothing
inside it to point at a different transport for sync. splitting sync from broadcast is wallet
policy, two base URLs, which is what LaDale said Nozy already does. from here i can measure a
sync over the dVPN path and put the throughput next to the mixnet numbers. is fast mode reachable
from nym-sdk yet, or NymVPN-only for now, and does it sit behind the same stream API?

worth adding from our side, on the unlinkability claim for that split: the two streams share no
session state, so broadcasts are unlinkable to the sync profile by default. that holds at the
transport. if both endpoints belong to the same lightwalletd operator, a sync and an anonymous
broadcast a minute later still line up on timing, and with a handful of users there’s no crowd to
hide in. it’s the one rule i put in the original post. not a criticism of the design, just the
part that ends up being wallet UX rather than transport.

on the loss rate, our numbers might narrow it for you, because they don’t fit bad mix nodes. from
the reproduction: reply-block budgets rotated one per trial inside a single process, so drift hits
every row equally. 100 trials each:

reply_surbs fragments/msg failures p50 implied per-fragment loss
1 1 51% 1,412 ms 51%
20 6 34% 1,863 ms 6.7%
100 29 35% 4,054 ms 1.5%
400 115 26% 7,529 ms 0.26%

the last column is what per-packet loss would have to be if a failure meant “at least one
fragment went missing”, independently. it moves two orders of magnitude across rows measured in
the same minutes, from the same client and gateway. independent loss can’t do that.

going the other way: at 10% per packet a 115-fragment message arrives with probability 0.9^115,
about five in a million. we measured 74% of them arriving. so something is retransmitting
fragments, and doing it well. the client’s own trace log shows it: in one 100-trial run at budget
100, 16 chunks arrived as duplicates because the ack got lost, 17 pending acks were already gone
when the client tried to remove them, and the retransmission machinery fired 21 times. that’s the
ack window you described, working. what i can’t account for is the residue that never recovers
and never errors.

my best guess is that retransmission draws on the same reply-block budget the application sized
for the reply, and stops when that empties. read off the curve, not confirmed. i can rule one
thing out for you: it isn’t the epoch boundary. i timed a sweep across a known one and the median
didn’t step.

one precision on the midnight line you quoted, so i’m not overclaiming. the sphinx failure does
get logged,
at ERROR, with the byte counts in it. what never happens is any of it reaching the caller.
write_all and flush return Ok, the far side’s accept() fires, its read never returns, and neither
end times out. so the one question i’d most like answered: is stream reassembly meant to give up
or surface an error when a fragment never arrives? if it did, gRPC would take care of the rest by
itself, and most of what my proxy does would stop being necessary. i’d be glad to delete that
code.

gRPC endpoints closing before the packet lands is worth ruling out, and i think we have. the
reproduction has no gRPC in it: two mixnet clients in one process, one echoing 64 bytes and one
dialling, about 200 lines, same behaviour on 1.21.4 and 1.21.5-rc.3. every deadline in it is mine
and deliberate. what dominates there is the same silent stall.

i’ll watch the parameterization video, thanks. what would help most is knowing which nym-sdk
fields correspond to those knobs, since that’s the surface i’m on: SURBs per request, ack
timeout, retransmission cap, per-layer delay.

last thing, and the reason i’m glad you turned up. your line about a dropped packet not being
silent is the behaviour i built on, and the residue is where it stops holding. i have the
write-up and the reproduction. happy to open it as an issue on nymtech/nym, or to send it
wherever suits you.

1 Like
11

Ok, Harry thanks for this feedback this exactly the calibration we need. On the sync vs send, I agree and Nozy Wallet is already on the hybrid. Nozy Wallet matches the Recommended — per-app hybrid: dVPN for block sync + mixnet for broadcast — Zcash × Nym

on the loss rate are useful to hear 10% target / ~20% internal. our bad-afternoon window was ~55–60% per-stream (bench r1), which is well above that. we’ll treat that as “report and wait for mix-node / SDK look” rather than as the steady-state number to design forever around.

Joaco’s probe+hedge is why wallet-visible stayed near 0% even when transport was ugly, but we’d rather the transport be healthier so defaults don’t need to be so aggressive.

on silent Sphinx drops, I noted. from the wallet/proxy side we only see “probe unanswered / hang until deadline.” if missing ACK should trigger retransmit instead of silence, documentation + SDK behavior there would help a lot for anyone building on streams. happy to file a minimal repro from Joaco’s contrib/nym or our client logs if that helps @maxnym.

12

thanks Joaco,

glad the PR landed on your end agreed on the split living in the wallet two base URLs. Nozy already keeps mixnet for submit and dVPN for compact sync when we’re not on local LWD; we won’t treat the proxy as a sync path.

if you measure dVPN throughput next to the mixnet numbers, we can put our July smoldvpn/zec.rocks probe beside it so Harry/Max have two machines. the SURB/reassembly question is above my pay grade

13

Hey, I’m back sorry this took so long to report back. I miss the window yesterday, but got it tonight so this what I got when. I ran the midnight heartbeat you asked for. one connect a minute, ten minutes either side of 00:00 UTC.

window: 2026-08-18T23:50:01Z → 2026-08-19T00:09:05Z (midnight 2026-08-19T00:00:00Z) client: started 2026-08-18T00:22:19Z, still serving, RestartCount 0, nym-sdk 1.21.5-rc.3, compose restart: unless-stopped behind :9068: still TCP connect/close only (no lightwalletd, no gRPC body). same as the Aug 15 load, so this is dialling, not a completed call.

local TCP: 20/20 ok, ~210 ms each, including the sample at 00:00:03Z. the local port did not drop during reclaim.

raw /metrics

before 2026-08-18T23:50:00Z:

lwd_mixnet_client_connections_total 0

lwd_mixnet_client_connections_unestablished_total 0

lwd_mixnet_client_first_round_failures_total 0

lwd_mixnet_client_rounds_total 0

lwd_mixnet_client_streams_opened_total 0

lwd_mixnet_client_establishment_seconds_count 0

after 2026-08-19T00:09:05Z:

lwd_mixnet_client_connections_total 20

lwd_mixnet_client_connections_unestablished_total 19

lwd_mixnet_client_first_round_failures_total 19

lwd_mixnet_client_rounds_total 38

lwd_mixnet_client_streams_opened_total 76

lwd_mixnet_client_streams_discarded_total{reason=“unanswered”} 76

lwd_mixnet_client_establishment_seconds_count 0

lwd_mixnet_client_connections_in_flight 1

health stayed serving both sides.

ladder reconstructs: 19 connections lost both rounds (3+1), 19×4 = 76 unanswered opens, 19×2 = 38 rounds. the 20th connect is the 00:09:05Z sample, still in_flight when we snapped AFTER. no stream answered in the window.

reclaim still fired, same shape:

2026-08-19T00:00:01.253Z WARN Not enough bandwidth

2026-08-19T00:00:01.360Z ERROR insufficient bandwidth … required: 2413B, available: 0B

2026-08-19T00:00:01.540Z INFO managed to claim testnet bandwidth

~287 ms this night. client did not restart.

read on the cliff: streams did not sail through. wallet-visible fail was 19/20. but unanswered started at 23:50Z, ten minutes before 00:00, and stayed unanswered after the claim. so this sample is not explained by the 200–525 ms accounting window alone. looks more like the serving half / path was silent for the whole ±10 min, with reclaim happening on schedule in the middle.

happy to leave it up for another night if you want a second window, or to put a real tiny call behind the port next time so upstream_connections can move. your server-side counters for 23:50–00:10 UTC on 18/19 Aug are the other half of this.

— LaDale / Nozy

14

LaDale, your window measured my outage, not the mixnet. sorry.

my serving half stopped talking to the network on 18 aug at 14:12:48Z, nine hours before you started, and was still down through your whole heartbeat. the gateway connection timed out, ten reconnects failed, and the sdk called it:

Failed to send sphinx packet to the gateway 20 times in a row - assuming the gateway is dead

every start after that waited 4200 s on the same gateway, timed out, exited, restarted. twenty restarts, twenty-three hours. the cycle covering your window began at 23:34:31Z and was still waiting at 00:44:42Z, so your 76 opens went to an address with no client behind it.

my gateway (node_id 3184) is still bonded but gone from the api’s described set and from the active entry list, and its ws and mix ports don’t answer. yours, 2762, is right there in the same list. the twenty-three hours are on me: i set with_wait_for_gateway(true) so a brief unbond delays startup instead of breaking it, and for a gateway that never comes back it turns into a silent loop, nothing louder than a 503 on /health nobody was reading.

new address. re-registering keeps the key material and changes only the gateway component:

HDfv77FgcWJTf4NFrxCvHRzZx4Zppc2pLiFUx6g2Lt3v.6VqPTWwZdmdFKEAyGW93hsPULp5XwEiVZh3GwHoXoqoK@6PkVkJ8nq882V1C95uCHUUoBWgo1ZzWHVpDtrhSjwDVn

node_id 2719, performance 0.98. checked end to end an hour ago, two GetLightdInfo calls at 9.3 s and 8.2 s. the old address is dead.

the server-side counters you asked for don’t exist: in memory, wiped by every restart, zero since 14:12 on 18 aug. in a crash loop the log is the whole story.

your run still bought three things. a picture of what a dead serving half looks like from the dialling side: every open accepted by the sdk, none answered, indistinguishable from a bad afternoon on the transport. i reproduced your shape against the dead address this morning, 6 connections giving 24 opens and 24 unanswered discards. the ladder reconstructing a third time, 19 connections at 4 attempts giving 76 opens with establishment_seconds_count 0, which says your client was healthy. and a gap in mine: it sat at “serving” through 19 connections that got nothing back, and would have sat there for a week. adr 0010 only reacts when the sdk refuses opens, so i’ll add a degraded state.

the midnight question needs a rerun. (my fifth event, 18 aug: 413 ms, 7 packets refused, same shape as the other four.) same heartbeat against the new address if you’re up for it, with a real call behind the port this time. GetLightdInfo is one round trip and it moves upstream_connections_total on my end, so both sides line up. i’ll watch mine live.

@harryhalpin, @maxnym, one for the report, smaller than i first thought. i pointed a client at the dead address this morning on purpose. the sdk knows what is wrong and logs it at WARN, 56 times in three minutes:

failed to send a repliable message - Failed to prepare packets - no node with identity 3xLD3rp... is known. 0 reply surbs will be returned

that never reaches the caller. the stream opens, the send is dropped, and everything above gets silence until its own deadline. so this is not the fragment case where nothing knows: the diagnosis is already there, one layer under the stream api. if it came back on the stream i would have known in a minute instead of guessing all night.

LaDale, that means your logs from that night should carry the same line naming my gateway, several per connection, if you still have them. that would confirm my side from yours.

1 Like
15

hey Joaco, that lines up and thanks for catching it.

our logs from that night have the same WARN you reproduced. in the heartbeat window 2026-08-18T23:50Z–2026-08-19T00:10Z the client logged 154 of:

failed to send a repliable message - Failed to prepare packets - no node with identity 3xLD3rp… is known. 0 reply surbs will be returned

first ones we scraped are 23:49:58Z, about a minute before the first connect. so yes: diagnosis was already in the SDK, never on the stream. /health stayed serving the whole time, which matches the gap you described.

we’ll treat 19/19 as dead serving half, not mixnet weather. midnight reclaim on our client still fired (~298 ms) that’s separate, idle cover traffic.

the real midnight run
the real midnight run1214×643 50.1 KB

The the data from the real midnight run I’m ready to try again when you are.

1 Like
16

LaDale, your 154 lines settle it. first one at 23:49:58Z, a minute before your first connect, naming my gateway. two machines, two gateways, same line.

one correction to the post above, and it is the interesting part: the gateway came back. node 3184 was absent from the described set and from the entry list when i wrote that. it is in both now, performance 1, about a day after it went. so my line about it not coming back was wrong. while you are staring at it, gone and gone for good look the same.

that also splits the silence in two. the run i quoted up there, and two more since:

  • 13:51, node absent from both lists: 56 WARNs naming it, the same line as yours.
  • 15:56 and 16:01, node back in both when i looked at 16:03: none at all. counters where i took them read the same, four opens per connection, nothing answered, establishment_seconds_count 0.

gateway out of the topology and the client knows. gateway alive with nobody home at that address and there is nothing to know. same counters either way, which is why “the transport is having a bad day” fits both and is wrong for both.

@harryhalpin, @maxnym, that narrows what i asked you for up there. only the first case is detectable at the sender, and it is the one already logged. the second is just what a stale address looks like from outside, so there is nothing for the sdk to report.

my own fix is smaller than the story: startup gives up after five minutes now instead of seventy. the restart loop is also the recovery path, since a start that lands after the gateway returns just connects. bounding it retries fourteen times as often. the old address stays dead either way. that half re-registered elsewhere.

and yes to another night. say when and i will watch my side live, against the address in my last post.

1 Like
17

Ok got it we can run one tonight let’s get this done asap.

18

Hi both. Great work in this thread - happy to answer the questions that still remain.

  • Firstly, I’d say that one architectural thing to think about is the websocket connection to the gateway itself. If you are running a single connection (aka a single client <> entry gateway session) overnight, this might be a problem; I’m not sure that the underlying WSS connection would be able to stay up that entire time, and instead switching to some kind of rotated client might be useful there.

  • From what you’ve built with the watchdog / probe, I would actually say this is not debugging code but some ‘reliability layer’ functionality that the SDK currently doesn’t have. Really nice to see - I’d be happy to look over a PR on abstracting this out as something you could run the SDK modules via, if that was something you’d like to do.

  • One point on SURBs, wrt the conversation above: a receiving client will retain 10 SURBs by default to use to ask the original sender for more, if its supply is running low. Just something to note.

  • Wrt the below:

the sdk knows what is wrong and logs it at WARN, 56 times in three minutes:

failed to send a repliable message - Failed to prepare packets - no node with identity 3xLD3rp... is known. 0 reply surbs will be returned

that never reaches the caller. the stream opens, the send is dropped, and everything above gets silence until its own deadline. so this is not the fragment case where nothing knows: the diagnosis is already there, one layer under the stream api. if it came back on the stream i would have known in a minute instead of guessing all night.

I can look at threading this up the stream abstraction from the core client to make this a proper connection error, and making this a proper error that can be caught in whatever logic you wrap the connection in, if that would help you guys out here?

Let me know if you have any other questions. happy to answer.

3 Likes
19

Hey Max, thanks for coming in on this, and for the offers. Answers in order.

The websocket

I don’t think the socket is what breaks here, though four days is not forever. What I have:

  • 13 to 18 Aug: five days, one process, one registration. It ended when the gateway left the topology, not on the socket.
  • Since 20 Aug 13:06 UTC: four days on one process, not a single reconnection attempt logged.
  • LaDale’s dialling half: 24 h, RestartCount 0.

What did break, on 20 Aug: the gateway reset the connection at 13:05:10. Ten reconnection attempts, five seconds apart. The hostname was unreachable and the fallback address refused. assuming the gateway is dead at 47 seconds, and the process exits. The node was accepting again about seventy seconds after it went, so five restarts later we were back on the same address at 13:06:23. Write-up and log: docs/measurements/2026-08-20-gateway-restart.md.

So that 47 second budget is what decided it. On 1.21.5-rc.3 there is no way to reach it from the SDK. Happy to send a PR that plumbs the two knobs through DebugConfig, defaults unchanged, if you want one.

On rotation, it depends what rotates. Those five restarts re-established the session on the same registration, and the address came back with it. A fresh identity is another thing: the gateway is the third component of the address, so a serving half that rotates publishes a new address and every wallet holding the old one loses it. That is why re-registering is a manual step here, and why 18 Aug went 23 hours waiting on a gateway rather than moving off it.

The question underneath: is there a supported way to keep one dialable address across a gateway change? The answer would change how we run this.

The reliability layer

Yes, I would like to write it.

What is generic in ours: probe a stream before handing it to the caller, run attempts in concurrent rounds instead of one at a time, discard a stream whose probe went unanswered rather than retrying on it, and count opened streams separately from established connections. The counting is the part I would keep. It is what told us our attempt budget was calibrated for the wrong rate.

The piece I would want to work out with you is what it asks of the wire: an Open that carries the first payload, and an ack that means those bytes arrived. The ack has to be for the payload, not for the open. What we measured is a delivered Open with the payload behind it gone: accept() fires and its first read never completes. An ack on the open alone would pass exactly the streams that are already dead. Without any of that the layer still works on an echo the caller writes at both ends. That is what we run today.

I will bring the wire format as an issue on nymtech/nym before any PR, unless you would rather it went somewhere else.

SURBs

I went back to the code and I had it wrong: it does not bite where I assumed. A stream attaches the budget twice before the far side answers, once on the Open and once on the first write, so the echo side holds 20 against a threshold of 10 and can reply. Where it does bind is a reply owed after a single message on the defaults: 10 attached against 10 held back, nothing spendable, and the receiver has to ask for more before it can answer at all. Are those two tens on purpose, or did they drift into each other?

My budget table has that case in it, the 1-SURB row, where two messages still leave 2 against the threshold. It does not explain the rest of the table. Sender tags are reused, and we measured the store climbing from 100 to 19,903 over 200 messages. The reserve stopped binding early, and the budget still moved the rate.

I will rerun it with the threshold at 0, and again with a fresh client per trial so nothing carries over, and post the numbers. If they say what I think they say, a docs patch for how the default and the threshold interact, since I read all of this out of the source. The repro is two hundred lines with no gRPC in it, and it is yours if you want to run it.

One more from the same corner. The reply store is dropped on every start: all five starts on 20 Aug logged the loaded data is inconsistent, and the first of them came straight after a shutdown that flushed and closed the pool cleanly. I can open an issue with that log if it is useful.

Threading the error up the stream

Yes, that would help, and thanks for offering. What we need is small: for no node with identity X is known, anything we can read at the caller. It is a property of the destination, not of one stream, so it does not have to arrive per stream to be worth acting on. An io::Error on the handle would be the nicest to consume. We would take the signal in any shape.

Worth keeping out of it: the other case from my post above, gateway alive and nobody home at that address. Nothing at the sender can know that one.

One neighbour while you are in there. send_to_stream warns and skips ahead when the reorder buffer passes 256. For a byte stream carrying HTTP/2 that is a hole punched in the middle of it. An error is more useful than a skip, and I am happy to write that one myself.

20

Max, I owe you a correction, and it’s a good one.

Before starting on the layer I went to develop to see what it would sit on, and found the orphan buffer that came in with #7057. On 1.21.5-rc.3, which is what we pin, a Data frame can reach the far side before the Open that registers its stream. It goes into send_to_stream, misses the map, and an else branch drops it without logging. That is the failure I have been describing for three weeks: the stream opens, accept() fires, the first read never completes, and nothing anywhere errors.

So I measured it this afternoon. Same rig as every number I have posted here, two clients in one process, 64 byte echo, no gRPC. One build on develop at ece291d, one on the pinned release.

400 trials on develop: zero failures. Same afternoon, same rig on the pinned release: 21%, twice. Over the first 200 trials of a long run, and again over 80 trials on a fresh client started 14 seconds after that run was stopped.

develop counts the mechanism itself, which is what makes this more than two rates in two windows. It logs each rescued frame at trace, and in those 400 trials it logged 90, every one seq 0, in 90 distinct streams. 22.5%. It logged zero streams whose Open never arrived, so nothing was going missing. The payload was arriving first, and the pinned release discards what arrives before the stream exists.

The two curves track each other:

reply surbs frames rescued on develop failures on the pinned release
1 38% 40%
20 24% 25%
100 27% 20%
400 1% 0%

Which points at the budget curve I posted on the 4th and could not account for. Every message carries the budget, and both go into the same lane in that order, so a bigger budget puts a bigger Open ahead of a bigger Data. That buys separation somewhere between enqueueing, transmission and reassembly, and at 400 it buys enough that the race is over before it starts. Which of the three does the work I can’t tell you from this run: there are no packet departure times in it.

Write-up, raw logs and the run that fell over halfway (something went wrong in that client and I can’t tell you what, kept because it is the kind of thing that gets mistaken for network weather): docs/measurements/2026-08-24-reordering-not-loss.md.

What this does to my last post:

  • The ack I asked for loses most of its case. I proposed an Open carrying the first payload and an ack for those bytes, on the strength of a residue that turns out to be this race. In an afternoon where the race is handled there is no residue left to point at. I would rather withdraw the wire question than keep you talking about it. If a run without this race still shows something unexplained, I will bring it back.
  • The layer survives, smaller. Probing a stream and discarding it when nothing answers is the right shape for a per-stream race with independent retries, which is why it worked before I knew what it was compensating for. That part I would still like to write, and it needs nothing from the wire.
  • The other two stand. The reconnection knobs are unreachable from the SDK on develop too, and send_to_stream still warns and skips ahead when the reorder buffer passes 256.

I spent three weeks pointing at your transport. In that run it delivered the first payload of 90 streams that would have died on the release I pin, and you had already fixed that ten days before I noticed.

LaDale, this may be what your 55-60% was. Worth a rerun on develop if you have an evening. Your gateway and your machine are the only way to get the number I cannot produce from here: whether the rate is the same somewhere that is not my box.

1 Like