Crosslink: Incentivized Feature Net

General
64

Hi @aquietinvestor,

Three quick follow-up questions before Round 2:

1. **ZingoDelegator version** — I was using `zingodelegator-0.0.1`, the only version available on the Play Store at the time. If this caused Orchard transactions to be unreadable by the scanner, that was a compatibility issue outside my control. Has this been confirmed with the Zingo team?

2. **Staking since block 131,000** — was this activity detected from my Mobile UFVK (ZingoDelegator) or my Desktop UFVK?

3. **Round 2 submission** — do I need to re-submit my UFVKs, or will my Round 1 submissions carry over automatically?

Thank you for your time and patience throughout this process.

-–

**Round 1 Participant Feedback**

I wanted to share a few observations from my Round 1 experience that may help future participants:

**1. Default miner_address** — the provided config template contained `tmRB9…` (the faucet address). The setup guide did not clearly state this needed to be replaced. A simple warning in the instructions would prevent others from making the same mistake.

**2. BFT finality and fork visibility** — with BFT stalled network-wide, nodes silently ended up on different forks. A public status page showing live BFT health would help participants know whether their activity is on the canonical chain.

**3. Wallet version clarity** — `zingodelegator-0.0.1` was the only available version. Documenting which wallet versions are supported each round would remove unnecessary uncertainty.

**4. Sync speed** — syncing from genesis takes 4-6 days. The bottleneck is peers, not hardware. A published chain snapshot would significantly improve onboarding for new participants.

Overall the experience was valuable. The community, ctaz.cash, and forum support were excellent throughout.

3 Likes
65

Thanks for your question. I want to confirm a couple of things with the team before I respond. I’ll follow up with you shortly.

2 Likes
66

One thing to note is that all ZingoDelegator releases currently use the version number 0.0.1 and include a separate build number. The current release is 0.0.1 (build 37), as shown in the screenshot below. If you could let us know which build number you were using, that would be very helpful.

IMG_9368
IMG_9368772×1600 136 KB

This activity was detected from your mobile wallet.

Yes, you’ll need to re-submit your UFVKs for Round 2. Similar to Round 1, we’ll announce a claim period with a submission deadline. At that time, you’ll send us your UFVKs along with your mainnet Orchard address. If you stake using both the desktop and mobile wallets, you’ll need to submit the UFVK for each wallet.

Cipherscan and ctaz have developed tools for this. Please see the following links:

Thank you for the feedback! It’s helpful to hear your thoughts on where we can improve the user experience.

4 Likes
67

There won’t be a network reset, so if you stake now it will count toward Round 2. Also, if you’re successfully mining every block in succession, that’s a good indication you’re on a sidechain rather than the main chain.

We do plan to announce the canonical chain for reward purposes. We’ll announce a specific block height and hash that participants can use to verify they’re on the correct chain. We’ll share those details by the end of the week so everyone knows they’re staking on the chain we’ll be scanning for rewards.

1 Like
68

Hi @aquietinvestor,

I wanted to share a technical update on my node’s current state, as I believe it may be relevant to other Round 2 participants as well.

**Current node status:**

My node has been stuck at block 137,307 for an extended period. After investigating the logs in detail, I identified the root cause. The node is not stalled due to hardware, network, or configuration issues — it is being blocked by the Crosslink module itself.

**Log evidence:**

The logs show the following errors repeating continuously:

```

CrosslinkNotReady { block_height: Height(137308) }

err=“not ready to crosslink-finalize blocks”

BFT-Finality is falling behind the PoW chain. Current gap to tip is 137305 blocks.

Block does not link anywhere known — dropping: height 137309, 137310, 137311…

```

The node is successfully receiving new blocks from peers, but the Crosslink module is rejecting every commit attempt with `CrosslinkNotReady`. As a result, all incoming blocks are being dropped and the node cannot advance past block 137,307.

**What I confirmed:**

I also verified BFT peer connectivity directly:

```

70.34.201.146:12301 — Connection refused

70.34.209.22:12301 — Connection refused

70.34.195.191:12301 — Connection refused

70.34.209.18:12301 — Connection refused

```

All four BFT peers are refusing connections, which confirms the finality layer is stalled network-wide. This is consistent with what you mentioned previously about BFT finality being stalled and the mid-July forking tool being targeted as a fix.

**Additional context:**

I also noticed the logs show two wallet addresses at startup:

```

MINER WALLET T-ADDRESS: tmRB9AEVsxNAQsqtJPqJUje9KCaAijpS77z

USER WALLET T-ADDRESS: tmEFiMK73Jd6WrQ9nqzj5YpePnqWYob54DB

```

I have since confirmed that `tmEFiMK73Jd6WrQ9nqzj5YpePnqWYob54DB` is my node’s own wallet address derived from the local secret.seed, and `tmRB9…` is the faucet address as previously clarified by @samhsmith. My config now correctly reflects my own address.

**Questions for Round 2:**

1. Is the mid-July forking tool still on schedule? Once it is deployed and BFT finality is restored, will nodes that are currently stuck at 137,307 automatically resume syncing on the canonical chain?

2. For Round 2 reward eligibility, will the period during which nodes are stuck due to `CrosslinkNotReady` be taken into account? Participants cannot advance through no fault of their own.

3. Should participants take any action now to prepare for the forking tool, or simply keep the node running and wait?

Thank you for the continued support and transparency throughout this process.

2 Likes
69

I’ll answer what I can.

  1. Yes, we will be releasing it soon. But no, I don’t think your node would automatically resume, I think it is either on a side chain or stuck due to a bad block created by some bug probably. CrosslinkNotReady only means that the incoming fat pointers to bft certificates in the PoW blocks cannot be validate in some way. I checket the servers and they are still up. I think you may be trying to connect to those server ports with TCP and getting Connection Refused. This won’t tell you anything as what we run on 12301 is a UDP socket and UDP protocol.
  2. There may be discretionary rewards. But those are by no means guaranteed. The software is buggy, part of the value you are contributing as a tester is working through these issues to make progress. That said, I am very unhappy that your node got stuck and will be working to make the software better.
  3. No action is required right now. My goal is to have the forking tool release in a couple days but we will see.
2 Likes
70

@alphiz22 - Are you running headless?

2 Likes
71

Yes, I am running headless on a VPS (Ubuntu). To handle the X11/GUI dependency in the binary, I set up Xvfb (virtual framebuffer) with DISPLAY=:99 so zebrad can initialize its GUI components without a physical display.

The node is running stably via systemd, connected to 16+ peers, and the RPC is responsive at block 137,307. The BFT layer appears to be the blocker — all 4 BFT peer endpoints (70.34.x.x:12301) show connection refused on TCP, which you clarified is expected since it’s UDP.

Thank you for the clarification on the forking tool timeline. I’ll keep the node running and wait.

2 Likes
72

Update: For Round 2 rewards, we’ll use the following canonical block:

  • Height: 187,572

  • Hash: 02fc9aa7652b2a9a9f6196a446265fed6012227373ed34e0b17c7a3298db005e

Please verify that your node matches this block. You can do this in one of two ways:

  1. GUI: Navigate to block 187,572 and confirm that the block hash matches the hash above.

  2. RPC: Run:

    getblockhash 187572

    Your node should return:

    02fc9aa7652b2a9a9f6196a446265fed6012227373ed34e0b17c7a3298db005e

If your node returns a different hash, you’re on the wrong chain.

If you find that you’re on the wrong chain, you’ll need to resync your node from scratch using the following config option, which forces the node to sync the correct PoW chain up to the specified block hash:

[state]
...
network_checkpoint_block_hash = "02fc9aa7652b2a9a9f6196a446265fed6012227373ed34e0b17c7a3298db005e"

If you need help getting onto the correct chain or verifying your node, please ask us here or in the Signal Group and we’ll help you troubleshoot.

We expect this to be a temporary process. Once the user-led forking tool is available and BFT finality resumes, it will be clear which chain is canonical, and participants should no longer need to manually verify their chain using a block hash.

4 Likes
73

Incentivizing real participation on the feature net makes sense, since finality edge cases mostly show up under genuine load. What behavior are you most hoping to surface that a standard testnet has not, reorg recovery or the finality gadget under contention?

1 Like
74

It isn’t so much that an incentivized testnet surfaces behaviors that a standard testnet can’t. The main benefit is that it attracts a much larger group of users and keeps them engaged. Instead of just a handful of participants testing specific scenarios, you have people running nodes, reporting bugs, building tools, and sharing feedback. The incentives reward them for the time and effort they put in, and in return we get much more testing and feedback to help improve Crosslink.

4 Likes
75

Hi Jason,

I have a short security report for you (PDF attached).

Quick summary so you know what you’re looking at:

  • I reviewed the tenderlink BFT engine and the zebra-crosslink finality path on the latest main (commit 5e60f15).

  • Main finding (F-1, medium): the finality path panics on malformed input instead of rejecting it. A BFT proposal with an empty header list deserializes fine, then hits an unguarded .first().expect() in validate_bft_block, panicking nodes that validate it. This lines up with what I’ve actually seen on my node (TFL panic, finalized height stuck at 0, ~39 restarts).

  • Two lower/latent findings (F-2, F-3): a missing roster/quorum re-check (the // TODO before validate_signatures) and validate_signatures() accepting an empty signature set. I want to be upfront: tenderlink already verifies votes against the roster, so these are defense-in-depth gaps, not a remote forgery. I’m not overselling them.

Everything is code-review-verified; I haven’t run a live PoC yet, but I’m happy to build one for F-1 if useful. All details, exact file/line locations, and suggested fixes are in the PDF.

Let me know if you’d like anything clarified.

(attachments)

crosslink_security_report.pdf (7.54 KB)

2 Likes
76

We appreciate you going to this effort and for your interest!

  • Thanks for flagging the first().expect() pattern; we’ll mark it for replacement going into the future releases. For the final production code it is absolutely necessary to be safe against adversarial inputs, certainly. However, I would be cautious to conclude that the factual cause of your crashes in this case was a peer broadcasting an empty headers list during live BFT sync – because, from my reading, there is no first().expect() on line 1033 of lib.rs (see crosslink_monolith/zebra-crosslink/zebra-crosslink/src/lib.rs at 5e60f15d3f425bff89439c5987d7105bc08d6367 · ShieldedLabs/crosslink_monolith · GitHub ). It could have been a different lib.rs, of course (filename overloading in Rust is fun), but I can’t seem to find a first().expect() in other files at line 1033. Can you clarify what the report means when it cites the TFL panic at line 1033? Are you able to provide more information about how long the node was running before the crashes, stack traces, logs, etc.?

  • Thanks as well for flagging the TODOs and under-checked code; generally because we are in a work-in-progress prototyping/productionization codebase, we don’t tend to concern ourselves quite just yet with code that is currently safe and functional but may become unsafe or nonfunctional over the course of future code changes. We tend to make frequent sweeping edits that modify functionality, and want to be able to afford to make those changes, so we leave some latitude for ourselves. We’ll flag these as items on the to-do list, but they’ll come further along on the priority list after other pressing features.

Please let me know if I’ve misunderstood anything. :smile:

1 Like
77

Agreed that paying real participants surfaces finality edge cases a quiet testnet never would. In the early seasons, do you expect stake to cluster around a few well known finalizers before delegators learn to spread it, and does the Protocol Guardian design try to counter that?

For Non Technical people how can they best help? Is there any place for then in the testing process right now?

2 Likes
78

Hell yeah, get in here! :slight_smile:

2 Likes
79

DM your Signal ID and we’ll add you to the Crosslink Updates group.

80

I created an issue on the crosslink-monolith repo, but I’m guessing it might be something that might also need the Zingo team to help out with since its tied to the Zingo CLI. Just posting here to share.

2 Likes