Crosslink: Incentivized Feature Net

Heading to bed, we have 17 finalizers and counting! Try the mobile app, super fun. :student:

8 Likes


I just started mining from my desktop computer. However, when I close the mining file, the wallet resets and each time it operates through a new account.

Since the videos are in English, I couldn’t fully grasp the details. I plan to learn the topic better and create educational content in Turkish.

Is there a different setting I can use?

2 Likes

(h/t @zk_nd3r & @Kenbak )

:right_arrow_curving_down:


:right_arrow_curving_down:

6 Likes

This is cool. I’m still on the fence about Crosslink, especially when people like Mert are publicly criticizing it. But I want to keep an open mind because it might be a necessary upgrade. Putting it on a testnet first is definitely the right move. Nice work!

2 Likes

It’s fine and helpful to criticize it but then I would like to see other propositions come forward to fix the probalistic finality dilemma with a mathematical or cryptographic solution. And denial is not an option unless of course if unstoppable private money comes with a lame disclaimer in fine prints: “subject to the will of POW miners not to reverse your transactions”.

4 Likes

I missed this workshop because I don’t remember attending it? :face_with_head_bandage:

2 Likes

Below are a few resources to get you started. Please note that the chain is currently halted, but should be back up within the next day or so. Participants will need to upgrade to V3 once it’s released.

Instructions

Quick Start Guide

Season 1 Workshop

5 Likes

Thanks, catching up and watching now.

3 Likes

Update: Crosslink Season 1, Version 3 of the testnet is now available. Prebuilt releases can be downloaded at the link below.

Important: Please follow the instructions carefully. If you participated in a previous version, you must copy your original seed phrase from the old folder into the new v3 folder before running the executable. This will ensure your cTAZ balance and any prior staking or delegation activity are restored.

6 Likes

The sound lol :sweat_smile:

1 Like

Update: Crosslink Season 1, Version 4 of the testnet is now available. This update should address some of the syncing issues users were experiencing.

Prebuilt releases can be downloaded at the link below.

Please note that if you were previously on V3, you can just run the executable. If you were on V1 or V2, you must copy your original seed phrase from the old folder into the new V4 folder before running the executable. This will ensure your cTAZ balance and any prior staking or delegation activity are restored.

5 Likes

windows version continue with mutch bugs is dificult operate staking system , balance missing , dont count faucet claims mutch times !

2 Likes

Sorry to hear you’re running into these issues. Can we connect so I can see if I can help? I’ll send you a DM now.

2 Likes

Thanks @aquietinvestor ! Appreciate the response

1 Like

Interested in joining! Can’t DM yet (new account). Would love to be added to the group via your DM.

1 Like

Update: One known property of BFT systems is that finality can stall if enough stake goes offline, which is what our testnet is currently experiencing. At the moment, it’s not yet clear whether the network will recover naturally as finalizers come back online, or whether participants will ultimately decide that some form of coordinated intervention is necessary.

Part of the purpose of the feature network is to learn how these situations play out in practice and to develop tools that help participants exercise their own agency rather than relying on Shielded Labs to dictate outcomes. Right now, we’re focused on building visualization tools that make it easier to determine which finalizers are online or offline, along with recovery tooling that could allow participants to coordinate an emergency network upgrade to socially slash stake delegated to inactive finalizers if the broader network decides that’s the appropriate path forward. Situations like this are one of the reasons the feature network exists in the first place.

We hope to have visualization tooling available later this week, while it will take another week or two to develop a tool that allows users to coordinate an emergency network upgrade. Thanks to everyone for their patience, and we appreciate everyone helping test these edge cases and participating in the process of understanding how the network responds under real-world conditions.

3 Likes

Crosslink Feature Net: Reflections on the First Six Weeks

It has been a little over six weeks since we launched the Crosslink incentivized feature net, where participants can join the network as miners or finalizers and stake to earn rewards. We wanted to provide an update on how the network has been progressing, the issues we’ve encountered so far, what we’ve learned from them, and what we are currently focused on improving.

As expected at this stage, the network is still immature and has been extremely rough around the edges. Participants have encountered crashes, syncing issues, stalled finality, and a variety of other bugs and edge cases across multiple iterations of the network, particularly around block production, syncing behavior, and network recovery. While that has understandably been frustrating at times for some participants, others have embraced the challenge by helping diagnose issues, coordinate recovery efforts, and better understand how the system behaves in practice. As a result, the feature net has already provided valuable insights into real-world network behavior, and overall we are encouraged by the progress being made.

Rather than repeatedly resetting the network every time a major issue occurs and only practicing under ideal conditions, our goal is to work through these problems in a live environment whenever possible. The kinds of failures, outages, and coordination challenges we are encountering are precisely the kinds of situations a production network must be able to survive. We believe working through them now will ultimately lead to a stronger network.

Stalled BFT

The most significant issue the network is currently facing is a stalled finality layer. Stalls are perhaps the most controversial aspect of BFT systems. Rather than risking transaction rollbacks, a protocol will halt finalization if it can no longer safely reach consensus, choosing safety over availability. This is the opposite side of the tradeoff made by the Nakamoto PoW that Zcash currently uses, which prioritizes availability and continued block production even under adverse conditions, but accepts the possibility of transaction rollbacks through chain reorganizations. Put simply, when network conditions become unfavorable, PoW tends to produce rollbacks while BFT systems tend to produce stalls.

In Crosslink, a BFT stall causes finality to stop progressing while PoW continues producing blocks. This can happen if enough stake goes offline. At this stage, it is still unclear whether the network will recover naturally as finalizers come back online or whether participants will ultimately decide that some form of coordinated recovery action is necessary.

As developers, it is often tempting to simply reset the network when it encounters a difficult state, much like rebooting a laptop when something goes wrong. However, because stalls are a known and expected behavior of BFT systems, and because the feature net is designed to expose real-world operational challenges, we believe there is greater value in understanding and working through these situations. Our focus is on improving the robustness of the software and providing better analytics, monitoring, and diagnostic tools. Ultimately, our goal is to give participants the ability to detect, analyze, coordinate, and recover from these kinds of scenarios themselves, in order to ensure that any future deployment is more resilient under real-world conditions.

To help participants better understand the state of the network, we recently released new visualization and diagnostic tooling focused on finalizer activity and network health. The tooling allows users to identify which finalizers appear online or offline based on recent voting activity, view aggregate online and offline stake, inspect individual finalizer status, and better interpret the current state of the BFT network. Recent updates also introduced additional filtering and the visualization of finalizers’ direct connection status, made possible by peer-to-peer topology improvements.

A major goal of this work is to ensure that participants have the information necessary to make informed decisions themselves rather than relying on Shielded Labs to dictate outcomes. One of the core ideas behind the Crosslink architecture is that different participants in the system have different responsibilities:

  1. Wallets protect users.
  2. Miners include transactions and produce blocks.
  3. Finalizers enforce finality.
  4. Stakers select good finalizers.
  5. Users slash stakers if they fail to do their job.

To that end, we are developing recovery tooling that could allow users to coordinate an emergency network upgrade if the broader network concludes that action is necessary, including mechanisms that could socially slash stake delegated to persistently inactive finalizers. The operational and coordination challenges involved here are exactly the sort of issues the feature net is intended to surface before any production deployment is considered.

What’s Next

When we originally introduced the incentivized feature net, the plan was for development to progress through a series of “seasons,” where each new season would effectively operate as a new network with updated software, fresh consensus history, and improved functionality. The idea was that participants could continue earning rewards while porting over their finalizer identities and transitioning between progressively more mature networks.

However, based on the issues we have encountered so far, we have decided to move away from the idea of regularly restarting the network and launching entirely new seasons on a predetermined schedule. As mentioned earlier in this post, we believe there is significant value in working through failures and recovery scenarios in a live environment rather than repeatedly resetting the network and starting over. While that approach can be slower and more painful in the short term, we believe it will produce a stronger network and give us a much deeper understanding of how the system behaves under real-world conditions.

Going forward, the plan is to continue iterating on the existing network through regular software updates that improve stability, fix bugs, expand tooling, and introduce more mature functionality over time, only restarting the network when deemed essential for development[1]. We will continue to provide regular updates on network progress, major issues, new tooling, and changes to the feature net as development continues.

Reward Distribution

Rather than tying incentives to discrete “seasons,” rewards will instead be distributed on a rolling six week cadence. Before each payout period begins, we will announce how much ZEC has been allocated for that six week period.

We will also be announcing the payout process for the first six weeks shortly. Because this is the initial distribution period, there will likely be a short delay before rewards are processed as we finalize the tooling and operational workflow. The current plan is for participants to submit a feature net wallet viewing key along with a mainnet Orchard address where rewards should be sent. As originally specified, rewards will be based on cTAZ earned through mining and staking activity during the payout period, and not total cTAZ holdings.

Participants will have a limited claim window to submit the required information and receive rewards. Any unclaimed ZEC remaining after that period may be used as discretionary rewards for contributors who went above and beyond in testing, debugging, reporting issues, or otherwise helping improve the network. As previously mentioned, Shielded Labs will not be a recipient of ZEC-based rewards from the incentivized feature net.

Conclusion

We appreciate everyone who has continued participating despite the instability and the inevitable frustrations that come with testing software at this stage of development. The feedback, failures, and edge cases encountered over the past six weeks have already significantly improved our understanding of the system and will help guide the next phase of development.

Despite the challenges, many participants have continued providing valuable feedback, testing edge cases, reporting bugs, developing tooling, and helping improve the network in meaningful ways. We especially want to thank @zk_nd3r and @kenbak for being consistently engaged and going above and beyond by building block explorers and contributing important infrastructure. We would also like to recognize @dismad, @pacu, Orchard Guardian, and Tom Z for their valuable feedback and contributions throughout the first six weeks.

We also understand that the current state of the network can make participation difficult or frustrating for some users. There is absolutely no expectation that everyone participate at this early stage. Participants are free to step away and rejoin later as the network becomes more stable and mature. As previously mentioned, the amount of ZEC allocated to the incentivized feature net is expected to increase over time as the network and tooling continue to improve. We also expect the network to continue running for an extended period of time, likely through 2026 and into 2027. In addition, we plan to continue hosting workshops and community calls on a regular basis, approximately once every six weeks, and there will continue to be plenty of opportunities for participants to get involved.


Thanks to @shielded-nate, @zooko, @azmr, and @philliptrudeau for their helpful insights and feedback on this post.


  1. Note: While we are moving away from planned seasonal resets, we may still reset the network when doing so allows us to simplify the codebase or make significant design improvements that would otherwise break compatibility. Backwards compatibility is not a primary goal at this stage, but it will become increasingly important as we move closer to production readiness. ↩︎

9 Likes

Thanks for the update. It’s been a pleasure taking part in this from the start.

The v6 peer discovery and GUI improvements have made a real difference in understanding what’s happening on the network. Something I’d love to see is making that diagnostic data even easier to share or export from the GUI. More and more people are going to be using AI to help them run nodes and interpret what they’re seeing, and having structured data they can just paste would make onboarding way smoother.

Hopefully we manage to unstall this BFT!

PS: The one thing that’s been rough is wallet sync restarting from scratch on every reboot.

4 Likes

Thank you, @aquietinvestor, appreciated.

Working through the rough states live, instead of treating every stall as a reset button, has made the featurenet much more useful. It exposes actual operational questions a finality layer needs to answer: what is still making blocks, what is finalized, who is participating, and what evidence operators need before coordinating recovery.

v6 peer-discovery and finalizer-visibility made that state much easier to reason about. +1 to @kenbak: making diagnostic state easy to export, paste, and inspect as structured data is a right next step for operators.

Glad to keep helping on the observability side.

6 Likes

We’ll be hosting another workshop on Wednesday, June 10 at 10am Pacific / 5pm UTC on the Zcash Global Discord. We expect to have version 7.0 available a couple of days before the workshop and encourage participants to upgrade in advance if possible.

During the session, we’ll:

  • Discuss the payout process and expected timeline
  • Demonstrate networking and syncing improvements included in recent releases
  • Encourage participants to start mining again
  • Review the current state of the network
  • Discuss options for resuming BFT finality
  • Gather feedback from participants
  • Answer questions and discuss next steps

@zk_nd3r will also provide a demonstration of the CTAZ block explorer and some of the tools it provides for monitoring network health, tracking finality, and diagnosing issues.

This is a good opportunity to get caught up on recent progress, provide feedback, and help us prepare for the next phase of testing. If you’re participating in the feature net or considering getting involved, we encourage you to attend.

We look forward to seeing you there!

9 Likes