ZCG and the Dev Fund: Timeline for Proposals and Polling

Actually - AI is writing better code and having to clean up all human code - why we are in a security crisis. And AI caught this bug. It absolutely can audit dev change quality and who is an expert in what. Saying otherwise means you either aren’t an industry expert or aren’t paying attention.

Also, he is 100% right that there SHOULD have been a full post mortem before diving headfirst into fighting a social (whatever word I am allowed to use here @Shawn ) war with ZCG.

I understand free speech but on a normal corporate project, she would be given the task of post mortem not endless social program opinions.

Her focus is obviously split.

1 Like

What are you talking about? I’m a co-author on the Halo paper and co-designer of Halo 2.

Do you you know why your AI is telling you that most of the caught soundness bugs were in Ying Tong’s PRs? It’s because she wrote almost all of the Orchard circuit code. And she did an exceptionally good job of it. I will not stand by while you slander former colleagues and mislead people with uninformed, drive-by analysis of mistakes in a very high-quality codebase, developed with review practices far above the pre-AI norm.

9 Likes

I did not advocate for any wallet to move to 400 ZATs at any point in time. I only advocated for the already ZIP editor agreed upon move to 1000 ZATs in wallets to happen faster. I even posted advocated for upgrading faster on the ZIP Add ZIP 2009: Reduce Marginal Fee to 1000 Zatoshis by aphelionz · Pull Request #1352 · zcash/zips · GitHub .

I hope you are aware, at every point you are assuming adversarial intent and layered conspiracies. For future readers, the first post in this accusation thread has been deleted.

I spot-checked the first PR your AI mentioned, orchard#67, which is relatively typical for orchard circuit PRs. I know this code well beause I designed Sinsemilla and wrote most of the design note this PR is based on, with contributions from Sean and Ying Tong.

The vast majority of comments in the review are about minor things: code comment suggestions, documentation consistency, API visibility issues, possible performance improvements, etc. There are also a small number of correctness issues that would have affected completeness for all instances, and therefore inevitably been found by tests. This all attests to the thoroughness of review.

One thing should be noted where the AI has got it wrong: I didn’t miss the soundness issues in this PR. I clearly said in my initial review, “I have not checked these constraints yet.” At that point I was checking for code quality and documentation issues. str4d reviewed the constraints first, and when I did get to the constraint review, I found the same issues he did. As far as I can tell, all soundness issues in this PR had been fixed by the time I approved it. (The zero-length input issue is not a bug.)

There are, depending on how you count, three high-level soundness-related issues in that PR: 1, 2a, 2b, and 3. The first is a non-trivial design issue that was thoroughly addressed; if you had more experience then you would realize that this is just part of the process of designing and implementing arithmetic circuits. The second is a layout error, fixed in the same PR here. It was found because of the thoroughness of the design note I wrote.

The third, fixed in this commit, has a roughly similar shape to the eventual bug that we missed. What should we conclude from that? Simply what I said before: it’s difficult and there are a lot of things to get right. Without AI and at a time when formal verification was completely impractical, mistakes could still be made despite competent implementation and what anyone can see to be very thorough review. That was the state of the art at the time. It’s consistent with everything I’ve ever said about the difficulty of writing these circuits and soundness risk.

4 Likes

Edit: @shieldedmark has said that the summary on the Arborist call was his misinterpretation/miscommunication of what you said. I apologize and withdraw my related criticisms.

@ValarDragon, did you or did you not have the conversation with @shieldedmark that he related on the October 1st Arborist call, and is his summary of the conversation broadly accurate? As a reminder, he said:

> I received a message from Dev at Valar Group yesterday [September 30], signaling his intention to put the marginal fee at 400 in Zakura and start telling wallets to upgrade on October 12.

Now, it’s technically true that, despite the context of just having told him that Zakura had changed it to 400 zats, “telling wallets to upgrade” could have meant to 1000 zats. Mark clearly took it, from the rest of his comments, as meaning to 400 zats.

You’re free to correct the record. But I think everyone can see that I was entirely justified, based on that summary, to come to the conclusions that I did. It is a fact that Zakura lowered its marginal fee to 400 zats for both template and eviction policy: zakura#1218 (it also changes the relay cap to 800 zats which is fine, that’s consistent with ZIP 2009). It was opened by you, approved by Evan Forbes, and merged by you on 30th September, the day Mark says you talked to him. So Mark’s summary is entirely consistent with the information available, and we could see at the time that it was.

AI didn’t catch it on its own. Taylor Hornby, engaged by Shielded Labs to hunt for exactly this kind of flaw, used the (then just 1-2 days released) frontier LLM at the Orchard circuit.

According to a public summary of his testing, Claude Opus 4.8 found the bug only one time in four under a general prompt, and Opus 4.7 missed it altogether.

But with a targeted prompt telling it exactly where to look, then Opus 4.7 also found it.

And when we say targeted we mean this kind of prompt: “review the variable-base scalar multiplication circuit for missing constraints that could lead to inflation or double-spending.”

The model needed a human expert who knew where to look.

Such is progress that current frontier models might just find it on their own, given a proper specification. Hence the focus on formal methods and proofs in Ironwood.

I’m just a subject matter enthusiast in these moon-math systems, but I know how to Google.

Attacks on anyone’s competence or character won’t make the software any safer. The attacks in this thread have been particularly distasteful, from people who should know better.

P. S. Pro TIP: If you use the voice feature of the ChatGPT app in your phone to make prompts, it reasons using the ‘Instant’ model. Not the higher models. Just wanted to let you know.

4 Likes

Hey, sorry for my contribution to the confusion here with my imprecise language on the Arborist call. In conversation I had with @ValarDragon, he advocated for:

  1. Changing the relay policy / mempool eviction threshold to 400 zats in Zakura
  2. Pushing the ZIP-2009 wallet cutover date to Oct 12, which would have had wallets adopt 1000 zats.

I hope this information helps.

3 Likes

@ValarDragon, I apologize. Mark’s description on the Arborist call of what you had said was clearly inaccurate, it was his misinterpretation/miscommunication of what you said.

However that happened, it’s quite a serious near-miss process failure given that it could have led to other wallets adopting 400 zats rather than 1000 zats. This is why I strongly prefer communication about specification issues in public, with sufficient time to avoid/correct any errors, and without diverging from the agreed timeline.

I will raise this at the R&D meeting today. It’s a problem because it increases the risk that a sandblasting-style attack could recur.

Edit: I forgot to raise this. I’ll do so at a future meeting.

1 Like

Thanks for adding context. If you’re aware of other relevant examples of quality/safety/velocity tradeoffs, please feel invited to bring them up.

There are three names on the paper, and those three stood on the shoulders of giants, as perhaps everyone credited with a breakthrough does.

Sean is clearly competent.

That understates it, Sean is beyond brilliant, as are Str4d, Daira-Emma, and Ying Tong.

I’m not sure what your track record is or what otherwise qualifies you to judge who should be credited with what, but your claims look borderline indefensible and sound uninformed.

6 Likes

For the record, Sean, Str4d, and I are equal co-authors on the Halo paper. The authors are in alphabetical order by surname, as is the usual convention for equal co-authors. Sean had the main idea of nested amortization, no question about that. I’m not going to dignify the rest of your trolling with a response.

3 Likes

Behold the power of an ignorant person with access to AI.

Why is this person still here with 3 rules violations?

Unacceptable behavior includes:

  • Personal attacks (ad hominem)
  • Trolling, insults, or derogatory comments
  • Harassment of any kind (public or private)
5 Likes

Hi @karma,

I wanted to ask your opinion about something.

Solana has published postmortems for seven major network incidents:

  • 4th Dec 2020 | ~6 hours downtime: Block repair and propagation failures stopped the network. Source

  • 14th Sep 2021 | ~17 hours downtime: Bot traffic exhausted validator memory, causing crashes and consensus failure. Source

  • 30th Apr 2022 | ~7 hours downtime: NFT-minting bots overwhelmed validators, exhausting memory and stalling consensus. Source

  • 1st June 2022 | ~4.5 hours downtime: A durable-nonce processing bug caused validators to disagree about block validity. Source

  • 30th Sep 2022 | ~8 hours downtime: A fork-selection bug prevented consensus after duplicate blocks were produced. Source

  • 25th Feb 2023 | ~20 hours of degradation and downtime: Third-party block forwarding overwhelmed propagation, leading to severe degradation and a coordinated restart. Source

  • 6th Feb 2024 | ~5 hours downtime: A program-cache recompilation loop halted finalisation. The bug had already appeared on Devnet. Source

These were failures of liveness, although the causes differed. There has been no comparable network-wide halt since February 2024.

Zcash has had a different failure mode. The Orchard soundness vulnerability disclosed in June 2026 involved an under-constrained element in the Action circuit that could have permitted undetectable counterfeiting within the shielded pool. The chain kept producing blocks throughout.

I assume the answer is not simply proof of work versus proof of stake. Ethereum is also proof of stake and, although it suffered temporary finality failures in May 2023, it did not require a network-wide restart comparable to Solana’s.

What explains the difference in how these projects develop, review, test and release consensus-critical software?

1 Like

A few specific questions (I am ignorant about much of it):

1. Why did known bugs reach Solana Mainnet? The December 2020 defect was already known, and the February 2024 bug had appeared on Devnet. What failed between identifying the problem and fixing it in production?

2. How do you test fork-selection logic? The September 2022 bug stopped consensus after duplicate blocks were produced. How do you reproduce these conditions in testing? What was being tested then, and what is tested now?

3. Why did bot traffic halt Solana but not Zcash? The obvious difference is throughput. Zcash’s sandblasting attack bloated the chain and hurt wallet synchronisation, but blocks continued.

NU7 reduced block intervals from 75 to 25 seconds and introduced limits on shielded activity to prevent spam. Project Tachyon aims to take throughput much further, potentially to 50,000 transactions per second.

So, looking ahead, which of Solana’s failures were inherent to operating at that scale, and which were avoidable?

4. What changed at Solana after February 2024? Which technical or procedural improvements explain Solana’s better uptime since?

5. How many people can independently review the code for each project? Auditing a Halo 2 circuit and auditing consensus code seem to me rather different problems.

Is there a meaningful way to compare their complexity, and how many qualified independent reviewers exist for each?

Could the small pool of ZK circuit specialists help explain why Orchard’s soundness vulnerability went undetected for so long?

I am interested in how much of this comes down to architecture, and how much to what each development team considers ready to ship.

Thank you,

An interested subject matter enthusiast.

P.S. I hope these technical discussions get split into their own thread, so they can be found and continued later.

1 Like

Thanks @karma in advance for your kind answer.

But in the meantime, my best guess, based on reading Neodyme’s analysis, Helius’s outage history and the security reports on Firedancer, is that Solana’s improved uptime since February 2024 is mainly down to better engineering, testing and release discipline, rather than an increase in throughput itself.

Is that a fair assessment, or am I missing something in the underlying architecture?

One thing I expect all fair-minded people can agree on is that Solana’s history offers a lesson for other projects:

Don’t accelerate the cadence of network releases beyond your capacity to test, review and safely deploy them.

Post-Mortems are excellent.

Unfortunately, this forum is not the place for them.

Consensus failures should not happen but they are not fatal like soundness bugs that allow infinite inflation.

When you do the post-mortem on the one Daira approved and ask why was Ying Tong building the circuit.

Rather than Sean Bowe who wrote Sprout, Sapling and had the core insights for halo

This is what comes back.

No accountability, nothing learned.

Sprout and Sapling repositories show zero soundness bugs trying to be PRd in rather than over a dozen in orchard which Daira defends.

The lesson from the post mortem is you must have maximally competent people writing the circuits.

I’m glad Sean is doing it again.

Competency is everything. Processes and specs cannot compensate for it.

However this forum only allows speech that fits an agenda. So not for post-mortems.

@Shawn will likely censor this technical critique like the previous ones above.

1 Like

I agree competence matters a great deal. But it isn’t either/or.

Sprout did have a soundness bug: counterfeiting (CVE-2019-7167). An attacker could have created fake ZEC without being detected. It was found by an ECC cryptographer in March 2018 and closed in the Sapling network upgrade that October, which moved Sprout onto a new proof system.

The flaw was in the proof system Sprout was built on, which experts had studied for years. That’s the lesson: the best people still miss things, which is why review, specs and process exist. They’re what caught it, and what caught the bugs you’re counting with your abacus, in Orchard’s pull requests.

Nobody reasonable would call the people behind Sprout incompetent. Sean Bowe’s work on Sapling and Halo is some of the best in the field, and Zcash is lucky to have him. The same goes for Daira-Emma, who co-authored Halo and has maintained the Zcash protocol specification for years, and for Ying Tong, who wrote most of the Orchard circuit and, as has been said here, did an exceptionally good job of it.

Counting caught bugs by author mostly shows who wrote the code. And bugs come in degrees. One caught in review is the process working. One found in live code before anyone exploits it, like Sprout’s counterfeiting bug or the recent Orchard one, is a close call, but still caught. Only one that gets exploited is a real failure. Zcash has had the first two, with no evidence of the third.

Post-mortems work best when they’re about what failed, not who.

1 Like

Why not so? Solana’s high profile failures have a lot to teach us, no?

I am interested to hear your opinion - as you are someone presenting themselves as knowledgeable enough to opine on these issues.

What did Solana start doing right after those awful failures pre-February 2024?