Seeking feedback about Observatory (network security) tooling

Community Grants
1

Hey everybody,

I wanted to get your input about building a more sophisticated logging and alerting platform designed to increase Zcash ecosystem analysis capability for facets and phenomena that are not tracked by an out-of-the-box node.

As Zcash adoption expands and the network scales, consumers, researchers, and businesses will all require high-quality data quantifying network health and security. Users need data and visibility related to both performance (latency, load, etc) and security (reorganizations, double spend attacks, etc). Our Observatory nodes and network will detect and alert for a variety of these phenomena. In addition to monitoring and reporting network security, quantified understanding of network characteristics will inform protocol design decisions related to Zcash scaling and privacy. A comprehensive Observatory NetSec system will increase Zcash’s attractiveness for commercial adoption and infrastructure integration.

The Observatory is a full-stack project with a variety of key deliverables & features:

  • Archive and analyze alternative/orphaned blocks and transactions
  • Visualize, detect, and alert on potential double spend attacks
  • Visualize, detect, and alert on probable selfish/stubborn mining
  • Statistics and visualizations for global network performance
  • Block propagation time
  • Miner-timestamp spoofing
  • Open-source public front-end/dashboard to visualize Zcash network health & security
  • Research database for Zcash analysis (nearest is BigQuery, lacking NetSec data)
  • The Observatory will be released as free open-source software

Once the Observatory is complete, we will have addressed several GitHub issues (many open since 2016):

The architecture is relatively intuitive, using Kafka to manage streaming events, feature engineering, and anomaly detection. This provides the data source for the API, research database, and front-end dashboard.

1600Ă—548 76 KB

There are two levels of logging and analysis, depending on whether you have one Observatory node (has been prototyped), versus collecting data from an Observatory cluster (upcoming development initiative). Comparing the “Single node” and “Observatory network” columns below shows how the former captures an informative but incomplete state of the network based solely on the state nearby peers, whereas a cluster of Observatory nodes with high coverage of the organic p2p network can draw robust conclusions about the overall system’s health and performance.

I’d love your feedback on other ideas for the Observatory: what data and analyses would you like to see? Do you think Zcash tooling like this would be an appropriate project for Zcash Open Major Grants (ZOMG) funding?

Thanks,
-:- Mitchell

7 Likes
2

Really hope you do submit a proposal, and that @ZcashGrants funds it!

2 Likes
3

Hi @mitchellpkt, the ZOMG would be interested in hearing out the proposal based on our past track record of looking at similar proposals (e.g. Ziggurat: the Zcash Network Stability Framework).

The more engagement you get on this topic, the better as well. We would like to see that the community and our core builders finds this valuable. Perhaps @alchemydc @dconnolly from the ECC and ZF, and stalwarts like @aiyadt @adityapk00 @hanh can weigh in.

1 Like
4

Thanks for pinging me @ml_sudo.

Great to hear from you @mitchellpkt! I believe that additional visibility and alerting at the network level creates significant value (in the form of risk mitigation) for the broader Zcash ecosystem. Would I be correct to assume that the proposed work would build upon the work your team did with ZF funding last year?

I sense that centralized exchanges would benefit from this type of visibility, monitoring and alerting. I suspect that many of them have already built similar systems but won’t be sharing the source code or ops runbooks with the community any time soon.

Would this proposal cover operating the solution as a service? Or is the scope limited to creating artifacts that others would need to stand up to deliver ongoing visibility, monitoring and alerting?

Keen to hear feedback from @earthrise.

Regards,

DC

3 Likes
5

I’m excited to hear that there is interest for this project! Yep, @alchemydc we will reuse the analysis framework and detector logic from the solo observatory project. The big difference this time will be in transitioning from batch analysis to an event-driven architecture. This will allow us to implement our analytics as stream processors so that live insights (forks, selfish mining, etc) can be delivered over APIs to downstream data consumers (dashboard, alarms, exchanges, etc). We will also make improvements to the data collection infrastructure by building clusters of observatory nodes that can capture a complete picture of the global network, rather than the limited localized snapshot recorded by a solo observatory node.

We’ll run the tooling for a few months to collect feedback and catch any bugs. Long-term operations would be submitted as a separate subsequent grant, so that it can be evaluated separately from the dev work.

3 Likes