I wanted to get your input about building a more sophisticated logging and alerting platform designed to increase Zcash ecosystem analysis capability for facets and phenomena that are not tracked by an out-of-the-box node.
As Zcash adoption expands and the network scales, consumers, researchers, and businesses will all require high-quality data quantifying network health and security. Users need data and visibility related to both performance (latency, load, etc) and security (reorganizations, double spend attacks, etc). Our Observatory nodes and network will detect and alert for a variety of these phenomena. In addition to monitoring and reporting network security, quantified understanding of network characteristics will inform protocol design decisions related to Zcash scaling and privacy. A comprehensive Observatory NetSec system will increase Zcash’s attractiveness for commercial adoption and infrastructure integration.
The Observatory is a full-stack project with a variety of key deliverables & features:
- Archive and analyze alternative/orphaned blocks and transactions
- Visualize, detect, and alert on potential double spend attacks
- Visualize, detect, and alert on probable selfish/stubborn mining
- Statistics and visualizations for global network performance
- Block propagation time
- Miner-timestamp spoofing
- Open-source public front-end/dashboard to visualize Zcash network health & security
- Research database for Zcash analysis (nearest is BigQuery, lacking NetSec data)
- The Observatory will be released as free open-source software
Once the Observatory is complete, we will have addressed several GitHub issues (many open since 2016):
- Public chain-fork detector
- Block Observatory
- Timestamp Observatory
- Cross-branch double-spend detector
- Internal chain-fork detector
- Concept for a distributed monitoring service
The architecture is relatively intuitive, using Kafka to manage streaming events, feature engineering, and anomaly detection. This provides the data source for the API, research database, and front-end dashboard.
There are two levels of logging and analysis, depending on whether you have one Observatory node (has been prototyped), versus collecting data from an Observatory cluster (upcoming development initiative). Comparing the “Single node” and “Observatory network” columns below shows how the former captures an informative but incomplete state of the network based solely on the state nearby peers, whereas a cluster of Observatory nodes with high coverage of the organic p2p network can draw robust conclusions about the overall system’s health and performance.
I’d love your feedback on other ideas for the Observatory: what data and analyses would you like to see? Do you think Zcash tooling like this would be an appropriate project for Zcash Open Major Grants (ZOMG) funding?