# Ziggurat 3.0 **URL:** \ **Category:** Applications\ **Created:** [November 14, 2022, 11:20pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350 "2022-11-14T23:20:01Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![shieldedmark](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/shieldedmark/32/36509_2.png) [@shieldedmark](https://forum.zcashcommunity.com/u/shieldedmark)\ **Post date:** [April 27, 2023, 11:28pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/21 "2023-04-27T23:28:47Z")
Sure, you can hit me up on [https://twitter.com/aphelionz](https://twitter.com/aphelionz) ---
**Author:** ![Autotunafish](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/autotunafish/32/7827_2.png) [@Autotunafish](https://forum.zcashcommunity.com/u/Autotunafish)\ **Post date:** [May 9, 2023, 7:06pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/22 "2023-05-09T19:06:10Z")
Ran crunchy and now p2p-viz w/ the state.json output, still playing with it but yeah thats cool ![16836591339387592992901015117122](https://global.discourse-cdn.com/zcash/original/3X/e/b/ebb3d8cf87fe0323aad5ae1a464f79a3b915dc31.jpeg) ---
**Author:** ![Autotunafish](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/autotunafish/32/7827_2.png) [@Autotunafish](https://forum.zcashcommunity.com/u/Autotunafish)\ **Post date:** [May 9, 2023, 8:51pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/23 "2023-05-09T20:51:35Z")
Is the crunchy filter option inclusive or exclusive? ---
**Author:** ![asmie](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/asmie/32/27695_2.png) [@asmie](https://forum.zcashcommunity.com/u/asmie)\ **Post date:** [May 9, 2023, 9:39pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/24 "2023-05-09T21:39:12Z")
Inclusive. Setting: ```auto --filter-type ``` means that only nodes matching FILTER\_TYPE will be included in output. ---
**Author:** ![kylegranger](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/kylegranger/32/27699_2.png) [@kylegranger](https://forum.zcashcommunity.com/u/kylegranger)\ **Post date:** [May 10, 2023, 8:02am UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/25 "2023-05-10T08:02:09Z")
Thank you so much! I enjoyed doing this. 3D and crypto both have a lot of math, which I love doing in **code** : making something **real** out of theory and abstractness. ---
**Author:** ![Autotunafish](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/autotunafish/32/7827_2.png) [@Autotunafish](https://forum.zcashcommunity.com/u/Autotunafish)\ **Post date:** [May 10, 2023, 7:41pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/26 "2023-05-10T19:41:54Z")
How do I run the crawler to generate my own sample file? Is that with the ips core script? Or is there more to that? ---
**Author:** ![asmie](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/asmie/32/27695_2.png) [@asmie](https://forum.zcashcommunity.com/u/asmie)\ **Post date:** [May 11, 2023, 8:40am UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/27 "2023-05-11T08:40:02Z")
Hi! Sample command you can execute from the root repo directory (just replace IP and PORT with starting peer - either DNS seeder or any zcash node): `cargo run --release --features crawler --bin crawler -- --seed-addrs "IP:PORT" --rpc-addr 127.0.0.1:54321` And then just grab the results from running crawler from the other console: `curl --data-binary '{"jsonrpc": "2.0", "id":0, "method": "getmetrics", "params": [] }' -H 'content-type: application/json' http://127.0.0.1:54321/ > latest.json` It will dump results in the JSON file. Please give the crawler some time to work 🙂 The progress will be printed on the console (you will see the messages about generating summary - that summary can be then taken using `curl`). Please also look at the: [https://github.com/runziggurat/zcash/blob/main/src/tools/crawler/README.md](https://github.com/runziggurat/zcash/blob/main/src/tools/crawler/README.md) to get some more information about the crawler. Regards! ---
**Author:** ![Autotunafish](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/autotunafish/32/7827_2.png) [@Autotunafish](https://forum.zcashcommunity.com/u/Autotunafish)\ **Post date:** [May 17, 2023, 3:04pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/28 "2023-05-17T15:04:17Z")
The picture describes the ‘state → ips → peer’ file loop from crunchy feeding back into the crawler. Would I extract those peers and set them as the crawler seed addrs? ![image](https://global.discourse-cdn.com/zcash/original/3X/7/b/7b20f66cc2fdc958ec0c1332be6c49acad1fe6fc.jpeg) ---
**Author:** ![asmie](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/asmie/32/27695_2.png) [@asmie](https://forum.zcashcommunity.com/u/asmie)\ **Post date:** [May 17, 2023, 10:41pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/29 "2023-05-17T22:41:15Z")
`crunchy` produces output both for `p2p-viz` and for IPS module. IPS module uses crunchy state and produces `peer.json` which is generally a plain JSON file. You can always open it under any text editor and grab peers manually. `peer.json` is not used by the crawler as a new seed source. Crawler can use this file to get peers which should be proposed to the particular node. It means, when the node is crawled, crawler can send to it `addr` message containing peer addresses. If the node accepts or decline them, it is a separate case and it is connected with network peering behaviour (policy about accepting/declining address proposals and peer sharing rules). ---
**Author:** ![Autotunafish](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/autotunafish/32/7827_2.png) [@Autotunafish](https://forum.zcashcommunity.com/u/Autotunafish)\ **Post date:** [May 18, 2023, 2:40pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/30 "2023-05-18T14:40:07Z")
I’ve noticed that running the crawler at different intervals produces slightly varied results. On the default five second interval it finds about 185-ish nodes total, only two of which are zebra nodes and that’s both for an amount of crawl time between a few minutes and a day. Running the crawler on a 100 second interval (obviously for a little bit longer) finds about 150-ish nodes total but this time six or seven zebras which is closer to what is believed to be the actual amount running between myself and some other devs at the ZFND. Have you experienced this? Do you have any thoughts or insight perhaps as to why the intensity of the crawl may be changing the results? The intensity/interval is what was first suggested as being potentially actionable here. ---
**Author:** ![asmie](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/asmie/32/27695_2.png) [@asmie](https://forum.zcashcommunity.com/u/asmie)\ **Post date:** [May 18, 2023, 6:38pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/31 "2023-05-18T18:38:20Z")
Different intervals shouldn’t influence on the crawler results. Crawler interval says mainly how often main loop of the application is executed and how often it should scan both known and not known network searching for nodes to connect to. Basically, new peers are crawled instantly, while those ones which has been crawled in the past, would be crawled again after some time. Different results on how many Zebra nodes are found can be caused by some other reasons. First reason can be connected with the way that peerlists are returned. Vast majority of nodes return 1k addresses but they know it much more - lists are randomized and during single run you can find different set of nodes. However, running crawler long enough should cause that the same peers would be visited more than once, allowing to get more peers. The second reason can be connected with Zebra behaviour we’ve observed - there are some time slots when Zebra stops responding for incoming connections, especially under heavy load. If crawler hits such slot it won’t get connected and won’t add that Zebra to the list of known nodes. Moreover, if Zebra node would be added initially to the known network and won’t respond when crawler will try to connect to it again, it won’t be reported in the final JSON file. We can discuss this problem on Discord. ---
**Author:** ![teor](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/teor/32/17846_2.png) [@teor](https://forum.zcashcommunity.com/u/teor)\ **Post date:** [May 18, 2023, 10:36pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/32 "2023-05-18T22:36:19Z")
> [@asmie](#): > > The second reason can be connected with Zebra behaviour we’ve observed - there are some time slots when Zebra stops responding for incoming connections, especially under heavy load. If crawler hits such slot it won’t get connected and won’t add that Zebra to the list of known nodes This is likely due to some of Zebra’s security features: - peer connection count limits - new peer connection rate limits - disconnecting some peers when Zebra is under heavy load If this impacts your use case, we can adjust these parameters, but in general standard node usage shouldn’t reach them, unless there are other peers or scanners are behaving badly. ---
**Author:** ![Rqnsom](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/rqnsom/32/27775_2.png) [@Rqnsom](https://forum.zcashcommunity.com/u/Rqnsom)\ **Post date:** [May 19, 2023, 11:19am UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/33 "2023-05-19T11:19:38Z")
So, we are aware there are more than eight zebra nodes in the network. And that’s because we are monitoring active connections in our zcashd node, where we can currently observe eight zebrad nodes which are now connected to our zcashd node. The other 22 connections are from zcashd nodes. And only two of those zebra nodes are reachable to us (when we try to initiate a connection), and we can crawl those anytime. Our crawler managed to detect those two easily. So these two nodes are behaving normally the most time of the day and seem to be in a healthy state. But the other six zebrad seem to be in some state where they are either constantly under heavy load or there’s some other issue here. We can observe the same issue with our local zebrad node - there’s something strange in the logs, and during this time, the zebrad cannot accept inbound connections - and we tried to help and recover the zebrad with some methods, but nothing helps, and we can observe that we have plenty of CPU/memory resources on the system. We observed this issue only because we were stubbornly trying to establish a connection towards these zebra because we were aware these nodes were in the network, but then we wondered why we could not reach those nodes. This kind of explains why we don’t see many zebrad nodes in our crawler report. ---
**Author:** ![teor](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/teor/32/17846_2.png) [@teor](https://forum.zcashcommunity.com/u/teor)\ **Post date:** [May 21, 2023, 8:25pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/34 "2023-05-21T20:25:10Z")
> [@Rqnsom](#): > > And only two of those zebra nodes are reachable to us (when we try to initiate a connection), and we can crawl those anytime. Our crawler managed to detect those two easily. So these two nodes are behaving normally the most time of the day and seem to be in a healthy state. > But the other six zebrad seem to be in some state where they are either constantly under heavy load or there’s some other issue here. Maybe those nodes don’t have a public IP address? This is a configuration choice, and some nodes can only make outbound connections due to NAT or firewalls. (`zcashd` can do this too if it is configured that way.) For anyone who is interested, we’re working through other potential bugs in Zebra in Discord: > **[Discord - Group Chat That’s All Fun & Games](https://discord.com/channels/809218587167293450/809251029673312267/1109936533923958845)** > > Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out. ---
**Author:** ![Rqnsom](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/rqnsom/32/27775_2.png) [@Rqnsom](https://forum.zcashcommunity.com/u/Rqnsom)\ **Post date:** [May 26, 2023, 1:35pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/35 "2023-05-26T13:35:53Z")
Okay, we have set up the inverse crawler (the node that just sits in the network and waits for inbound connections) to catch and inspect these hidden zebrad nodes in the network. > This is a configuration choice, and some nodes can only make outbound connections due to NAT or firewalls. (`zcashd` can do this too if it is configured that way.) And indeed! It seems like some of these zebrad nodes are hidden behind a firewall (or don’t have a listening IP), so our normal crawler can’t reach those. That got us thinking - we could extend the crawler with some passive, silent listener to hunt down those invisible nodes in the network. Also, an interesting unrelated finding: Some of the zebrad nodes we found often create two connections to our listening node ---
**Author:** ![teor](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/teor/32/17846_2.png) [@teor](https://forum.zcashcommunity.com/u/teor)\ **Post date:** [May 29, 2023, 11:46pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/36 "2023-05-29T23:46:44Z")
> [@Rqnsom](#): > > Also, an interesting unrelated finding: > Some of the zebrad nodes we found often create two connections to our listening node Does your listening node have multiple public IP addresses? We’ve just fixed a bug in Zebra that could have made our address book peer state get overwritten with an older state. So please wait for the next release and re-test for this issue. ---
**Author:** ![Rqnsom](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/rqnsom/32/27775_2.png) [@Rqnsom](https://forum.zcashcommunity.com/u/Rqnsom)\ **Post date:** [May 30, 2023, 7:27am UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/37 "2023-05-30T07:27:42Z")
> Does your listening node have multiple public IP addresses? Our listening node uses just one public address - but we do have multiple different nodes on the same address. One node uses 8233, and the other nodes pick a random port. > We’ve just fixed a bug in Zebra that could have made our address book peer state get overwritten with an older state. So please wait for the next release and re-test for this issue. We left our synth node running for a few days, and then when we came back online on Monday, we found one of those ‘hidden’ zebrad created nine connections towards our synth node - so we can easily re-test this for the next release. ---
**Author:** ![teor](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/teor/32/17846_2.png) [@teor](https://forum.zcashcommunity.com/u/teor)\ **Post date:** [May 30, 2023, 9:32pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/38 "2023-05-30T21:32:41Z")
> [@Rqnsom](#): > > > We’ve just fixed a bug in Zebra that could have made our address book peer state get overwritten with an older state. So please wait for the next release and re-test for this issue. > > We left our synth node running for a few days, and then when we came back online on Monday, we found one of those ‘hidden’ zebrad created nine connections towards our synth node - so we can easily re-test this for the next release. Just making sure: do these connections all last for exactly the same time, or are they only overlapping for a short while? Or do they not overlap at all? Because the way our code is set up, the overlap should only be around 30 seconds. (Depending on exactly what it causing it.) ---
**Author:** ![Rqnsom](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/rqnsom/32/27775_2.png) [@Rqnsom](https://forum.zcashcommunity.com/u/Rqnsom)\ **Post date:** [May 31, 2023, 11:49am UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/39 "2023-05-31T11:49:26Z")
Okay, let us show you our output from Monday (I’ll hide the actual IP here) - and this zebra node here reports the `Zebra:1.0.0-rc.8` version in the handshake procedure. ```auto IP:61725 (Initiator) connection established for 261659.187967142 seconds IP:63300 (Initiator) connection established for 200290.488654367 seconds IP:63475 (Initiator) connection established for 186612.76220032 seconds IP:49607 (Initiator) connection established for 153232.705936181 seconds IP:49860 (Initiator) connection established for 146238.805304459 seconds IP:52390 (Initiator) connection established for 122062.857630915 seconds IP:53090 (Initiator) connection established for 116131.909325629 seconds IP:57706 (Initiator) connection established for 111394.034115723 seconds IP:60784 (Initiator) connection established for 109360.912254869 seconds ``` > [@teor](#): > > or are they only overlapping for a short while? Or do they not overlap at all? So it’s a mix of both. In the above printout, we only show the currently active connections, it doesn’t have any historical connections, so there might have been established more than nine connections at some time. From what we see, this issue seems to be more common with unreachable zebra nodes. But there was also one case with one reachable `Zebra:1.0.0-rc.7`, which also created three non-overlapping connections: ```auto IP2:41406 (Initiator) connection established for 208085.12634777 seconds IP2:56146 (Initiator) connection established for 148684.528463004 seconds IP2:36398 (Initiator) connection established for 20099.454923311 seconds ``` ---
**Author:** ![teor](https://sea2.discourse-cdn.com/zcash/user_avatar/forum.zcashcommunity.com/teor/32/17846_2.png) [@teor](https://forum.zcashcommunity.com/u/teor)\ **Post date:** [May 31, 2023, 10:43pm UTC](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350/40 "2023-05-31T22:43:06Z")
> [@Rqnsom](#): > > Okay, let us show you our output from Monday (I’ll hide the actual IP here) - and this zebra node here reports the `Zebra:1.0.0-rc.8` version in the handshake procedure. > > … > > > [@teor](#): > > > > or are they only overlapping for a short while? Or do they not overlap at all? > > So it’s a mix of both. In the above printout, we only show the currently active connections, Thanks for helping us diagnose this, if it is the bug I’m thinking of, it should be fixed in 1.0.0-rc.9. Do you know if there are multiple Zebra instances behind that IP address? > [@Rqnsom](#): > > it doesn’t have any historical connections, so there might have been established more than nine connections at some time. > > From what we see, this issue seems to be more common with unreachable zebra nodes. > But there was also one case with one reachable `Zebra:1.0.0-rc.7`, which also created three non-overlapping connections: I think we’ll only treat this as a bug if a single Zebra instance makes multiple outbound connections to the same IP address, and they overlap for a long while. It could be valid node behaviour to: - make a new connection and then close the old one, - connect via multiple addresses to the same node, or - accept multiple inbound connections from the same IP (because we have no way of telling the difference between NAT, multiple instances on the same IP, and multiple connections from a single instance) [Previous page](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350.md?page=1) [Next page](https://forum.zcashcommunity.com/t/ziggurat-3-0/43350.md?page=3)