Three weeks ago Josh published the first ZODL weekly update after the ECC drama. One sentence turned my attention on, at the end of the post.
As a node operator at home, I decided to do my part to contribute to this framework with my own resources. So I set up a Gitea Mirror instance you can now browse at https://codez.ombie.cash/ (it sounds like code zombie cash). I’m mirroring at the moment 392 github repositories from 18 organizations every 4 hours (about 60 GB of storage).
Unfortunately I’m able to mirror and sync only organizations, not single users like ZecHub. I need to find a scalable solution for this issue (suggestions are welcome)…
Of course, feel free to suggest new repositories, people or organizations to be included!
Latest updates of gitea and gitea mirror brought many issues and I was forced to stop both services. I’ll keep you posted when CodeZ will be available again.
I’ll continue to track and harvest other orgs and single repositories (ie. ledger / keystone, zechub, single devs), but I think the goal of this little project can be wider than archiving assets and preserve memory.
If the whole codebase history is in a single place, we can compute metrics about content (code, issues) and people (contributors and their relations). Overall and time-based.
Example: we can build the bipartite graph of contributors and repositories and project it to the social network of contributors (linked by common repos).
Was there any attempt to perform such statistics analysis on zcash community in the past?
I underestimated the impact that modern bots infesting the web can have on a small, exposed service. I had to activate all cloudflare anti-bot features, rate limiting, routes blocking and aggressive caching to protect my gitea instance.
You don’t need gitea api, you can clone some repo and develop the code, then I will run it locally on full archive (tens of GB). Take a look at the example project…
Maybe I’m misunderstanding something, but I don’t see why this would have to involve cloning when all that needs processing is contributor data. If API access is an option, cloning would seem like overkill.
Probably I need to add some more details, thanks for pointing that out.
Right now CodeZ tracks more than 500 repo using 270 GB on disk. Millions of commits (and 60k issues and 370k comments). I can’t serve that amount of data via REST API. Also github has strict limits using personal access token.
Data flow from disk is faster than on network. And you can decouple raw data access from computation. So my idea is simple: participants can develop and test against a little and manageable archive similar to the real one. Then I’ll run the code on the full archive using my own resources.
on 
