opened 04:56AM - 28 Mar 17 UTC
A-wallet
A-dependencies
S-needs-clearer-scope
licensing
A-wallet-database
On #2194, @zooko wrote:
> I am a huge fan of sqlite and would happily give my :… +1: to using that. It has a long track record of reliability in many deployments, [their test policy](https://www.sqlite.org/testing.html) is top notch and we could learn from it! I have had extensive — and entirely positive — experience working with sqlite. The developer docs are great.
>
> If anyone wants to argue against sqlite on performance grounds, that's fine, but I'm going to listen to this argument only if you bring a realistic performance test that shows (at least approximately) how many z-addresses/t-addresses/transactions/etc. incur how much of a performance penalty in a realistic scenario. My guess is that sqlite is more than performant enough for most or all practical scenarios.
@veikkoeeva wrote:
> A vote for SQLite here too, an excellent cross-platform and development friendly choice.
>
> An added note, as this may not be evident, but if ZCash were to adopt the convention that all queries will be explicitly identified and used so that the ZCash "engine" would care only that the variable and column names and types remain the same, it would be possible to switch the DB engine or the implementation on the fly. This gives the added benefit one can adapt the structures taking account the hardware and amount of data, for instance, as in sending "computation and data" between the engine and the program. Prior art is availabe at https://github.com/dotnet/orleans/tree/master/src/OrleansSQLUtils, see different scripts (the queries are keyed [like this](https://github.com/dotnet/orleans/blob/master/src/OrleansSQLUtils/CreateOrleansTables_SqlServer.sql#L252) and loaded and used by these keys). This way one can make use of the various tooling suites and backends depending on any platform variables. This is, I suppose, mostly like Stackoverflow operates too. As to why this might be important for a running system, maybe [this post mortem](https://blogs.msdn.microsoft.com/bharry/2016/02/06/a-bit-more-on-the-feb-3-and-4-incidents/) indicates as to why. As the amount of data grows or there are problems, one could adjust queries on the fly.
>
> On tangential, this would also allow one to transparently to make use features such as at https://www.pipelinedb.com/ (assuming one would switch from SQLite to PostgreSQL) or pipe data through the DB interface to any other storage (e.g. https://msdn.microsoft.com/en-us/library/mt143171.aspx) by simply modifying the queries appropriate – as it would be transparent to ZCash. I.e. making ZCash perhaps more readily integrable to various other systems.
@radix42 wrote:
> One thing I'd like to make note of is that currently all calls to access either database (LevelDB or Berkeley DB) are very much peculiar to the database in use. I'd recommend first factoring out all calls that result in database access into some nice abstraction layer, starting of course with the current databases that are in use. Then safely swapping out the backend(s) will be a whole lot easier. I bloodied my head against this some months ago when I made the start of a switch to lmdb. It was not pretty because of this.
@veikkoeeva wrote:
> @radix42 I second that too. Which is, in fact, how the aforementioned _Storage Providers_ in the just linked Orleans work. This is repeating the obvious, but for completeness sake, there's an interface and an implementation class and then a loader the loads the implementation. One backend is ADO(.NET), i.e. "relational databases" and that specific backend works as just mentioned.
@ebfull wrote:
> @nathan-at-least We tentatively believe that sqlite would be a nice replacement for Berkeley DB, and would like you feedback. If we don't hear from you soon, we will assume you agree.
@zooko wrote:
> [...] there are two different uses of DBs in Zcash — 1. the chain state DB that contains UTXOs, tree states, and stuff like, and 2. the wallet data file. The latter is where we currently use BerkeleyDB, and we definitely want to get rid of it (unreliable, problematic licensing, frequent backward-incompatible releases), and the latter doesn't ([I guess](https://github.com/zcash/zcash/issues/2194#issuecomment-288827202)) need better-than-sqlite performance.
>
> I also [agree] with what @veikkoeeva and @radix42 are saying above — we should use an abstraction layer: https://github.com/zcash/zcash/issues/2194#issuecomment-289211140, https://github.com/zcash/zcash/issues/2194#issuecomment-289028249
See also #1469, which focuses on licensing issues with BDB; and #1694, an example of a user losing funds apparently due to BDB database corruption.
Can we talk about this? Its been an issue for years and is still stuck in the security backlog
str4d
January 21, 2021, 8:59pm
2
Hmm, we should close those GitHub projects (we migrated to ZenHub last year because GitHub on its own didn’t provide what we needed, so those columns haven’t been updated).
As for this specific proposal, it was shelved while we were developing Sapling and then the mobile SDKs. Our current preferred direction is to migrate away from the C++ wallet entirely, to a Rust wallet (which would share its business logic with the mobile SDKs, and could also be compiled to WASM for use in browser extensions).
The first step towards this is the just-merged Data Access API , which starts the process of splitting the specific SQLite database format used by the light client mobile SDKs, out from the wallet logic (to enable the latter to be backed by other databases, or used in WASM contexts where you don’t have direct access to I/O). Note that even if zcashd moved to using SQLite, it would most likely use a completely separate database format from the mobile SDKs.
3 Likes
str4d:
As for this specific proposal, it was shelved while we were developing Sapling and then the mobile SDKs. Our current preferred direction is to migrate away from the C++ wallet entirely, to a Rust wallet (which would share its business logic with the mobile SDKs, and could also be compiled to WASM for use in browser extensions).
That sounds wonderful!
I like almost everything about that answer, thanks! My remaining question relates to the ZenHub portion. Is there a public link for Zcash there? It is nice for the community to be able to track some of these priorities and in some cases actively contribute.
str4d
January 21, 2021, 9:21pm
4
ZenHub doesn’t provide any way to create a public view of a board AFAIK. Currently, the best view into what the ECC Core team is working on for each sprint, is the GitHub milestones created by ZenHub across the various repos (e.g. Core Sprint 2021-02 Milestone · GitHub and Core Sprint 2021-02 Milestone · GitHub for the current sprint).
2 Likes
Thanks. That is helpful too