Zcash data format

aristarchus · October 18, 2020, 2:35pm

Are there plans to change the format of the blockchain data stored on a full node, either in the blocks or in the chain state directories? I saw something I think on GitHub about changing to SQL but don’t know if that is happening.

@daira, since you are an author of the spec, do you know about this? Thanks!

anon16456014 · October 18, 2020, 7:07pm

github.com/zcash/zcash

Evaluate sqlite as a replacement for BDB

opened 04:56AM - 28 Mar 17 UTC

daira

A-wallet A-dependencies S-needs-clearer-scope licensing A-wallet-database

On #2194, @zooko wrote: > I am a huge fan of sqlite and would happily give my :…+1: to using that. It has a long track record of reliability in many deployments, [their test policy](https://www.sqlite.org/testing.html) is top notch and we could learn from it! I have had extensive — and entirely positive — experience working with sqlite. The developer docs are great. > > If anyone wants to argue against sqlite on performance grounds, that's fine, but I'm going to listen to this argument only if you bring a realistic performance test that shows (at least approximately) how many z-addresses/t-addresses/transactions/etc. incur how much of a performance penalty in a realistic scenario. My guess is that sqlite is more than performant enough for most or all practical scenarios. @veikkoeeva wrote: > A vote for SQLite here too, an excellent cross-platform and development friendly choice. > > An added note, as this may not be evident, but if ZCash were to adopt the convention that all queries will be explicitly identified and used so that the ZCash "engine" would care only that the variable and column names and types remain the same, it would be possible to switch the DB engine or the implementation on the fly. This gives the added benefit one can adapt the structures taking account the hardware and amount of data, for instance, as in sending "computation and data" between the engine and the program. Prior art is availabe at https://github.com/dotnet/orleans/tree/master/src/OrleansSQLUtils, see different scripts (the queries are keyed [like this](https://github.com/dotnet/orleans/blob/master/src/OrleansSQLUtils/CreateOrleansTables_SqlServer.sql#L252) and loaded and used by these keys). This way one can make use of the various tooling suites and backends depending on any platform variables. This is, I suppose, mostly like Stackoverflow operates too. As to why this might be important for a running system, maybe [this post mortem](https://blogs.msdn.microsoft.com/bharry/2016/02/06/a-bit-more-on-the-feb-3-and-4-incidents/) indicates as to why. As the amount of data grows or there are problems, one could adjust queries on the fly. > > On tangential, this would also allow one to transparently to make use features such as at https://www.pipelinedb.com/ (assuming one would switch from SQLite to PostgreSQL) or pipe data through the DB interface to any other storage (e.g. https://msdn.microsoft.com/en-us/library/mt143171.aspx) by simply modifying the queries appropriate – as it would be transparent to ZCash. I.e. making ZCash perhaps more readily integrable to various other systems. @radix42 wrote: > One thing I'd like to make note of is that currently all calls to access either database (LevelDB or Berkeley DB) are very much peculiar to the database in use. I'd recommend first factoring out all calls that result in database access into some nice abstraction layer, starting of course with the current databases that are in use. Then safely swapping out the backend(s) will be a whole lot easier. I bloodied my head against this some months ago when I made the start of a switch to lmdb. It was not pretty because of this. @veikkoeeva wrote: > @radix42 I second that too. Which is, in fact, how the aforementioned _Storage Providers_ in the just linked Orleans work. This is repeating the obvious, but for completeness sake, there's an interface and an implementation class and then a loader the loads the implementation. One backend is ADO(.NET), i.e. "relational databases" and that specific backend works as just mentioned. @ebfull wrote: > @nathan-at-least We tentatively believe that sqlite would be a nice replacement for Berkeley DB, and would like you feedback. If we don't hear from you soon, we will assume you agree. @zooko wrote: > [...] there are two different uses of DBs in Zcash — 1. the chain state DB that contains UTXOs, tree states, and stuff like, and 2. the wallet data file. The latter is where we currently use BerkeleyDB, and we definitely want to get rid of it (unreliable, problematic licensing, frequent backward-incompatible releases), and the latter doesn't ([I guess](https://github.com/zcash/zcash/issues/2194#issuecomment-288827202)) need better-than-sqlite performance. > > I also [agree] with what @veikkoeeva and @radix42 are saying above — we should use an abstraction layer: https://github.com/zcash/zcash/issues/2194#issuecomment-289211140, https://github.com/zcash/zcash/issues/2194#issuecomment-289028249 See also #1469, which focuses on licensing issues with BDB; and #1694, an example of a user losing funds apparently due to BDB database corruption.

github.com/zcash/zips

[ZIP 402] Specify database formats with witness-related data in a separate file, no longer storing it in wallet.dat

opened 07:02AM - 19 Nov 16 UTC

bitcartel

ZIP idea ZIP number assigned has ZIP stub

Consider saving witness related data to a file, e.g. witness.dat instead of stor…ing it in wallet.dat. - Separation of data, to isolate and only store keys in wallet.dat. - Performance by avoiding contention between different subsystems trying to write to wallet.dat - Avoid frequent writing of data to wallet.dat on block rescan and reindex. - Reduce risk that when a system crash occurs, wallet.dat is being written to Related, see zcash/zcash#1749

aristarchus · October 19, 2020, 1:31am

I can’t tell from the GitHub discussions if this is definitely happening. Also unclear if all the data files will be changed or just some subset.

I’m curious about this since we don’t currently have great blockchain analysis tools for Zcash and switching to sql might make things easier. Also the current format seems to get corrupted occasionally.

str4d · October 19, 2020, 2:24am

No. The discussions are around changing the wallet database (BDB). The blockchain data (concatenated network bytes) and the chain state (LevelDB) aren’t being considered at the moment.

jmsjsph · October 19, 2020, 3:45am

Like…
Select all transactions from Zcash where fee<0.001;
?

That would be so cool!

daira · October 19, 2020, 11:31am

These would only change the wallet database (currently stored as wallet.dat, not the chain data). There might be some refactoring of what is considered wallet data: it’s likely that we will split long-term keys from data that can be reconstructed.