I have some crude benchmark data I’d like to share here. I will clean up the docs and produce some more benchmarks next week, but I want to get more information into the community’s hands even if it’s a little rough, right now.
Here goes…
This graph is our performance benchmarks for code derived from the last release (1.2.5):
This graph is our performance benchmarks for code derived from the new ShardTree release (1.3.0):
TL;DR:
In our OLD 1.2.5 release this scenario took just over 40 seconds to complete witness constructing sync for the spend key.
In our current ShardTree 1.3.0 release this scenario took around 5 seconds to complete witness constructing sync for the spend key.
Each graph runs the Code Under Test through the same scenario (syncing 1153 blocks, where each block contains a single block-reward transaction).
In column 1 (blue dots) the client being tested does not have any relevant key (and shouldn’t be affected by a performance upgrade to witness construction).
In column 2 (red dots) the client being tested has a full-viewing key encumbered with the coinbase transaction, and will therefore decrypt the transaction, but will NOT construct a witness to spend.
In colummn 3 the rightmost (green dots) show the performance of a full-key client, that must compute witnesses, after the same decryption step that occurs for the red (view key) case.
The code we wrote to produce these graphs can be found here:
I’d like to clean this presentation a bit (e.g. run 10 replicates per test-case, to show variance etc.)
We HAVE run many 10-replicate benchmarks, but the results were so consistent and the time cost for iterations on development were high enough that we dropped to 2 replicate benchmarks. Relative to the between condition spread, the variance is small, and the findings are very robust across code versions.
Obviously there’s a ton of room for improvement:
- controlling the benchmark environment
- running against more conditions including a matrix of interactions (e.g. device architecture, vs. blockchain state)
- increasing the replicates
- listing summary statistics (e.g. variances)
That having been said the between code version, and between case and control variable results are very stable and have been informally replicated on many developer systems, and across many code versions. We believe the observed performance increase is not an artifact of test conditions, and is user-relevant in the real world.