The original cache design we implemented for zaino was reliant on assumptions that we made when zaino was still scoped to be a lightwalletd replacement. The scope of zaino grew considerably before our local cache implementation began, but the difference between what lightwallets need and what full node wallets need was not fully understood until closer collaboration with the ECC brought it to to light.
To this end, we are designing a new cache, with the following requirements:
- Cache sidechain blocks, instead of only maintaining a copy of the best chain.
- Track all chaintips, with special handling for whichever one is currently the ‘best’ tip.
- Ensure each consumer can get a self-consistent view of cached state, even if a reorg occurs mid-request.*
- This includes the ability to chain together multiple api calls, which all look at a consistent view of cached state, while we allow zaino to continue to sync new blocks during this process.
- Handle many more fields and data types, in order to better serve consumers other than lightclients.
- Structure fields of zaino data types to specify if they are needed for building queries, or stored to avoid making queries to the backend for frequently-needed data.
- This is also a precursor to future work, in that it will enable us to adapt how much data is cached. future options include:
- removing all optional data to minimize disk usage
- caching more fields for faster service when validator bandwidth is a concern (at the cost of more disk use)
- caching more or less data depending on what the zaino instance is expected to be serving
- This is also a precursor to future work, in that it will enable us to adapt how much data is cached. future options include:
*This is currently already somewhat supported, by removing some of the incorrect ‘optimizations’ our cache was relying on, but it is nonetheless another incentive to redesign our cache. (See `zaino-state` is vulnerable to state corruption during reorgs · Issue #295 · zingolabs/zaino · GitHub for the original issue)