Low hanging fruit: add REST endpoints to LightwalletD

Technology
1

While it would be best for everyone to run their own node, realistically this is too much of a burden for first time users (that might be only using a smartphone). The lightwallet rpc is essential to serve this use case, but currently it only has a grpc based API. We should lower the barrier to entry to interact with the RPC.

Take a look at the architecture diagram of the Zephyr project that got funded by a Zcash community grant.

image
image1446Ă—871 53.9 KB

The lower part of the diagram could be eliminated if lightwalletD offered REST endpoints.

As you can see months of development time and effort was spent on this issue. To reduce development time and cost in the future it is best for the lightwalletD to expose a REST api.

This is also not a protobuf vs JSON problem. The issue really lies with gRPC.

gRPC’s complex use of HTTP/2 makes it impossible to implement a gRPC client in the browser, instead requiring a proxy.[…]

gRPC is a good choice when connecting servers in a datacenter. But it is more complicated to interact with than a typical REST api. This tradeoff means that there is a higher barrier to entry for developers. It makes some applications impossible or very hard to create.

How the issue can be addressed

The lightwalletd includes an http server anyway for the metrics endpoint:

So we could expose the grpc endpoints as rest endpoints as well.

To give an overview of how lightwalletd endpoints typically work:
For example this function will get a transaction from zcashd, unmarshal the json response and turn it into grpc for the wallet of the zcash user that is connecting to the lightwalletd. lightwalletd/service.go at e3d9c11f14673fe1828fa7dc92ba292586a7c565 · zcash/lightwalletd · GitHub

We can do the exact same thing, but instead of gRPC, expose this as a normal http REST endpoint. The logic stays the same.

The benefit of this is that we don’t need to use gRPC specific tooling that is niche compared to the curl / javascript fetch ecosystem. This means development time and costs get reduced and more experimentation is enabled.

12 Likes
2

I’m sorry but I have to disagree with that. The light walletd protocol is mostly binary in nature and streaming. It is not designed to be exposed to end users. For instance, there is nothing related to wallets or accounts in it, just transactions and blocks.

1 Like
3

thanks for your work on warp sync and ywallet @hanh and all the other projects you have done for the Zcash ecosystem!

That is not what I meant.

4

Well - if you are a dev, it shouldn’t really be a problem. As you said, grpc was designed for machine to machine communication.
Just to be clear, I have nothing against offering more options but I don’t think it will make much difference in the webapp case. Because, the difficulty is elsewhere.
And if you do REST, you have to implement message encoding and streaming.

5

there is no need to do streaming. You can just do polling. Encoding also is not an issue. The problem is just with HTTP/2. Monero works perfectly fine with a REST api.

I bothered to record a short explanation of why grpc is trash:

even as a dev I cant be bothered to deal with some obscure http/2 tooling. Aside from the fact that http/2 is a non starter for the platform I care about anyway.

1 Like
6

The latency on the Monero REST API is so high that for sync, it is barely usable: ~10 blocks / sec.

3 Likes
7

Think there is a large gap in the usage requirements of why X protocol is being used. In the trivial case of any client sending a couple requests over a minute, all of these kinda work. The larger issues, as others mention in a sync, is when a client needs to send many request over some interval. Even HTTP1.1 will cap streams in a browser. A HTTP1.1 client outside of a browser is fine, but HTTP2 is not a myth (sure HTTP2 Push was a joke). They are just setup for different use cases. There are still issues with HTTP3 over QUIC too. Some networks don’t allow UDP etc.

FWIW, GitHub - fullstorydev/grpcui: An interactive web UI for gRPC, along the lines of postman does help manual cmd world of GitHub - fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers. Also some of this can be maneuvered in wireshark too.

8

that just makes zero sense.
For background syncing we are talking blocks / minute anyway.

And catching up with the chain is a bandwidth problem not a latency problem.
using http is simply not a bottleneck.

“I am not going to bother with any of this” is the answer of 99% of all devs. It is just super cumbersome.

9

Well - as someone who has actually written both a zcash wallet and a monero wallet, I can tell you the latency makes a huge difference :slight_smile:

10

I only implemented a Monero wallet and not a Zcash wallet. So I have to call you Senpai now, right?

If latency makes a huge difference, there is a problem with how you implemented it.

We are not implementing a video game here. This is a system that produces new data on the order of “per minute” and not “per second”. So if network latency is an issue, there is just something fundamentally wrong with the specific implementation.

11

So you will be using HTTP 1.1 with the prometheus http server, more or less?

I’m not really sold that gRPC is rejected by 99% of devs, this seems exaggerated. The actual commands in lwd are not that complex. Backend or frontend should be able to handle it (it’s not like no one uses it across other ecosystems). It’s very templated json, but I see you points on the other side.

12

You seem to be talking about a front end to a remote wallet.
How does your Monero wallet implement the restoration from seed & birth height?