Low hanging fruit: add REST endpoints to LightwalletD

[spirobel]

While it would be best for everyone to run their own node, realistically this is too much of a burden for first time users (that might be only using a smartphone). The lightwallet rpc is essential to serve this use case, but currently it only has a grpc based API. We should lower the barrier to entry to interact with the RPC.

Take a look at the architecture diagram of the Zephyr project that got funded by a Zcash community grant.

The lower part of the diagram could be eliminated if lightwalletD offered REST endpoints.

@little.slingshot

• wasm—>js bridge related work (handling gRPC calls)
• debugging RPC calls and Envoy Proxy interaction

@little.slingshot

To bridge HTTP requests and the lightwalletd server and protocol, an Envoy proxy is introduced. Also our wallet-core, executed as WASM module is has to use wasm-->js bridge

to ask browser to perform RPC calls destined at lightwalletd. This results in slightly cumbersome multi-step architecture with lots of intermediaries and moving parts. And that’s where most of our work and research effort was put in the last few months.

As you can see months of development time and effort was spent on this issue. To reduce development time and cost in the future it is best for the lightwalletD to expose a REST api.

This is also not a protobuf vs JSON problem. The issue really lies with gRPC.

gRPC’s complex use of HTTP/2 makes it impossible to implement a gRPC client in the browser, instead requiring a proxy.[…]

gRPC is a good choice when connecting servers in a datacenter. But it is more complicated to interact with than a typical REST api. This tradeoff means that there is a higher barrier to entry for developers. It makes some applications impossible or very hard to create.

How the issue can be addressed

The lightwalletd includes an http server anyway for the metrics endpoint:

zcash/lightwalletd/blob/e3d9c11f14673fe1828fa7dc92ba292586a7c565/cmd/root.go#L422

So we could expose the grpc endpoints as rest endpoints as well.

To give an overview of how lightwalletd endpoints typically work:
For example this function will get a transaction from zcashd, unmarshal the json response and turn it into grpc for the wallet of the zcash user that is connecting to the lightwalletd. lightwalletd/service.go at e3d9c11f14673fe1828fa7dc92ba292586a7c565 · zcash/lightwalletd · GitHub

We can do the exact same thing, but instead of gRPC, expose this as a normal http REST endpoint. The logic stays the same.

The benefit of this is that we don’t need to use gRPC specific tooling that is niche compared to the curl / javascript fetch ecosystem. This means development time and costs get reduced and more experimentation is enabled.

[hanh]

@spirobel

gRPC is a good choice when connecting servers in a datacenter. But it is more complicated to interact with than a typical REST api. This tradeoff means that there is a higher barrier to entry for developers. It makes some applications impossible or very hard to create…

I’m sorry but I have to disagree with that. The light walletd protocol is mostly binary in nature and streaming. It is not designed to be exposed to end users. For instance, there is nothing related to wallets or accounts in it, just transactions and blocks.

[spirobel]

thanks for your work on warp sync and ywallet @hanh and all the other projects you have done for the Zcash ecosystem!

@hanh

It is not designed to be exposed to end users.

That is not what I meant.

@spirobel

This tradeoff means that there is a higher barrier to entry for developers. It makes some applications impossible or very hard to create.

[hanh]

Well - if you are a dev, it shouldn’t really be a problem. As you said, grpc was designed for machine to machine communication.
Just to be clear, I have nothing against offering more options but I don’t think it will make much difference in the webapp case. Because, the difficulty is elsewhere.
And if you do REST, you have to implement message encoding and streaming.

[spirobel]

@hanh

And if you do REST, you have to implement message encoding and streaming.

there is no need to do streaming. You can just do polling. Encoding also is not an issue. The problem is just with HTTP/2. Monero works perfectly fine with a REST api.

I bothered to record a short explanation of why grpc is trash:

grpc is goofy, HTTP/2 is 💩

@hanh

Well - if you are a dev,

even as a dev I cant be bothered to deal with some obscure http/2 tooling. Aside from the fact that http/2 is a non starter for the platform I care about anyway.

[hanh]

@spirobel

Monero works perfectly fine with a REST api.

The latency on the Monero REST API is so high that for sync, it is barely usable: ~10 blocks / sec.

[mdr0id]

Think there is a large gap in the usage requirements of why X protocol is being used. In the trivial case of any client sending a couple requests over a minute, all of these kinda work. The larger issues, as others mention in a sync, is when a client needs to send many request over some interval. Even HTTP1.1 will cap streams in a browser. A HTTP1.1 client outside of a browser is fine, but HTTP2 is not a myth (sure HTTP2 Push was a joke). They are just setup for different use cases. There are still issues with HTTP3 over QUIC too. Some networks don’t allow UDP etc.

FWIW, GitHub - fullstorydev/grpcui: An interactive web UI for gRPC, along the lines of postman does help manual cmd world of GitHub - fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers. Also some of this can be maneuvered in wireshark too.

[spirobel]

@hanh

The latency on the Monero REST API is so high that for sync, it is barely usable: ~10 blocks / sec.

that just makes zero sense.
For background syncing we are talking blocks / minute anyway.

And catching up with the chain is a bandwidth problem not a latency problem.
using http is simply not a bottleneck.

@mdr0id

FWIW, GitHub - fullstorydev/grpcui: An interactive web UI for gRPC, along the lines of postman does help manual cmd world of GitHub - fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers. Also some of this can be maneuvered in wireshark too.

“I am not going to bother with any of this” is the answer of 99% of all devs. It is just super cumbersome.

[hanh]

@spirobel

And catching up with the chain is a bandwidth problem not a latency problem.
using http is simply not a bottleneck.

Well - as someone who has actually written both a zcash wallet and a monero wallet, I can tell you the latency makes a huge difference

[spirobel]

@hanh

Well - as someone who has actually written both a zcash wallet and a monero wallet, I can tell you the latency makes a huge difference

I only implemented a Monero wallet and not a Zcash wallet. So I have to call you Senpai now, right?

If latency makes a huge difference, there is a problem with how you implemented it.

We are not implementing a video game here. This is a system that produces new data on the order of “per minute” and not “per second”. So if network latency is an issue, there is just something fundamentally wrong with the specific implementation.

[mdr0id]

So you will be using HTTP 1.1 with the prometheus http server, more or less?

I’m not really sold that gRPC is rejected by 99% of devs, this seems exaggerated. The actual commands in lwd are not that complex. Backend or frontend should be able to handle it (it’s not like no one uses it across other ecosystems). It’s very templated json, but I see you points on the other side.

[hanh]

You seem to be talking about a front end to a remote wallet.
How does your Monero wallet implement the restoration from seed & birth height?