Arti: A pure-Rust Tor Implementation for Zcash and beyond

holmesworcester · April 30, 2021, 12:37pm

This makes sense to me and is helpful for our decision!

We can! And we’re waiting to hear back now from the various teams to make sure that Arti integration will be on their roadmap.

teor · May 4, 2021, 12:22am

nickm_tor:

teor:

Most wallets just use DNS, and the Zebra and zcashd nodes use generic IP address objects. But the Tor protocol still makes quite a few internal distinctions between IPv4 and IPv6. So IPv6 support isn’t automatic.

Given this need, here’s what I see as the best way forward. I think we should move minimal IPv6-only client support forward from 1.0.0 into 0.0.1, but leave complete support for 1.0.0. This way integrators and beta testers can run on IPv6-only networks from the beginning.

By “minimal IPv6-only client support”, I mean that clients that only have IPv6 should be able to bootstrap and connect to the Tor network and work reasonably okay. (This would probably involve a not-very-optimized ‘happy eyeballs’ style approach.) Unless I’m wrong, doing this part won’t add more than a day or two to our 0.0.1 estimates.

…

How does that sound?

I remember writing a few happy-eyeballs proposals for Tor.

Now there are more Tor relays and authorities on IPv6, happy eyeballs should be efficient enough for testing. (And much easier to implement than in the legacy Tor codebase.)

nickm_tor:

teor:

Looking at the milestones and tickets, it seems like Arti is aiming for a similar async architecture to Zebra:

That’s right. We’ve been trying to structure our crates into (roughly) three tiers with respect to async-ness:

The lowest level of code doesn’t know about async-ness at all. It’s just straightforward blocking code.[^*]

The middle tier knows about async programming and futures, but nothing about specific async runtimes. Because of that, it can’t spawn tasks, listen, launch network connections, sleep, etc.

The top tier knows about runtimes, as abstracted through a compatibility crate.

[*]: To make it easier to add asyncness later on, we require that expensive cryptographic operations are in different function calls from everything else though. For example, signature checking and parsing are always separate calls.

teor:

The Zebra team is happy to do an architecture walkthrough if you’d like. We’re also working on a list of common async Rust pitfalls.

Both of those sound excellent to me. I’d also love to bring Zebra devs in on an Arti walkthrough some time, if you’re interested. (I don’t expect I’d teach you much, but you would probably spot areas where we could do better.)

That would be great! Let’s wait on any grant decision, and then do the walkthroughs soon after.

We also just merged our “Async Rust in Zebra” documentation today:

github.com

ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0011-async-rust-in-zebra.md#summary

- Feature Name: async_rust_in_zebra
- Start Date: 2021-03-30
- Design PR: [ZcashFoundation/zebra#1965](https://github.com/ZcashFoundation/zebra/pull/1965)
- Zebra Issue: [ZcashFoundation/zebra#1593](https://github.com/ZcashFoundation/zebra/issues/1593)

# Summary
[summary]: #summary

Zebra programmers need to carefully write async code so it doesn't deadlock or hang.
This is particularly important for `poll`, `select`, `Buffer`, `Batch`, and `Mutex`.

Zebra executes concurrent tasks using [async Rust](https://rust-lang.github.io/async-book/),
with the [tokio](https://docs.rs/tokio/) executor.

At a higher level, Zebra also uses [`tower::Service`s](https://docs.rs/tower/0.4.1/tower/trait.Service.html),
[`tower::Buffer`s](https://docs.rs/tower/0.4.1/tower/buffer/struct.Buffer.html),
and our own [`tower-batch`](https://github.com/ZcashFoundation/zebra/tree/main/tower-batch)
implementation.

# Motivation

This file has been truncated. show original

The difference in priority between the select() function and select! { } macro was particularly surprising to us.

teor · May 5, 2021, 11:33pm

Here are some other things I’ve been thinking about as I’ve looked over arti’s milestones:

There’s an experienced Rust developer review as part of the 1.0.0 release. By that time, your architecture and data design will be pretty fixed - can you get an experienced Rust dev to look at them before your APIs stabilise? (0.0.1 or 0.1.0)

The Zebra team has had a similar lock-in experience in some of our async Rust code. For example, we wish we had made our std::thread::Mutex<AddressBook> into a tower::Service much earlier. Because now we’ve built on top of the mutex abstraction, it’s going to take more work to make all its callers async. (And it’s exposed in the API of the zebra-network crate, but it really shouldn’t be. More refactor work!)

But in general, we’ve really benefited from having experienced Rust developers work on our designs, implementations, and code reviews. I’ve definitely learned a lot!

I also wonder if the security audit cost is potentially a bit low for a protocol as complex as Tor? (That said, there will obviously be legacy features that you will never implement in arti.)

And thinking about timeframes for that audit: it might be helpful for the arti security audit to be done around the time Zebra has its first audited client release. But we haven’t done detailed planning for our client work yet, so I don’t have any specific timeframes for Zebra. And I suspect zcashd’s requirements will be the real driver here, because they’re in production already.

nickm_tor · May 12, 2021, 7:53pm

Hi again, @teor, and sorry for the delay. My schedule has been wild and I’ve been more distracted than useful.

Can you get an experienced Rust dev to look at them before your APIs stabilise? (0.0.1 or 0.1.0)

Yes; we’re hoping to actually start getting our API feedback as soon as we can, especially for developer-visible APIs.

Incidentally, as for Tower stuff, I’m hoping you or somebody will have time someday to introduce us to the why of Tower in a system like ours. I’m getting a sense what Tower does and what its APIs are for, but not a great sense of where Tower would logically fit into Arti. (Maybe there’s a part of Zebra I should look at, in order to get the key idea here?)

teor · May 14, 2021, 3:24am

Zebra is designed as a series of microservices: A New Network Stack For Zcash - zcash foundation

One of the benefits of Tower is that Services are composable with Layers. So it’s very easy to wrap a network service in a hedge, timeout, concurrency limit, or buffer. And if you’ve used Rust generics for your service types, the rest of your code doesn’t need to change at all.

Zebra uses about half the layers listed here: tower - Rust

And we also have our own tower-based batch verifiers for zero-knowledge proofs and signatures: Composable Futures-based Batch Verification - zcash foundation

A similar design might be helpful for arti - you could have:

a directory microservice,
a guard-set microservice for guard connections (like tower-discover),
a microservice that handle circuits and cell decryption, and
an ed25519 batch verification service (like Zebra!)

nickm_tor · June 4, 2021, 10:12pm

A small update that might be of interest, as a proof-of-concept:

We’ve heard from a group of researchers who had the need to integrate anonymous communications into a contact tracing app that’s going to be used by several million users. The addition of anonymous communications would enable the privacy-preserving collection of usage statistics, key to assess the performance of the app. It would also enhance the security and privacy of new functionalities.

They decided to use the Tor network to safely route communications without deploying any new infrastructure. They decided to use Arti because—for their particular use case and threat model—the security features that Arti currently lacks are not necessary.

They’ve experimented with adding Arti support and they report:

Arti worked for them out-of-the-box on Android and iOS
They haven’t had trouble integrating it or making it work for them in practice.
There is a fairly good chance that the app version using Arti will be soon in production.

They don’t want to go public yet, since the feature is a work in progress, but they are happy to be contacted to answer questions about their use case and plans, and how further development of Arti would help their projects. But please ping me if you’d like me to put you in touch with them to discuss their experience.

Shawn · June 8, 2021, 10:18am

@nickm_tor — great news! ZOMG is pleased to announce that “Arti: A pure rust Tor Implementation for Zcash and Beyond” has been approved to Milestone 1.1.0!

Funding to milestone 1.1.0 is approximately $670,000. With this grant, we’re going to begin offering some projects a choice between fixing the values of milestone payouts in ZEC or USD.
If you choose USD, you’ll receive exactly the USD values you proposed. If you choose ZEC, the ZEC amounts will be determined at today’s market prices, and you’ll receive those ZEC amounts for your milestones as they are met whether the market value of ZEC goes up or down. Given our understanding of Tor’s structure and needs, and given the size of the grant, we assume you prefer to fix the values of milestone payments in USD. If so, can you confirm this? Thanks!!

(Note for others following this discussion: we’re working on a separate post to explain why we came to this decision about ZEC vs USD milestones and what the caveats will be, and that will be posted soon! The short version is that we realized that in our current process ZOMG could potentially have become unable to meet its commitments, so we’re changing our process to address that.)

Please feel free to DM us @ZcashGrants so we can schedule a meeting to discuss next steps and answer any questions. We will also need to revise your existing grant proposal on the grant’s platform before we can proceed with approval there.

Congratulations!! We look forward to seeing Arti and the impact it can have on Zcash users network level privacy!

Shawn · June 8, 2021, 10:32am

Also, we have reason to believe that this project may receive media coverage.

Our press release for any inquiries:

tokidoki · June 8, 2021, 12:10pm

Congrats @nickm_tor and all people at Tor

Happy to be a part of community who cares about the future of privacy for all

https://twitter.com/ZcashID/status/1402235896649293825

Shawn · June 8, 2021, 1:17pm

extrapo · June 10, 2021, 6:23pm

Congrats for the Tor Team!

Heavy Tor user here, 5 years ago on the CaosComputerCamp I gave a workshop on making a crawler for Hidden Services, also generating some DataVizualizations… Was very lucky to meet some of you with Gabriella Coleman in the train back to Berlin… lots of intellectual generosity…

Long Live to Tor!

https://events.ccc.de/camp/2015/wiki/Session:Deep_Graphics

bdavila · November 9, 2021, 4:48pm

On October 29th we launched the first release of Arti, v0.0.1 [0]. This release is ready for general testing, but not yet ready for production use. You can read about the features and changes in the changelog for this release [1][2…8].

bdavila · December 1, 2021, 7:04pm

Arti 0.0.2 is released: API groundwork, refactoring, config, and optimism

teor · December 28, 2021, 3:35am

I recently spent about a day adding Tor support to Zebra’s isolated connection API using arti-client.

When Zebra adds support for sending user-generated transactions, we can use this API to make sure they aren’t linked to the sender’s IP address. (Or to any other transactions we send.)

Overall the process went really smoothly. The API was easy to understand, the code is short and easy to read, and basic tests seem to work. (I wasn’t able to do comprehensive end-to-end tests, because Zebra doesn’t support creating transactions yet.)

Here’s some detailed feedback about some tricky parts of the integration.

Dependency Versions

The dependency versions that arti requires seem quite strict for a library crate. We needed to do a few dependency upgrades for Zebra as part of integration. These upgrades were simple using cargo upgrade. But sometimes dependency upgrades might need code changes.

Can arti depend on the earliest supported minor release of each dependency in Cargo.toml?
You can use Cargo.lock to specify exact dependencies for binary crates.

Dependency Troubleshooting

It would be helpful to mention:

installing pkg-config, and
setting OPENSSL_DIR and SQLITE3_LIB_DIR for troubleshooting header or linker errors.

I also needed to set LD_LIBRARY_PATH, but that’s probably a Nix derivation bug.

The troubleshooting doc was a bit hard to find, maybe it could be linked from the readme.

Useful APIs

There are a few missing APIs that would help make our code more efficient, and might reduce our load on the Tor network:

An API that bootstraps Tor if needed, then returns a cloned Tor client

Making a shared TorClient instance was a bit tricky, because it’s hard to lazily call async code in Rust, and then share the result. So I ended up wrapping the shared value in an Arc<Mutex<_>>. But this seems like a bit of a waste, because there’s already a lot of locking inside TorClient.

Here are some API changes that might have made that easier:

an API or example code that initializes a shared TorClient when it is first used, then clones the shared instance for future requests
a non-async API that configures and allocates a shared TorClient, then bootstraps it when it actually gets its first request (or when an async bootstrap method gets called)
if possible, making arti_client::Error cloneable, so arti futures be used with FutureExt::shared

A TorClient config that makes sure all connections are isolated

In Zebra, we don’t plan on sending blockchain blocks over Tor, because it’s all public information. So we’ll just be occasionally sending new user-generated transactions over Tor.

So it would be helpful to have a TorClient config that guarantees all connections are isolated. It might also save some preemptive circuit load on the network. (Unless all preemptive circuits are guaranteed to be isolated already - I can’t remember.)

But I can understand if you don’t want to expose this as a configurable client mode. Because it will place more load on the network.

Minor API nitpicks

arti_client::Result is a useful type alias, it would be nice for it to be available outside the arti crates. That way, our tor-specific APIs don’t have to use BoxError everywhere.

It seems like the stream timeout config builder isn’t exposed by the client config builder. But we didn’t need it anyway.

In Zebra we’ll probably end up creating a tower::Service that takes hostnames, and returns arti_client::DataStreams. So we might not actually need any of these API changes - we can do it all in the service. Cloneable errors would be nice though!

nickm_tor · December 29, 2021, 5:55pm

Thanks for the detailed feedback, @teor! So you know, we’re on vacation till 6 January, but once we’re back we’ll be opening a bunch of tickets for these issues, and probably asking some questions too. I want to give this feedback a high priority in January. (And a happy new year to you!)

teor · December 30, 2021, 2:11am

Thanks @nickm_tor!

Just letting you know that I’m taking some leave in January. I can answer questions once I’m back.

I’m happy to answer questions on individual tickets on your bug tracker. It would be hard to track all those details on a single forum thread. (But we should post the ticket links here, so interested people can follow along.)

nickm_tor · January 10, 2022, 1:56pm

Thanks, @teor! This is really helpful. Below I’ll link to the tickets we’ve opened on our tracking system, and include any questions or comments we have for you.

This is arti#275 and arti!228, now merged. One question we had here, though, is why cargo upgrade was necessary on your end. (Were you using earlier versions that weren’t forward-compatible with the ones that Arti required? Do you happen to remember which crates were trouble?)

This all sounds like a good idea; it’s now arti#277.

Agreed that this is a good idea. The cloneable-Error issue falls under arti#247; I’ve opened arti#278 for the rest.

I think we should try to find a way to provide this feature without too much risk of people turning it unnecessarily. I’ve opened arti#279 for this.

(As an aside: preemptive circuits are treated normally with respect to circuit isolation. That is, two isolated streams can be placed on the same preemptive circuit iff those streams can share a circuit.)

Agreed; this is now arti#280.

[qoute]
It seems like the stream timeout config builder isn’t exposed by the client config builder. But we didn’t need it anyway.
[/quote]

Thanks; that omission sounds like an oversight. It’s now arti#281

Thanks for the feedback: It’s exactly the kind of thing we were looking for at this stage, and it’s been extremely helpful Even if Zebra does wind up doing all of this with a [tower::Service], making these changes will probably help other projects as well.

teor · January 25, 2022, 12:38am

In 2021 we paused our dependency upgrades for a few months while we focused on implementing the Zcash consensus rules. We also delayed some upgrades until the rest of the ecosystem caught up, because we wanted all our dependencies to use the same version of common crates.

So sometimes we run behind the latest crate versions by 3-12 months.

We also try to minimise the number of duplicate crate versions we have. I wrote a detailed comment on the arti issue explaining our reasons.

Here are the exact lists of dependencies we upgraded at each stage:

Dependencies that Zebra uses even when arti is turned off:

upgraded bytes 0.5.6 via http 0.2.1, this got rid of some duplicate dependencies
upgraded a bunch of proc macro crates, more duplicate reduction

I’m not sure if there were any specific incompatibilities here. But bytes has had some in the past.

The rest of the version changes probably aren’t that interesting. But I’m listing them just in case.

Arti-only dependencies, after unification with Zebra’s existing dependencies.

Version changes when we moved from a specific commit to the tagged 0.0.2 release.

Version changes since I opened the PR at the end of December.

Shawn · February 2, 2022, 2:45am

anon35140610 · February 2, 2022, 2:58am

I love that they always post blog updates and that at the end of the updates, they always give a shoutout to ZOMG / Zcash Community Grants. It’s a good way to bring exposure to the program.

Topic		Replies	Views
Arti Year II Funding Plan: Onion Services and more! Community Grants	36	3124	October 3, 2024
Proposal review: Arti, a project to rewrite Tor in Rust: onion services and beyond! Applications	6	911	September 9, 2022
Arti, a Rust Tor client: 0.0.1 released! Ecosystem Updates	0	635	October 29, 2021
Public Review, Application: Arti: a pure-Rust Tor implementation Applications	2	608	April 28, 2021
Zcash on Tor : A Community Funded Proposal Community Collaborations	29	13457	September 20, 2017

Arti: A pure-Rust Tor Implementation for Zcash and beyond

Related topics