Zcash + Open Index Protocol Integration

Community Grants
#1

This is a summary

Read the full proposal on GitHub

Zcash + Open Index Protocol Integration

In addition to private finances, to live freely we need public & private spaces online protected by code rather than terms of service.

Applicant background

Hi, we’re Amy & Devon James. We had the idea for using a blockchain to index metadata with file and value transfer addresses to facilitate completely decentralized media distribution in 2014 and built the first implementation with @ryan.taylor (Web Developer, formally at ECC & Livestreamer for Zcash Foundation). Here is a 5 min overview video and some project highlights + links to apps & demos we’ve done since.

Also, some exciting news - NYC property records will be published to OIP, building on our work with Wyoming & Overstock subsidiary Medici Land Governance.

Description of Problem or Opportunity

There are not decentralized public and private spaces on the web.

Information access and communication on the web is dominated by siloed, centralized spaces like YouTube, Reddit & Twitter where it is common practice to censor information, demonetize creators, and prevent people from communicating both privately and publicly.

Proposed Solution

To live freely, we need to be able to speak, share and transact in both private and public. Private finances & private spaces need to be interoperable with public transactions and a public space.

We propose to add interoperability with Zcash to Open Index Protocol (OIP) - initially for private payments to unlock indexed information, and ultimately to create private indexing of information using the Zcash encrypted memo field. In Phase 1 of our work, we’ll also build a Zcash branded YouTube-style web app we’ve been calling “Ztube.” See details on Github.

We’ll also create 4 Zcash-focused videos as part of our What Kind of Internet Do You Want? series

Solution Format

We’ll deliver open source code and documentation to run “Ztube” on a web server, including:

  • “Ztube”, a Zcash branded YouTube style web application, written in html, css & js, using react & redux with a MongoDB User Database.
  • OIP daemon & ElasticSearch DB
  • PIN QT full node & explorer
  • Zcashd full node & Zcash explorer

Videos will be released via:

  • YouTube, Instagram & Ztube (others TBD).
  • Promoted on social media
  • We will also give ZOMG mp4 files of the videos under a Creative Commons license

Technical approach

OIP is a layer two specification for how to index information in a fully decentralized way. It uses the Public Index Network (formerly Florincoin) to store public metadata, initially used BitTorrent & currently uses IPFS for file storage & distribution, and uses BTC, RVN & PIN for payments.

For Phase 1 we propose to 1) add both transparent & shielded Zcash payments to unlock content to the OIP daemon 2) build a Zcash branded web app (Ztube) for content distribution with private payments, and 3) build upon Zcash’s past work with ZEC-BTC atomic swaps to integrate ZEC into a PIN-based DEX so that ZEC token holders can publish content to the public index and avoid additional KYC/AML hurdles involved in joining another exchange.

See details for Phases 1 & 2 on Github.

3614×4428 837 KB

How big of a problem would it be to not solve this problem?

Centralization and censorship on the web threatens privacy, information freedom, and freedom of speech. Online freedom is at stake and losing freedom online could threaten our ability to live freely in our real lives as well.

Execution risks

We don’t have prior experience working with Zcash, so there could be unknowns in implementing shielded payments that cause delays. We’ve allocated time for research, and will draw on the experience of advisor Ryan Taylor as needed.

Unintended Consequences

Public content distribution and private payments could be used by criminals to attempt to distribute illegal content like underage pornography or pirated materials. However, evidence of these crimes will be available to prosecute them because the index is public and indelible.

Evaluation plan

Each milestone has objective metrics for testing software development success and includes one video release.

Schedule and Milestones

Read on Github

Budget

We budget $46,584 per month for 9.5 months. This is based on 3 developers at an average of $11,219 per month, part-time sysadmin at $500 per month, video creator/project manager at $4000 per month, part-time production contractor at $2000 per month, Zcash advisor at $200 per month, and 10% overhead. The total is: $442,548.

5 Likes
#2

My initial thought on this project is that is ambitious, wide-reaching, and would add more use cases for Zcash. The team is obviously enthusiastic and productive. Four Zcash videos alone about What Kind of Internet Do You Want, a video on ZOMG grant process, and interviews with a Zcash person are worth quite a lot considering the paltry presence Zcash has on YouTube, not to mention the other promises of Ztube, etc.

I think more technical people can ask questions about design aspects of the proposal, but this proposal (and team) seems to me to be the type of thing ZOMG and the community should work with to fund.

3 Likes
#3

Thanks for the support! I’m working on a Q&A style video for What Kind of Internet Do You Want? next - thinking about what questions to cover in the episode. Would love suggestions if anyone has any :slight_smile:

3 Likes
#4

Would love to chat and collaborate! We are working on a video series as well that could synergize great with you guys. Zcash Mini-Documentary & Educational Series - #11 by 37L

3 Likes
#5

Hi, the project sounds interesting. I’ve been in the YouTube space for over a decade, and have seen a number of attempts to create a YouTube competitor; there are definitely a lot of challenges in creating a competitor project to YouTube.

I have a couple questions re the “Ztube” app. The biggest problem in the video space is the cost of storage, bandwidth and content delivery networks. Is there any user testing or research that you’ve done that could address this difficulty? Funding video storage/bandwidth via ads is considered difficult, and for people to pay for content there would need to be a good amount of quality videos (if a subscription system is used), or a pay as you go system with excellent UX and a Creator outreach program. Video is definitely an interesting application for IPFS, but I believe that issues could come up when it comes to scaling and with metadata leakage (rest-time encryption/decryption for video streaming could be tricky, and network traffic together with IP addresses would likely be semi-public).

In terms of technical architecture, would it work as a federated network, or a single centralised instance of a MongoDB database and ElasticSearch index? Or would the MongoDB/ElasticSearch setup act as caches, pulling data from the Zcash blockchain, similar to ZECpages?

Also, are you aware of Twitter’s Bluesky project? They are exploring the use of IPFS and may end up working on some kind of video integration/support.

What would differentiate Open Index Protocol Integration from existing applications/protocols like Odysee/LBRY? I know that many people have been moving from YouTube to Odysee. Could there be interoperability with PeerTube and ActivityPub?

2 Likes
#6

Would absolutely love to! I used to work in theatre and miss collaborating with others so much!

#7

Great questions, thank you!

I’ve been in the YouTube space for over a decade, and have seen a number of attempts to create a YouTube competitor; there are definitely a lot of challenges in creating a competitor project to YouTube.

  • Thats awesome! Whats your channel?
  • We don’t think of Ztube as a competitor to YouTube - it wouldn’t be reasonable for us to take on the biggest video platform on the web. Rather, think of YouTube as Goliath and think of Ztube & the many other versions of it that will be built all using the same protocols as an army of Davids. We can’t take them on alone - we need each other to bring freedom back to the web.

I have a couple questions re the “Ztube” app. The biggest problem in the video space is the cost of storage, bandwidth and content delivery networks. Is there any user testing or research that you’ve done that could address this difficulty? I believe that issues could come up when it comes to scaling and with metadata leakage (rest-time encryption/decryption for video streaming could be tricky, and network traffic together with IP addresses would likely be semi-public).

A few thoughts…

  • In a way this would be a good problem! Large bandwidth requirements would mean that there is a large user base which would bring value to the platform and thus offset the costs.
  • The first version of Ztube will not have livestreaming capability (meaning the bandwidth requirement will be significantly lower)
  • In terms of research, we’ve hosted versions of video sites as we’ve been developing over the years and the bills have been very low. Typically when using a hosting service, they allow unlimited bandwidth up to a certain peak amount - and the cost for the service is extremely reasonable.
  • Anecdotally I can say that we’ve met with the largest streaming service and several studios and when we mentioned that this tech could save money on their storage & distro they shrugged and said it wasn’t a significant cost, that the content itself is the expensive part. This isn’t the same because Ztube won’t be producing content, but it may want to invest in marketing to help onboard users. However it puts the cost in perspective, perhaps if Ztube is very successful it would ultimately have a marketing budget that exceeds these costs, but nonetheless revenue would exceed it.
  • Finally, we’re also looking at using either webtorrent or js-ipfs to let end users opt into sharing their upload bandwidth. Your concern about their address being attached to the content addresses would be applicable here, which is why it would be opt in, and a VPN can generally address this concern.

In terms of technical architecture, would it work as a federated network, or a single centralised instance of a MongoDB database and ElasticSearch index? Or would the MongoDB/ElasticSearch setup act as caches, pulling data from the Zcash blockchain, similar to ZECpages?

  • The MongoDB is the user registration db for this app specifically so it would be driven by the web app
  • The elasticsearch database is a cache of data from the blockchain - index info from the Public Index Network blockchain, and payment info from other chains like Zcash - in the next stage of our plan it will pull index information also from zcash encrypted messages on shielded transactions, extending the capability to private distribution.

Also, are you aware of Twitter’s Bluesky project? They are exploring the use of IPFS and may end up working on some kind of video integration/support.

  • Yeah Bluesky is exciting! We’ve talked with them and Bitspill is in their group chat.

What would differentiate Open Index Protocol Integration from existing applications/protocols like Odysee/LBRY? I know that many people have been moving from YouTube to Odysee.

  • Odysee/LBRY is proving that users are looking for new options, and we congratulate them on how many users they have brought into the decentralized ecosystem!
  • The main difference between LBRY & OIP is interoperability. LBRY does everything in house - they built their blockchain, they built their own storage & distro protocol, content unlocks are limited to the LBC token only - each of these are decentralized and that is great, but because they don’t use interoperable open protocols, they are completely responsible for development. Also, there is no direct financial incentive for other apps to build on their protocol other than holding LBC tokens and hoping they go up. OIP didn’t build the PIN blockchain, uses many storage/distro protocols (BitTorrent, IPFS, etc), can use anything to pay for content unlocks (BTC, et al) and has built in incentive for applications to use it due to the Salutary Protocol incentive design which creates opportunity for value capture at both the protocol layer and the application layer via providing services which creates incentive for developers to build apps with it.
  • I don’t mean to pick on them but there are a couple specific issues with the LBRY protocol like that are concerning - anyone can have their channel name taken at anytime via an auction, and they are under investigation with the SEC for their token sale.
  • Livestreaming on Odysee is accomplished via RTMP streaming, so while its not in our plan for the initial release of Ztube, it’s totally feasible to add it in a reasonably short amount of time.
  • Also, Odysee/LBRY could use OIP for indexing content that is published there just like other siloed services.

Could there be interoperability with PeerTube and ActivityPub?

  • PeerTube quite probably yes. Activitypub not very easily as they are different CRUD mechanisms so would have some complicating redundancy (kind of like trying to make Polkadot & Interledger interoperable - possible but difficult, and one would have to adopt the others technical design approach to do it)
1 Like
#8

Thanks! The videos are quite old, but I somehow managed to get the top search result for how to use ms dos :sweat_smile: .

Are there any specific hosting services you’ve tried? AWS is really popular for video, but they’re quite pricey, especially when it comes to bandwidth, and there’s a lot of problems in terms of centralisation and potential for censorship, i.e. ability to automatically delete videos with machine learning. I personally would probably go with OVH (I believe they offer unlimited bandwidth for video streaming now) or CloudFlare Stream (they require their video player to be used though for billing purposes).

I think the main concern is that if a video goes viral, bandwidth costs can spiral pretty quickly. A 500MB video watched fully by 1,000 people is 500GB of bandwidth, which is $42 of bandwidth with AWS CloudFront.

Hmm, a DID or Zcash authentication system could be interesting to explore for registration in the future, might not be mature enough to use currently though.

:+1: great!

I’m wondering how usable ActivityPub extensions would be for video. I think just having ActivityPub compatible OAuth2 could be interesting for federation; as far as I can tell, it should be possible to pick and choose parts of the specification.

#9

Thanks! The videos are quite old, but I somehow managed to get the top search result for how to use ms dos :sweat_smile: .

So cool! I’m hoping that will happen with our What Kind of Internet Do You Want? series since there is still so little content on how blockchain & Web 3 will change the internet. I recently released a video about Porn & Blockchain that has a deep fake of me and talks briefly about Zcash, hope it becomes the top search result for that someday haha

Are there any specific hosting services you’ve tried? AWS is really popular for video, but they’re quite pricey, especially when it comes to bandwidth, and there’s a lot of problems in terms of centralisation and potential for censorship, i.e. ability to automatically delete videos with machine learning. I personally would probably go with OVH (I believe they offer unlimited bandwidth for video streaming now) or CloudFlare Stream (they require their video player to be used though for billing purposes).

I think the main concern is that if a video goes viral, bandwidth costs can spiral pretty quickly. A 500MB video watched fully by 1,000 people is 500GB of bandwidth, which is $42 of bandwidth with AWS CloudFront.

We use OVH (plus associates, SoYouStart & Kimsufi), and Hetzner. Sounds like you are already well versed in this but for anyone else reading who, like me, might not have these kind of details top of mind I reached out to our sysadmin and this is what he said:

“If your question is the cost of a 500MB video watched 1000 times on OVH - the answer would simply be $0. But instead you should consider the difference of how a file served by AWS vs OVH would handle during high demand.

On OVH, it’s typically up to you to serve the file in a reliable way, but with AWS depending on the service you leverage you can have them do it for you. So on OVH in some cases you’d pay for extra nodes to keep it online.”

Hmm, a DID or Zcash authentication system could be interesting to explore for registration in the future, might not be mature enough to use currently though.

To answer this let me draw a distinction between two different kinds of “users”. A user who can publish content to OIP is called a “publisher”, and their ability to publish is controlled at the protocol layer, with the requirement simply being that they create a publisher registration message before they publish other kinds of records. This message format has changed somewhat over the years, but it’s primarily focused on the publishers public key associated with the private key that they plan to use to sign their records with.

If you’re familiar with the specifics of DIDs, you probably recognize that OIP publisher address registration is an example of the DID approach, this is out of pure coincidence, we developed it a few years before the DID spec existed but we’re excited to formally apply it, and yes this project is a great opportunity to do so.

Publishers do not need to provide an email address or anything else in order for their records to be valid within OIP. However, at the application layer, users are a little different - they tend to forget their passwords and want to then use their email account to reset it, so within the DDX reference client we have a registration and login system built around our HDMW. Registering there just adds a new entry to a local database, not in a blockchain. However, since this login gets the user access to an HDMW wallet, and they’d then use that to register a Publisher within OIP at the protocol level, they are related to one another, but I just wanted to make it clear because your questions apply to them each separately - DIDs we can apply at the protocol level by updating our publisher registration message format slightly so that it fully conforms to the spec, and the zcash auth system would be looked at as a way to upgrade our somewhat traditional JWT web-server auth system for providing users easy and secure access to their ability to publish. That said, we’re not yet familiar with the zcash authentication system, so we’ll need investigate to find out if it is mature enough to use yet.

I’m wondering how usable ActivityPub extensions would be for video. I think just having ActivityPub compatible OAuth2 could be interesting for federation; as far as I can tell, it should be possible to pick and choose parts of the specification.

I don’t think I understand what you mean here - are you saying it would give the application access to a library of videos published using ActivityPub in some federated way or something else?

1 Like
#10

I skimmed through your video, I liked it :slightly_smiling_face: . Re the OnlyFans story, this is a really good video on it (by someone who has struggled with YouTube “censorship”), which talks about the lack of human connection between platforms and creators: youtu.be/WRT9WucVaB0 . I think platform admin and creator communication needs to be built into protocols and apps.

I think 10 years ago (for me at least) it came down to luck and video title. Now that the YouTube algorithm is based on watch time, together with the platform being highly saturated, it’s much harder. I’d say the key now is consistency: daily/weekly uploads of quality content with high quality production (multiple camera angles). For long podcast style videos, uploading additional Joe Rogan style snippet videos are best imo: cutting 4 minute chunks from a 30min video; audience retention is key now. Clickbait title (without being misleading) and good thumbnail are important too. YouTube is the second biggest search engine in the world, but it’s pretty useless imo. Video quantity (i.e. upload 500 videos, and some will become popular) is needed to build up a recommendation score, the algorithm will ignore you if you don’t upload a video at least once a week (from my research at least).

Ah, great! Apparently part of the reason OVH is such good value is that they’re subsidised by the French government, to make sure Europe is competitive in the server space :smiley: . SoYouStart & Kimsufi use old hardware, so need to be extra careful to make sure there’s automated failover & backups in the case of a hardware failure.

Yeah. I’m tempted to experiment with a hybrid OVH dedi and DigitalOcean CDN approach for video streaming, to get good peak traffic management and edge caching.

I’m not familiar with the DID spec yet, sounds good though.

I don’t think that’s something that exists yet, but it might be possible in the future with the Zcash Metamask like browser extension in development. The idea would be to use the RPC command signmessage as some kind of UI that proves ownership of private keys for a transparent address. A lot of UX work would need to go into it, like wallet integration.

PeerTube uses the Mastodon specification in a way that a Mastodon user can comment on a PeerTube video, since they use the same OAuth2 APIs.

#11

Here is a link to the proposal that is open for public review - ZF Grants - Zcash + Open Index Protocol Integration.

#12

Hi Amy and Devon,

Building large online libraries using decentralized tech is something I care about a lot, so I’m sorry to say that we decided not to fund your proposal.

The thing that made it hard for us was the combination of:

  1. The high budget for this project relative to other grants (our second highest so far, I believe)
  2. The significant risk of this functionality not finding a market of eager users (a risk any user-facing product faces)
  3. The fact that, if successful, the benefits aren’t to the core Zcash tools and technologies… it would be more along the lines of “now people can do another cool thing with Zcash”.

All of these individually are potentially okay, but together the risk/reward balance seemed out of range for us.

We all wish you luck in your project, as do many folks in the Zcash community. We’d welcome proposals that avoid these pitfalls, like proposals focused more narrowly on content, or proposals that are explicitly building small proofs of concept and have commensurately smaller budgets.

I hope this makes sense!

2 Likes
#13

Point 3 seems more like a good thing than a bad thing! We need cool things to do with Zcash :slight_smile:

I suppose I would urge the applicants to break up some of the grant into bite size pieces and reapply! That could help with Point 1 concerns, along with what Holmes alluded to about proof of concept work.

Perhaps some market research about potential users could help assess the risk of it finding a user base to address point 2.

1 Like
#14

:slight_smile: I agree!

I guess it’s more that it comes in second place to tools that are directly bringing financial privacy and freedom, when it comes to budget or priority.

So for example, we would stretch to make a large grant for a wallet, or for a “ZUSD” stablecoin, or for tech that made Zcash more private (like the Arti grant) but it’s harder to justify a very large grant for the ability to use Zcash somewhere.

And we would give a smaller grant to someone doing something cool with Zcash, but not one of our largest grants.

But I definitely didn’t mean to say that “making more cool things to do with Zcash” is not a fundable category. It totally is.

3 Likes
#15

^^A reasonable explanation that may help not only these applicants with future proposals, but others in the space as well.

I’m glad these proposals and responses are public. They can really be used for applicants to either improve their grants or for new applicants to look back on and avoid the pitfalls of previously rejected proposals as well as identify the strengths of approved grants.

6 Likes
#16

Yes! This is why we like to give thoughtful replies.

5 Likes
#17

Hi Holmes & ZOMG judges,

Bummed of course but thanks for detailing your thinking!

We wanted to work with you because we like the Zcash ethos of privacy & freedom and this seemed like a project that would be fun & meaningfully expand our shared goals of freedom on the internet. We appreciate you saying you’d welcome a new proposal but are weighting whether to invest more time and have a couple questions -

  1. Sounds like the biggest issue is that its not core to Zcash tools and technologies. We wondered if these grants were intended only for Zcash developer tools based on what had been previously funded, but had hoped using Zcash for privately consuming or publishing content and information would be considered core to the mission in the white paper of building technology that enables us to take back our freedom and live freely. I’m confused by point #3 that “now people can do another cool thing with Zcash”… does this mean you are focused exclusively on financial privacy and not interested in other ways that Zcash can be used for privacy like content distribution or private indexing of information?

(Background on this question: The idea for adding Zcash to the OIP spec as a place to index content (rather than just using Zcash to make payments for content) came to me when I was on the community call and noticed that ZF was using Google Sheets to organize survey information, for a privacy-first company I was surprised, but also found it understandable because I’d recently been lovingly reprimanded by my friend Dr. Robert Epstein, a well known technology & privacy expert, on my own weakness for the google crack haha - I kept thinking about it and realized that we could build the same kind of document sharing product with Zcash, with the ability to have private & collaborative documents. This would obviously be a large undertaking so we proposed the foundational work that would be necessary, thinking that either our team or another team could expand on it from there).

  1. Before we applied I attended a community call and spent time researching on the forums and heard ZOMG judges say that you would get on the phone with applicants or reach out to them with questions, but that didn’t happen in our case, so we’re a bit worried that your feedback that we can reapply is more of a kind wrapper around a rejection than a genuine suggestion - would you be able to share whether the main objection was more about the nature of the project or the price of the proposal? In other words, are we simply too far outside the scope of what you are interested in funding at this time? Or do you feel that if we scale back the proposal and cut the budget to a functioning proof of concept without a polished user interface that it would be something you’d fund?

Lastly, in regards to your second question wondering whether there’d be sufficient demand for it, we’re confident we could make the case that there definitely is with some metrics, stats and articles but we just wanted to make sure we were proposing something that your community would be interested first before compiling that info.

Thanks very much!

1 Like
#18

I completely agree with your comments about what is necessary to grow on YouTube these days! We’re actually in the midst of working on changing our filming studio to make it sustainable for me to create content consistently. Goal is 1/week to start but who knows maybe we’ll work up to daily! Also working out the production systems to make the clips videos which I think will help with growth as well. I’ve appreciated your thoughtful comments & hope you’ll subscribe to the channel and continue to share your feedback!

Ah, now I get what you are saying about ActivityPub & agree it would definitely be something to look into further! :slight_smile:

1 Like
#19

Before investing too much time and resources into YouTube, it’s worth having some time to think about what you want to get out of it, and a strategy on maximising ROI and tracking analytics. YouTube used to have keyword tracking tools, but I believe they removed them. Some ideas/advice on YouTube SEO here: 6 Keyword Tools for Better Video Ideas and SEO - YouTube. LinusTechTips is a good model to follow in terms of content: I especially like the formats of Techquickie and TechLinked, they’re fairly short and engaging. The main thing YouTube has which other platforms don’t is high visibility and discoverability. If the algorithm decides to promote you, a lot of people can find your content, but the platform is highly saturated, like Instagram. Weekly videos are probably the best balance between quantity/quality.

Great, I’ve subscribed :slight_smile: Feel free to contact me here, on Matrix (@1337bytes:matrix.org) or Twitter; will be happy to share more feedback.

Am sorry that the proposal didn’t get funded. I’m passionate about video sharing platforms and have wanted to develop a video platform for a long time; I’ve realised though that it would be a lot of work to do myself and my main hope now is to see some kind of open standard and effort emerge that will replace YouTube and empower creators. It saddens me seeing how many people have built careers off YouTube only to be abandoned by the platform that profited off them, with no recourse when they get demonetised or incorrectly banned.

I’ve been working on an open-source modular social media component system lately and am keen on helping projects use it; if you’d be interested in it and you go ahead in the future with developing a video platform app, or similar project with OIP, I’d be happy to provide UI components. I believe UI and UX are the main hurdles in creating something that competes with platforms like YouTube/Instagram.

2 Likes
#20

@holmesworcester hope you see this, not sure how the forum notification system works with replies. Thanks for sharing your initial feedback & would appreciate your thoughts on our questions if you don’t mind :slight_smile: