What is this developer kit
and why are you giving it away for free?
The ConsenSys Web3 Studio in Durham, NC has a pretty sweet job. Our team of developers, designers and writers spend our days coming up with what we hope are novel, unexpected ways to use blockchain and Web3-related technologies.
We publish stories about these uses in the form of “product reviews from the near future” on Medium . Then we build some of these stories into developer kits like this one. The kits are open source.
Success for us is you taking the kit’s ideas and open source code and building your own web3 applications.
What is SoJourn?
SoJourn is a really simple example of what you can do with an approach to Web3 storage that our team simplified for a React Native application. It’s a note taking application...or rather, it is a set of UI and back-end components that you can use to construct your own Web3 note taking application.
Think of SoJourn like the picture on a box of Legos. The important thing is what you do with those pieces, and we hope you find a few of them unique and useful in your own projects.
What makes SoJourn a “Web3” note-taking application? The way it stores data, minimizing the need to trust institutions to keep it safe.
How it Works
SoJourn lets you write notes or edit existing notes. Then it:
Stores and encrypts the notes locally on an iPhone using the Apple keychain, so that all the notes are instantly available to the user;
“Notarizes” the data by storing a hash onto a public blockchain (we’re using the Ethereum Rinkeby testnet to demonstrate), so that a user can prove that the original note was saved on a particular date and not changed since;
Encrypts each note, divides it into parts (using Shamir’s Secret Sharing algorithm ), and distributes each of them to different locations on IPFS. We’re using the Infura gateway as an example, but you can add your own.
This means the user doesn’t have to worry about losing their data if they lose their device. But – and this is important – the user also doesn’t have to worry about a centralized cloud service keeping the data secure.
Why do Web3 Data Storage This Way?
We thought of many practical uses of this kind of Web3 Storage pattern. The simplest is a diary app that lets you record things, for example, like an account of harassment, which you want to be sure can’t be hacked into but which you can also produce years later and prove that you didn’t just recently make the story up.
The whole point of our team is to give you a leg up on your own ideas, but we did publish one story that might give you ideas:
The SoJourn story was published on Medium on October 4, 2018 and received a lot of interest from developers and startups generally. In the story, a journalist is taking sensitive interview notes with a political informant and really wants to be sure that nobody but her will ever access them unless and until she wants to share them. We thought the user, Zoe the journalist, made a good case for a professional need for the functionality.
The SoJourn User Experience
This open source note taking app is about as basic as you can get, but it gives you a foundation for adding all your bells and whistles.
Scan through existing notes and select to open the contents.
Create a note quickly with one tap.
Write and update a note and then save, notarize and store the latest version to the Vault.
Restore from Vault
The Vault uses a Web3 approach to secure data storage, so that the user doesn’t have to worry about losing their device and their data with it.
We have not added the restore function in this UI, because it relies on decisions you will make about how to handle passing data to the new device. Get the restore function code, full details of the Vault module, its use of Shamir’s Secret Sharing and IPFS, and ideas on how to mod it for your project in the github repository.
Left for you
To be a complete note taking app, clearly your project will want to allow the user to do all sorts of things, like formatting text, sorting the note list, and searching. These are basics that can be done in any number of ways and are usually where designers and app developers make their mark, so we have left this to you. We hope that the clarity we tried to achieve with the structure of the code and the docs make it easy for you to add functionality.
Because IPFS stores files permanently, the Vault is capable of versioning. We implemented storage on device more simply, so the devKit doesn’t enable the UI to provide version control as-is. There are many well-known ways to implement versioning, so we leave that to you.
The Basic Notes App - UI
The notes app is structured as you would expect for a redux based React app. For example, the screen to edit a note is in the note module directory . The same directory contains reducers and selectors providing the business logic for managing apps.
What makes the SoJourn framework different from a typical cloud note-taking application is that the Vault, which uses Shamir’s Secret Sharing to save a series of values to many IPFS locations. This eliminates the chance of a central cloud admin accessing a whole encrypted file and breaking its encryption. An attacker in this scheme would have to know which IPFS addresses correspond to the user’s data, and would still need the user’s private key to decrypt the file.
This is “belt-and-suspenders” security for users that are willing to trade some of the benefits of “Google Docs” style cloud apps for a high level of assurance that cloud storage will not result in the potential for a bad actor at a cloud company to find, access, decypher or hand over data.
Encrypt the Notes
We use AES to encrypt the data during the Done function in the application.
AES requires an input of a 256 bit private key. The devKit demonstrates this by generating a private key.
The entire purpose of the vault is to allow a user to access their data from other devices. In order to do that, they need to be able to share their private keys with other devices safetly. We did not implement that in the dev kit, but there are a few options. Each one effects the user experience in different ways. One way is to generate the key using a menomonic. This would force a user to keep track of a 12 word phrase. The pros of this is that it is a simple solution, the con is that uPort and other wallets use mnemonics, and it might be hard to keep track of multiple. Another solution is to use Integrated Encryption Scheme , a variation on Diffie Hellman, to securely share a private key with other devices. Yet another solution is to use a QR code to pass the private key between devices.
Grind the Notes into Little Bits
There’s a lot of history that you can learn about Adi Shamir and his secret sharing algorithm (SSSA), which we decided to use for this pattern. But for the purposes of this devKit, here are the essentials:
We needed a way to split up a file in such a way that there is no way to gain any information about the original content from any single piece, even if decrypted. In fact, the pieces should be useless without all the necessary segments together. SSSA does this and turns out to have other unexpectedly useful benefits, like better redundancy and quicker file reassembly, and that should give developers a lot of options for cool user features.
The key terms you need to know to work with this module are:SECRET
For SSSA, the Secret is simply any bit array, but we implemented the Secret specifically as a Base64-encoded string, so that SSSA can process the AES Encrypted file that we are passing it.SHARES
We won’t get into polynomials and how SSSA does its magic. But you need to know that the generateShares() function returns a set of hex values which, through combine() allow you to reconstruct the original data IF you have all the shares together to pass into that function.THRESHOLD
Saving Shares to decentralized storage involves the risk that some of the storage locations may not be available at any given time. If your Threshold value is set to the same number as Shares, you will need to be able to pull every Share successfully in order to reconstruct the file. On the other hand, if you set the Threshold value to 1, you would only need one of the Shares to reconstruct the file. That would kind of defeat the purpose of this whole bag of tricks, but you could do it. In a production application, it’s up to you to decide the optimal ratio of Threshold to total Shares, for redundancy.
Storing the Shares
Once you have the data sliced up in to the Shares, they need to be stored somewhere. On this implementation, we chose to use IPFS via the Infura Gateway.
Infura stores data on many AWS instances distributed globally. It should be noted that currently Infura is managing this in a somewhat centralized way, but Infura plans to run IPFS fully decentralized, so consider this a proof of concept...and it may be good enough for many users’ privacy requirements.
IPFS stores data by hashing the content and using that hash as the address to access the data. You can then use the hash to enable any browser to pull up the data at that location. In short, all IPFS locations are accessible to the public. That sounds like the opposite of privacy, perhaps, but thanks to the previous step of splitting the data into Shares, the probability of an attacker finding the correct set of Shares to reconstruct the file is astronomically small. Pretty cool, yeah?
Storing the secret locations to find the Shares
If the user can’t supply the locations of a Threshold number of Shares to their data, the data is as good as gone. They will never find it out there in IPFS. So it is critical that your app store the file locations somewhere in a secure way, so that the user can use it to restore their data. It should also be stored away from the user’s private key for AES, so that an attacker would have to gain access to two separate resources in order to both reconstruct and decrypt the file.
IPFS returns a hash of each Share location, and the code will concatenate the hashes into an array, which, depending on how your application manages this, you will use to let the user restore their data (e.g., in the event of them losing their phone).
We built the SoJourn devKit with an eye toward providing a clean, open source Shamir’s Secret Sharing library that works for React Native apps. Mission accomplished.
Until recently, there were no well-known ways to do Web3 storage in a way that we considered acceptable for very secret data. It’s one thing to worry about a system admin at AWS, but we’d take that over some guy somewhere with a hard disk on a public network storing my file, even if it is encrypted.
However, there are new ways emerging of splitting up files, rendering them useless without also having many other slices, and storing them on decentralized infrastructure. One service is Storj’s upcoming version 3.
In order to add Storj as another location for the Shares, modify how you upload shares for use with the Storj Uplink Client. If you don’t want to use Shamir’s and want to simply replace the Vault with standard Storj, check out Storj’s tutorials on GitHub for the uplink client and S3 gateway.
In fact, you might want to consider modifying the Vault to not only store shares on IPFS but also on Storj and other storage providers, for an additional assurance that no single gateway provider trafficked in enough shares to reconstruct the file. All depends on how paranoid you want to be. (Storj might also be a good choice for storing the encrypted location file that SoJourn needs to restore data to a new device.)
The Notary service handles the hashing of user notes onto a blockchain, preferably a mainnet like Ethereum’s, so that there is a solid chance of long-term continuity and strong tamper-resistance.
If we were doing this all over again, we’d probably make the Notary library stand apart, or use something like Chainpoint but here’s what we have:
Then, at the user’s request, we use Web3.js in the note app to use the Notary function on the smart contract.
Authentication & Wallet
The SoJourn devKit for demonstration uses iPhone Face ID for app authentication and to decrypt the local copy of the notes data.
The Notary module uses gas. We made this choice to demonstrate using the Ethereum Rinkeby testnet. That means the app needs a wallet and we implemented uPort’s which the user can set up when the app is first loaded.
Have Fun Stormin’ the Castle!
So there it is. We hope that the code and our descriptions of the Web3 storage pattern above, along with the story of the SoJourn notes application, give you great ideas and a leg-up building your own awesome applications.
Remember, this is totally open source code on an Apache 2 license.
What do we want in return? Two things:
The Making Of
We want to give you a detailed look at our thinking around how we built this devkit. Often when building things you run into challenges and gain tribal knowledge that we want to share, with you. To see all of our videos, check out our YouTube Channel.