OBLIVIOUS.IO

Tutorials

To make Oblivious as easy as possible to integrate into your application, we supply library-compatible language bindings for some languages and cloud providers. These libraries make adding secure storage to your application easy. Let's use Node.js as an example. Just change out the library, add a configuration file, and (optionally) change the name of the S3 bucket you're using and the rest of your application should work as expected.

Node.js tl;dr

var fs = require('fs');
var S3 = require('aws-sdk/clients/s3');

var filename = '../mytestfile.txt';

var s3 = new S3({
  accessKeyId: process.env['AWS_ACCESS_KEY_ID'],
  secretAccessKey: process.env['AWS_SECRET_ACCESS_KEY'],
  region: process.env['AWS_REGION'],
});

s3.upload({
  Bucket: 'unencryptedbucket',
  Key: filename.split('/').pop(),
  Body: fs.createReadStream(filename)
},
function(err, data){
  if(err) console.log('error:', err);
  else console.log('done');
});              
              

var fs = require('fs');
var S3 = require('oblivious.io/clients/s3');

var filename = '../mytestfile.txt';

var s3 = new S3({
  accessKeyId: process.env['AWS_ACCESS_KEY_ID'],
  secretAccessKey: process.env['AWS_SECRET_ACCESS_KEY'],
  region: process.env['AWS_REGION'],
  oblivious: {
    passphrase: process.env['OBLIVOUS_PASSPHRASE'],
    token: process.env['OBLIVIOUS_TOKEN'],
  }
});

s3.upload({
  Bucket: 'encryptedbucket',
  Key: filename.split('/').pop(),
  Body: fs.createReadStream(filename)
},
function(err, data){
  if(err) console.log('error:', err);
  else console.log('done');
});              
              

If you inspect the contents of your 'encryptedbucket' after making these changes, you'll see entries with unique ids instead of the filenames (Key's) typically found.

Getting Started

In this tutorial you will

1. See how to add Oblivious to an existing app
2. Upload a file through Oblivious to S3
3. Download a file through Oblivious from S3
4. Upload several files with different passphrases and see how they are partitioned

If you haven't already, you should create an account through our signup form. This will grant your credentials and (optionally) configure your 2FA (Two-Factor Authentication) token.

Next, let's build a test project using the examples provided by AWS. You will need AWS credentials to make this work (free tier works just fine). Since the first two scenarios in the examples (listing and creating buckets) are unchanged and work as a passthrough with the Oblivious libraries, let’s move on to a more interesting example, ‘Uploading a File to an Amazon S3 Bucket’.

Make an Example Project

You’ll need to create a new project for this, and make sure your AWS credentials are in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Here’s a quick copy/paste for a project setup:

$ echo $AWS_ACCESS_KEY_ID
[big alphanumeric response]
$ echo $AWS_SECRET_ACCESS_KEY
[another big alphanumeric response]

$ mkdir -p foo && cd foo && npm init -y && npm install aws-sdk --save
            

Here’s the code as supplied by AWS:

// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Set the region 
AWS.config.update({region: 'REGION'});

// Create S3 service object
s3 = new AWS.S3({apiVersion: '2006-03-01'});

// call S3 to retrieve upload file to specified bucket
var uploadParams = {Bucket: process.argv[2], Key: '', Body: ''};
var file = process.argv[3];

var fs = require('fs');
var fileStream = fs.createReadStream(file);
fileStream.on('error', function(err) {
    console.log('File Error', err);
});
uploadParams.Body = fileStream;

var path = require('path');
uploadParams.Key = path.basename(file);

// call S3 to retrieve upload file to specified bucket
s3.upload (uploadParams, function (err, data) {
  if (err) {
    console.log("Error", err);
  } if (data) {
    console.log("Upload Success", data.Location);
  }
});
            

Save this to a file named s3_upload.js, make sure to set a REGION in the code, and (as the AWS tutorial suggests) you can run this with:

node s3_upload.js BUCKET_NAME FILE_NAME
            

You can trust it was properly uploaded or check your S3 console to verify that it was. (hint: Don’t hunt for a file to upload. Use the package.json file you just created. It has nothing sensitive in it.)

Use Oblivious in a Project

To update this app to use Oblivious.io, you’ll need to:

1. Add the library to your project
   $ npm install oblivious --save
2. Exchange your Oblivious credentials for a token

   
   With curl without 2FA enabled:
   $ curl -d '{"username":"username", "password":"password"}' \
   -H "Content-Type: application/json" \
   -X POST https://api.oblivious.io/tokens
   
   With curl and 2FA enabled:
   $ curl -d '{"username":"username", "password":"password", "twoFactorToken":"123456"}' \
   -H "Content-Type: application/json" \
   -X POST https://api.oblivious.io/tokens
                   

   Or, with the included helper utility (follow the prompts for user/password/passphrase):
   

   
   $ node_modules/.bin/login
                   

3. Change out the standard aws-sdk library for Oblivious (line 3), and add the configuration block for the passphrase and token (lines 10-13):

   
   // Load the AWS SDK for Node.js
   var AWS = require('aws-sdk');
   // Set the region 
   AWS.config.update({region: 'REGION'});
   
   // Create S3 service object
   s3 = new AWS.S3({apiVersion: '2006-03-01'});
   
   // call S3 to retrieve upload file to specified bucket
   var uploadParams = {Bucket: process.argv[2], Key: '', Body: ''};
   var file = process.argv[3];
   
   var fs = require('fs');
   var fileStream = fs.createReadStream(file);
   fileStream.on('error', function(err) {
       console.log('File Error', err);
   });
   uploadParams.Body = fileStream;
   
   var path = require('path');
   uploadParams.Key = path.basename(file);
   
   // call S3 to retrieve upload file to specified bucket
   s3.upload (uploadParams, function (err, data) {
     if (err) {
       console.log("Error", err);
     } if (data) {
       console.log("Upload Success", data.Location);
     }
   });
                     

   
   // Load the AWS SDK for Node.js
   var AWS = require('oblivious');
   // Set the region 
   AWS.config.update({region: 'REGION'});
   
   // Create S3 service object
   s3 = new AWS.S3({
     apiVersion: '2006-03-01',
     oblivious: {
       passphrase: process.env['OBLIVOUS_PASSPHRASE'],
       token: process.env['OBLIVIOUS_TOKEN'],
     }
   });
   
   // call S3 to retrieve upload file to specified bucket
   var uploadParams = {Bucket: process.argv[2], Key: '', Body: ''};
   var file = process.argv[3];
   
   var fs = require('fs');
   var fileStream = fs.createReadStream(file);
   fileStream.on('error', function(err) {
       console.log('File Error', err);
   });
   uploadParams.Body = fileStream;
   
   var path = require('path');
   uploadParams.Key = path.basename(file);
   
   // call S3 to retrieve upload file to specified bucket
   s3.upload (uploadParams, function (err, data) {
     if (err) {
       console.log("Error", err);
     } if (data) {
       console.log("Upload Success", data.Location);
     }
   });
                     

That's it! You've converted this example app to use Oblivious.io. Let's run it and watch some encrypted data get uploaded.

Encrypt and Upload Content

Now, save this with a new filename ‘upload-oblivious.js’ and run it again just like the standard version:

            $ node upload-oblivious.js BUCKET_NAME FILE_NAME

            Example: node upload-oblivious.js obliviousbucket package.json
            

If you open the AWS console and look at the contents of the bucket, you’ll see an object where the Key is a UUID. You can inspect this file and see that it’s just encrypted text - indistinguishable from random noise. However, you can download this file with Oblivious and regain the plaintext.

Download and Decrypt Content

// Load the SDK for JavaScript
var fs = require('fs');
var AWS = require('oblivious');

// Set the region 
//AWS.config.update({region: 'REGION'});
AWS.config.update({region: 'us-east-2'});

// Create S3 service object
s3 = new AWS.S3({
  apiVersion: '2006-03-01',
  oblivious: {
    passphrase: process.env.OBLIVIOUS_PASSPHRASE || '',
    token: process.env.OBLIVIOUS_TOKEN || '',
  }
});

var file = fs.createWriteStream(process.argv[4]);

s3.getObject({
  Bucket: process.argv[2],
  Key: process.argv[3]
}).createReadStream().pipe(file);

// usage: node getobject-oblivious.js bucketname keyname outpufilename
            

Then we can run it like this:

$ node getobject-oblivious.js BUCKET_NAME KEYNAME OUTPUTFILE_NAME

Example: node getobject-oblivious.js obliviousbucket package.json ./dl_package.json
            

Take a look at your download file (dl_package.json) and you’ll find the original package.json you uploaded. Does this seem simple? Good. Then we’re on the right track. Now that the standard CRUD lifecycle has been established, let’s do something cool.

Do Something Cool

The passphrases with Oblivious are more than just security. They develop a natural index and partition for data. We’re going to upload a few files with different passphrases and see how they appear to Oblivious. I suggest you do this with the example repo.

Create a couple of files:

`date > filename1` (wait a few seconds) 
`date > filename2` (wait a few seconds) 
`date > filename3`
            

Let’s upload the first two files with one passphrase:

$ OBLIVIOUS_PASSPHRASE=passphrase1 node upload-oblivious.js obliviousbucket filename1
$ OBLIVIOUS_PASSPHRASE=passphrase1 node upload-oblivious.js obliviousbucket filename2
            

And the third file with a different passphrase:

$ OBLIVIOUS_PASSPHRASE=passphrase2 node upload-oblivious.js obliviousbucket filename3
            

Now, let’s check the listing of the bucket with the first passphrase:

$ OBLIVIOUS_PASSPHRASE=passphrase1 node listobjects-oblivious.js obliviousbucket
            

The output should look something like:

Success { IsTruncated: false,
  Marker: '',
  Contents: 
   [ { Key: 'filename1',
       LastModified: '',
       Size: 0,
       StorageClass: 'STANDARD',
       Owner: [Object] },
     { Key: 'filename2',
       LastModified: '',
       Size: 0,
       StorageClass: 'STANDARD',
       Owner: [Object] },],
  Name: 'obliviousbucket',
  Prefix: '',
  MaxKeys: 1000,
  CommonPrefixes: [] }
            

Let’s try it again with the other passphrase:

$ OBLIVIOUS_PASSPHRASE=passphrase2 node listobjects-oblivious.js obliviousbucket
            

Yields:

Success { IsTruncated: false,
  Marker: '',
  Contents: 
   [ { Key: 'filename3',
       LastModified: '',
       Size: 0,
       StorageClass: 'STANDARD',
       Owner: [Object] } ],
  Name: 'obliviousbucket',
  Prefix: '',
  MaxKeys: 1000,
  CommonPrefixes: [] }
            

Try again with some random passphrase:

$ OBLIVIOUS_PASSPHRASE=asdfasdfgh node listobjects-oblivious.js obliviousbucket

Success { IsTruncated: false,
  Marker: '',
  Contents: [],
  Name: 'obliviousbucket',
  Prefix: '',
  MaxKeys: 1000,
  CommonPrefixes: [] }
            

Hooray! Natural data isolation! As you might guess, these files can’t be decrypted without the same passphrase used in listing and you’d be right. Attackers without the passphrase can’t even get the number of files, filenames, or file sizes. Using Oblivious, you can create a per-user passcode that will fully isolate access to that user’s files with their passcode. Even if the S3 bucket were set to public and entirely compromised, the data would be perfectly secure. If you’re making a picture storage site, only the user with the passcode can access those pictures. If you’re storing backups, only the user with the passcode for the backups will even know they exist. Standard username and passwords become less important because the passphrase is what protects the files and metadata.

Oblivious isn’t storing or indexing any of the passphrases. You can create as many as you like and no one will ever know. The isolation is based entirely on a mathematical proof, but that’s better described in another document.