Paul Wilton

The problem: flaky connections cause upload of large files to S3 to fail
Uploading files to S3 securely direct from a browser is a common pattern. There
is plenty of information available on how to do this. Typically most examples
documented online use standard S3 signed POST requests (either from direct from
a html form or via an XHR request). This often means using some server side API
call (which you need to build) to generate an AWS signed request, which can then
be used in your web app client side. While being fairly easy to implement, for
very large files a flaky connection or a dropped packet often means a failed
upload and retries. What you need is multipart, chunked uploads that are
resilient. However this is not trivial to achieve with S3.

The solution: secure chunked multipart managed uploads
If you use the AWS SDK there are essentially two routes. Use the S3 REST API and
manage file chunks yourself. This is the low-level approach and is complex.
Alternatively you can use the AWS SDK Managed Uploads API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property]
, in which the SDK does all the hard work for you to manage file chunking,
upload, resilience, and the reconstruction of the file parts into a single S3
file at the end. However to use this you have to supply AWS credentials, and
this is not something you typically want to expose client side. However, if your
upload feature is behind some oAuth or OpenID authentication layer (as in my
case, and I suspect is often the case if you are allowing uploads to your
backend) then you are in business. You can setup AWS to work as OpenID Connect
provider. Finally you need to integrate this with some client side file uploader
library, such as Dropzone [http://www.dropzonejs.com/]. I am going to
demonstrate step-by-step how to :

a. Setup an S3 bucket to receive the uploads with a suitable access policy and
CORS settings.
b. Setup AWS as an OpenID Connect provider for some external oAuth service - in
this case Auth0 [https://auth0.com].
c. Integrate AWS S3 Managed Uploads securely with Dropzone to handle multiple
parallel resilient uploads for very large files, with progress bars, and cancel.

Step 1 - Create an S3 bucket
In your AWS tenancy, create an S3 bucket (e.g. my-bucket) and a CORS policy as
follows.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-meta-custom-header</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>Authorization</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>


You can revoke all public access to the bucket, as we will be writing to the
bucket using an OpenID Connect user that has assumed an IAM Role with the
correct policy to access the bucket.

Step 2 - Create an OpenID Connector Provider in AWS for Auth0
This allows us to use AWS as an intermediary for validating JWT OpenID tokens
against an oAuth service, demonstrated here with Auth0.
Your web app users will authenticate (login) with Auth0 - The oAuth response
from Auth0 authorize endpoint delivers an OpenID JWT token to the browser, which
you typically store in localStorage for securing subsequent requests from the
client to your backend.
To setup the AWS OpenID Connect provider you need the oAuth service root
endpoint, and the oAuth application clientID. In Auth0 login to your dashboard
and in your application settings you can get this info.

The oAuth endpoint is : https://{your-auth0-domain}/ (note the trailing slash),
e.g. https://appname.auth0.com/

The application Client ID is a 32 character Base62 encoded string, e.g. 
6sR4LF3UpznU4T26S3xRheGhjEKdni4H

Now login to your AWS tenancy, navigate to IAM account management > Identity
Providers. Click on Create Provider, and enter the Auth0 details from above as
follows, ensuring you add the trailing slash on the provider URL:

Then click Next and Create the provider.

Step 3 - Create an IAM Role for the OpenID Provider
Navigate to Policies in IAM, and create a new Policy as follows using the JSON
editor, setting the correct name for {my-bucket}:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::{my-bucket}",
                "arn:aws:s3:::{my-bucket}/*"
            ]
        }
    ]
}


Note, this policy will grant the authenticated user assuming the role all
actions on the bucket.. you may want to make this more restrictive. You could
also add other AWS grants here too, so your user can access other AWS services
if need be. Name your policy to something sensible, eg OpenIdS3Policy.

Now navigate to IAM Roles, and create a new Role. Choose Web Identity as below
and choose your OpenID Provider you just created, and the Auth0 Client ID as the
audience in the drop down options :

Next, Set the permissions of the role by choosing the OpenIdS3Policy created
previously. It is easy to find your policy by selecting policy type Customer
managed from the drop down. Save the Role with a suitable name eg OpenIdS3Role.

Step 4 - Grant access to the S3 bucket
Now we need to grant access to the S3 bucket for the IAM Role just created.
Copy the ARN for the Role, and navigate to your bucket setttings, and choose 
Bucket Policy. Add a Policy as follows, using the ARN for the role, and the
correct bucket name :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permissions",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:GetBucketLocation",
          "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}"
    },
    {
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}/*"
    }
  ]
}


That is all the AWS work needed. An authenticated OpenID Connect client can now
access your S3 bucket by assuming the role of an IAM user. When the AWS
connection is made, AWS authenticates the supplied JWT token against Auth0 using
the AWS Identity Provider. Thus a user has to be logged into an Auth0 session to
be able to access the bucket. AWS matches the aud value in the decoded JWT token
with the trust relationship condition specified in the role we created. This is
where the all important trailing slash I mentioned comes into play. This is an
artefact of using Auth0 as an OpenID identity service. Auth0 adds a trailing
slash to the aud value in the JWT token it generates. If we dont add the same,
AWS cannot match the role with the token when it tries to assume the identity of
the connecting user. If you use a different oAuth provider than Auth0 you may
not need this trailing slash. Think of it as an Auth0 'gotcha'.

Step 5 - Integrate with Dropzone
Dropzone [http://www.dropzonejs.com] is one of the better, and widely used open
source file upload JS libs. It is very customizable and mostly event driven so
is easy to modify. Out of the box Dropzone uses XHR requests to send files to a
remote server. We are going to override Dropzone's file uploading mechanism to
use the AWS SDK Managed Upload API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property] 
instead.

First, install (or make available) the AWS SDK in your web app, e.g.

npm install aws-sdk --save


Now define an accept callback function, that Dropzone invokes when a file has
been accepted for upload. This lets us add an S3.ManagedUpload instance to the
file object being queued by Dropzone for upload :

import S3 from 'aws-sdk/clients/s3';

function acceptCallback(file, done) {
  // options for the managed upload for this accepted file
  // define the bucket, and the S3 key the file will be stored as
  var params = {Bucket: 'my-bucket', Key: file.name, Body: file};
  // add an S3 managed upload instance to the file
  file.s3upload = new S3.ManagedUpload({params: params});
  done();
}


Now we need to override the Dropzone uploadFiles function to upload via the AWS
SDK instead of XHR. Note we are using the file.s3upload instance we added to the
file object when the file was accepted by Dropzone. We emit Dropzone events on
success and failure, and we also listen to the AWS httpUploadProgress event, and
emit the equivalent Dropzone event, so Dropzone can render upload progress for
each file :

// override the uploadFiles function to send via AWS SDK instead of xhr
Dropzone.prototype.uploadFiles = function (files) {
  for (var j = 0; j < files.length; j++) {
    var file = files[j];
    sendFile(file);
  }
};

function sendFile(file) {
  file.s3upload.send(function(err, data) {
    if (err) {
      theDropzone.emit("error", file, err.message);
    } else {
      theDropzone.emit("complete", file);
    }
  });

  // listen to the AWS httpUploadProgress event, and emit an equivalent Dropzone event
  file.s3upload.on('httpUploadProgress', function(progress) {
    if (progress.total) {
      var percent = ((progress.loaded * 100) / progress.total);
      theDropzone.emit('uploadprogress', file, percent, progress.loaded);
    }
  });
}


If we want our users to be able to Cancel uploads mid-progress we need to
implement an abort function that can be invoked on the Dropzone canceled event :

function abortUpload(file) {
  if (file.s3upload) file.s3upload.abort();
}


Finally you need to initialize your Dropzone on some DOM element on your page
and
setup AWS Credentials for the S3 managed uploads. To do this we create an 
AWS.WebIdentityCredentials object using the ARN of the IAM role we created in
Step 3, and also inject the JWT ID token from your local storage/session - the
token returned by the Auth0 client side authentication workflow. When we
initialize Dropzone we also set up the accept and canceled event callbacks.

import AWS from 'aws-sdk/global';
import S3 from 'aws-sdk/clients/s3';

function initializeDropzone(dropzoneElement) {

  var idToken =  'your-jwt-openId-token'; // get from local storage/session
  AWS.config.credentials = new AWS.WebIdentityCredentials({
    RoleArn: 'arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role',
    WebIdentityToken: idToken
  });

  theDropzone = new Dropzone(dropzoneElement, {
    autoQueue: false, // if you want to manually trigger uploads
    accept: acceptCallback,
    canceled : abortUpload,
    init: function () {
      this.on('removedfile', function (file) {
        abortUpload(file)
      })
    }
  })
};


All done. In these examples, I have shown the minimum you need to do to get S3
Managed Uploads working with, for example, the Boostrap Dropzone demo here
[http://www.dropzonejs.com/bootstrap.html]. There are of course lots more
features of Dropzone you can implement, but this should be sufficient to for you
to build fully resilient secure S3 uploads using Dropzone and OpenID
authentication for very large files.

Dropzone

Dropzone Blog Posts