Steve Liles

React and SVG make a powerful combination: Declarative UI component library
meets declarative graphics language — a match made in front-end developer
heaven.

In this mini-series we’ll be looking at rendering interactive graphical UI’s
using components built with React and SVG.

These days SVG icons are ubiquitous, but those are not the sort of components
we’re talking about.

We’re going to put together the interactive floor-plan UI using simple React
code that renders SVG elements instead of HTML. We’ll take it relatively slowly,
building through the basics into more advanced topics.

 * Part 1 — Declarative Graphics (this post)
   In which we look at the match made in heaven that is React and SVG
 * Part 2 — Arcs, Angles, and Transformations
   Keeping geometry simple with groups and transformations
 * Part 3 — Clip-paths
   An adventure in composite geometry
 * Part 4 — Masks
   When clip-paths don’t cut it: masks to the rescue
 * Part 5 — Animating components 
   Where we animate individual components
 * Part 6 — Animating the viewBox 
   In which we fluidly zoom and pan our entire UI

Let’s dive straight in!

Declarative Graphics
React developers enjoy first-class support for HTML elements in JSX:

const SomeComponent = () => (  
  <div>
    <p>hi!</p>
  </div>
);

But did you know that this applies to inline SVG elements too?

const SomeOtherComponent = () => (
  <svg viewBox="0 0 100 100">
    <circle cx={50} cy={50} r={30} fill="red"/>
  </svg>
);

Out of the box we can write  components that compose SVG primitives to create
complex interactive UI’s, just as you would with more typical HTML based
components.

(Note for React-Native folks: you’ll need to install react-native-svg, which was
looking pretty good last time I worked with it. Here
[https://stackoverflow.com/a/39041997/473338] is a simple but pretty powerful
example I created a few years ago).

Simple composition of declarative graphical components (https://62zqd.csb.app/)
Notice that we did all this drawing without a single line of imperative code,
and the declarative markup of our ghosts is very readable and accessible to
anyone familiar with React — a far cry from what this would look like if we
tried to do it with the Canvas API.

All your elements are belong to us
To draw anything with SVG, we first need an <svg>element to define the drawing
context within which all descendant elements are rendered.

We can have many <svg> elements in our page, each describing an isolated drawing
context — e.g. one <svg> per icon, chart, or other snazzy graphical widget.

Alternatively, we can root many graphical components within the same <svg> and
create an entire UI in a single drawing context.

Viewport and ViewBox
At a minimum an <svg> element should specify how much screen-space it will
occupy — its viewport. This can be done with CSS styles or width and height
attributes.

<svg width="200px" height="200px">...</svg>

We’ll typically want to specify a viewBox too, to explicitly define the
“user-space” portion of our drawing that is made visible in the viewport.

Think of the viewport as a window into the world defined by the SVG
(“user-space”), and the viewBox as zoom and pan settings that control what parts
of the user-space you can see through the window.

The viewBox attribute is a string which specifies, in order, the user-space x
and y coordinates of the top-left visible corner, and the width and height of
the viewBox.

<svg viewBox="x y width height">...</svg>

For example, “-100 -100 200 200” describes a square viewBox 200 units high and
200 units wide, centred around the origin at (0,0).

<svg width="10" height="10" viewBox="-100 -100 200 200">...</svg>

If we don’t specify it explicitly we get a viewBox that maps width and height
1:1 to the viewport, with the user-space origin (0,0) at the top-left corner.

To re-iterate: the viewport describes how much space the <svg> element takes up
within your HTML page. The viewBox describes the portion of the image that is
visible within that viewport. By changing the viewBox we can zoom in or out, or
show entirely different portions of the SVG image.

The only difference between these two SVG’s is the viewBox and the background
colour (https://0oy6p.csb.app/)If the proportions of your SVG viewport in pixels
don’t match the proportions of the viewBox, the default behaviour is to preserve
the aspect ratio and centre and fit the viewBox within the available viewport.
This behaviour can be controlled by varying the preserveAspectRatio
[https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/preserveAspectRatio] 
attribute.

By default the viewBox will be centred and “fit”within the SVG viewport (
https://52fbc.csb.app/)The separation of coordinate systems between the viewBox
and the viewport means that within our SVG we can work with coordinates that
makes sense for our specific drawing.

For example, if we’re drawing an interactive widget we can make the maths easy
for ourselves by working in percentages and specifying a viewBox of “0 0 100
100”.

Or if our widget has radial symmetry and we want to easily draw from the centre,
we can use a viewBox of, say, “-50 -50 100 100” centred at the origin (0,0).

In this series we’ll be drawing floor-plans, so we could choose to work in
metres, or feet-and-inches, or whatever system of measurement our plans use.
We’re actually going to use millimetres, just for the simplicity of working in
whole numbers.

We’ll describe our floor-plan with some simple JSON, specifying the room
geometry as coordinates of corners working clockwise around the room.

Now we can set up our viewport and viewBox, allowing a little padding around the
floor-plan so it isn’t too snug in the viewport.

import data from './floorplan-data.json';
const App = () => (
  <svg 
    width="500px"
    height="500px"
    viewBox="-1000 -1000 14000 11000"
    style={{backgroundColor:'blue'}}
  >
    // ... floorplan components here ...  
  </svg>
);

With our viewport and viewBox sorted, we can get on with drawing.

Declarative Drawing Primitives
SVG includes a number of drawing primitives ranging from the simple <line> and 
<circle> to the more flexible <path>. Mozilla Developer Network has a good 
reference [https://developer.mozilla.org/en-US/docs/Web/SVG/Element] of the
available elements.

Shapes can be filled and/or stroked, where fill is the colour applied within the
bounds defined by the shape, and stroke is the colour applied to the outline of
the shape. The width and other styles of the stroke can be controlled with
attributes or CSS.

We’ll be drawing our initial floor-plans using two simple primitives: <line> and 
<circle>.

<line> describes a line between a pair of (x, y) coordinates.

<line x1={100} y1={100} x2={200} y2={200} stroke="red"/>

<circle> describes a circle of radius r centred at (cx, cy).

<circle cx={50} cy={50} r={20} fill="green" stroke="black"/>

Room with a View
To draw our building we’ll pass the data into a Floorplan component which will
render out the various elements of the floor-plan — initially just the rooms.

const Floorplan = ({ data: { rooms } }) => (
  rooms.map(r => <Room {...r} />)
);

For the Room component to draw the walls we’ll need to map over pairs of
consecutive corner coordinates, linking them together to form walls. For example
a rectangular room with corners a, b, c, and d, will have four walls
(a-b),(b-c), (c-d), (d-a).

We can pull out those pairs with a simple function, perhaps wrapped in a useMemo 
hook for efficient re-renders.

const walls = useMemo(() =>  
  coords.map((_, i) => {
    const a = coords[i];
    const b = coords[(i + 1) % coords.length];
    return [a, b];
  }),
  [coords]
);

Now drawing the walls is as simple as using the SVG <line> element to describe a
line between the pairs of connected corners.

That works, but it isn’t very pretty. Let’s extract a Wall component and a 
Corner component, and get a bit more creative.

In our wall component we’re now drawing two lines per wall — first a thick white
line, then over-drawing the white line with a thinner, dark blue, dashed line.

Corner is very simple — just a circle, stroked white and filled dark blue.

The Room component now simply composes these higher-level Wall and Corner 
components instead of SVG primitives.

https://codesandbox.io/s/react-svg-basics-rk1t3You may have noticed that we
nested our Walls and Corners in an SVG <g> element. This is a logical grouping
element — the <g> on its own doesn’t render anything visible — but it does give
us a neat way of performing transformations such as rotations and translations
on entire sub-hierarchies of our SVG, and a means of capturing events from those
sub-hierarchies by attaching event handlers to the <g> element.

Recap
In this post we saw that:

 1. We can easily draw arbitrary shapes using SVG primitives and
    straight-forward, declarative React code — no plugins, libraries, or
    imperative code required.
 2. Complex graphical UI’s can be built by composing together increasingly
    higher level components, just like any other React application.
 3. The separation of viewBox and viewport lets you draw in a coordinate space
    that suits you, then render the result in a viewport appropriate to the
    final display requirements.

In “Part 2 — Arcs, Angles, and Transformations”, we’ll dig further into those 
<g> elements and look at paths and transformations while adding camera’s to our
floor-plan.

Until next time, happy drawing!

Software Development

Graphical UI's with SVG and React,  part 1 - Declarative Graphics

Julian Everett

One the most important tasks for software architects in the early stages of any
technology delivery program is to reach “architecture conceivability”, or
consensus agreement with senior developers that the technical proposition could
actually conceivably be workable. Starting from a position of defensive
scepticism, it typically entails an outside-in narrative exploration of the
potential end-to-end program execution flows at a level of detail between the
Component and Code levels in the C4 Architecture Model [https://c4model.com/] 
for key use cases.

The granularity of the discussion will switch depending on the level of
technical risk entailed by the component. If it is providing a standard
capability that is self-evident to to those present how to code, then there is
no point doing the more detailed code discussion. If it is encoding anything
with a non-trivial level of technical risk, then dropping down to the next level
will be required until everyone is roughly on the same page about how it needs
to work. In this way it acts a bit like a code preview (as opposed to a code
review), functioning in the same way that a delivery prespective
[https://www.malotaux.eu/nrmc.php?id=prespectives] is more efficient than a
retrospective in terms of mitigating risk.

Once and only once this has been completed, you will have established baseline
viability for the program. The narrative described will be unquestionably wrong,
inefficient, full of flaws – that’s not the point. The point is that it will
represent a first, stuttering, throwaway sketch out which a simple, resilient
system design can eventually be evolved. Gaping holes and pointless layers of
cruft will already be visible. Beyond that, most of the selective pressures
driving that evolution will be created later by writing tests and the feedback
from coding itself. 

That’s all fine. However, until you’ve established architecture conceivability,
in reality you’ve got nothing. It is hard to over-emphasise the importance of
this point. Context and Container-level views are meaningless slideware (or at
very best, vague indicators of business process alignment) until they are backed
up by Component and Code-level views. That is solely how they gain their value,
by proxy as maps to decomposing and navigating the the Code-level views of the
platform. Without the code, they are essentially maps to nothing.

Technical Architecture

Architecture Conceivability

Fabio Colasanti

For Data Science to truly live up to expectations, it must embrace software craftsmanship.

Data Science is about building software

Earlier this month I was chatting with my old friend Pierluigi Casale
[https://www.linkedin.com/in/pierluigi-casale-41271430/] (Data Scientist at
TomTom, AI lecturer and researcher) about what seems to be the question of the
century: "What is Data Science"? Tongue in cheek, I offered my very own answer:
"Data Science is that thing you get when someone with a PhD plays with a Jupyter
Notebook, and a technical architect refuses to let it anywhere near production
because it doesn't have a single unit test". 
Although sporting a PhD himself, Pierluigi agreed that my slightly caustic
definition was indeed something he observed himself, and that he too was wary
of. He mentioned how he often concludes the many talks he gives at conferences
around the world with a diagram from the notorious paper "Hidden Technical Debt
in Machine Learning Systems
[https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf]
" by D. Scully et al.


There is indeed a widespread assumption that Data Science and Machine Learning
belong into a hazy limbo somewhere between data analysis, analytics, and R&D.
Within this mystical space, a set of practices otherwise regarded as inefficient
seem to be accepted as the norm: manual mining and manipulation of data; one-off
transformations virtually impossible to replicate; casual scripting of code with
total disregard of any development standard; artefacts laying around and code
running here or there...
When Scully talks about "glue code" and "plumbing", he is pointing at the right
problems but possibly underplaying the issue by being too humble about it. So
forgive my bluntness and let me put that straight: Data Science is about
building software.

Whether the context is academia or enterprise, research or commercial alike, the
adoption of Software Development best practices, tools and methodologies is the
key for efficient delivery of outcomes. At Data Language we have worked with a
number of clients at different levels of "enlightenment", and witnessed first
hand how injecting software craftsmanship into Data Science can lead to
profoundly transformational effects. We can also testify how the development of
our Tagmatic [https://datalanguage.com/tagmatic] service has been indeed 10%
Machine Learning and 90% software engineering: not much we could have done with
our shiny model without things like containerisation, high availability, high
throughput, load balancing, scaling, automation, security, governance,
monitoring, and many others. 

The adoption of established best practices in Software Development will boost
productivity, shorten the feedback cycles, and produce more robust code that is
easier to change, maintain, and ship.

Six things we at Data Language do and you should too
1. Ditch the darn notebooks
I'm fully aware this is highly controversial bordering sacrilegious for many,
but the truth is these type of editors foster bad habits and are holding you
back. They are like comfy, safe, cute trikes you got used to riding when you
were three; but if you actually want to go somewhere other than your garden,
you’ve got to move on and get yourself a bike. 
At this year's JupyterCon in NY, the talk widely regarded as best of the lot was
"I don't like notebooks.
[https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68282]"
by Joel Grus. I wholeheartedly invite you to check it out.

2. Version control
A common refrain in favour of notebooks, is that they enable collaboration.
Oddly enough, when further engaged on this topic and asked about which version
control tool they use, what's their branching strategy, how often they merge,
etc... it all unravels. 
Collaboration is mighty important, so make sure you get on top of your version
control strategy. (And ditch the darn notebooks)

3. Test automation and CI
Arguments for Test Driven Development and Continuous Integration are widely
available and universally recognised as the single crucial factor that enables
faster, safer, better code. The very same arguments apply to Data Science. The
ability to move faster with your code enables quicker feedback and shorter
iterations: you will be able to experiment and prove ideas quicker, boosting
productivity. You will also end up with code that is designed better, and it can
be transitioned into a production environment more easily.

4. Infrastructure as code
In order to take full advantage of the ongoing democratisation of computing
power, it is crucial to write code that is portable and can be deployed wherever
is faster to run it. In a typical development setting, we want be able to
provision computing resources in the cloud, deploy our code there, execute it,
collect outputs and dispose of the remote resources. And we want to be able to
do this in a fully automated fashion, at the push of a button.
Many ways to skin this cat, but if I were to single out one thing you should do,
and learn to do well, is containerisation. Plenty of great Docker learning
resources around: get stuck in!

5. Data management
As in any other scientific setting, it is mighty important for experiments to be
repeatable under the same conditions. We want to make sure immutability of
inputs is preserved through transformations. Versioning and artefacts management
tools play an invaluable role in the development of Data Science solutions. This
goes hand in hand with practices discussed in points 2) and 3), and falls
naturally from there.

6. Metadata management
Often the journey is just as valuable as the destination. As much as possible
(and to the extent that is useful, of course!) we want to be able to retain key
information about the process at hand, so that we can better interpret, analyse
and enhance its outputs. The spectrum here is broad, from basic logging to
systematic capturing and analysis of metadata. The point is: we want to be able
to fulfil this function in the way that maximises the insights offered by the
data being collected. 
Metadata management is at the core of many activities that are either routine
(eg compare performance of models), or required for production (eg backout
strategy for a failing data pipeline), or further reaching into the realms of
ethics and accountability (eg quantitative justification of a Machine Learning
prediction). It is important we do it well.

Conclusions
I fully appreciate some of the recommendations above may feel daunting to the
more traditional Data Scientists, due to such activities not (yet!) being
typically associated with their profession. To some extent, we witnessed
something similar in the past with network and infrastructure specialists, now
evolved in DevOps that design and build infrastructure solutions through code,
fully embedded in the fabric of development teams from the onset, sharing tools
and methodologies. 
Similarly, the key for enabling the evolution of Data Science from R&D practice
to fully fledged enterprise function, lays in bringing data scientists and
developers closer: before long the distinction will blur, and specialist
knowledge will thrive on a common substrate of working practices. 

At the end of the day, remember, Data Science is about building software.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. We deliver practical data science implementations for both our own SaaS products, and for our clients' bespoke platforms and products, for the analysis of data and media at scale.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. Find updates, News and insights on Data Science here.

Data Science - our Work, Features, News and Updates

Data Science

Paul Wilton

The problem: flaky connections cause upload of large files to S3 to fail
Uploading files to S3 securely direct from a browser is a common pattern. There
is plenty of information available on how to do this. Typically most examples
documented online use standard S3 signed POST requests (either from direct from
a html form or via an XHR request). This often means using some server side API
call (which you need to build) to generate an AWS signed request, which can then
be used in your web app client side. While being fairly easy to implement, for
very large files a flaky connection or a dropped packet often means a failed
upload and retries. What you need is multipart, chunked uploads that are
resilient. However this is not trivial to achieve with S3.

The solution: secure chunked multipart managed uploads
If you use the AWS SDK there are essentially two routes. Use the S3 REST API and
manage file chunks yourself. This is the low-level approach and is complex.
Alternatively you can use the AWS SDK Managed Uploads API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property]
, in which the SDK does all the hard work for you to manage file chunking,
upload, resilience, and the reconstruction of the file parts into a single S3
file at the end. However to use this you have to supply AWS credentials, and
this is not something you typically want to expose client side. However, if your
upload feature is behind some oAuth or OpenID authentication layer (as in my
case, and I suspect is often the case if you are allowing uploads to your
backend) then you are in business. You can setup AWS to work as OpenID Connect
provider. Finally you need to integrate this with some client side file uploader
library, such as Dropzone [http://www.dropzonejs.com/]. I am going to
demonstrate step-by-step how to :

a. Setup an S3 bucket to receive the uploads with a suitable access policy and
CORS settings.
b. Setup AWS as an OpenID Connect provider for some external oAuth service - in
this case Auth0 [https://auth0.com].
c. Integrate AWS S3 Managed Uploads securely with Dropzone to handle multiple
parallel resilient uploads for very large files, with progress bars, and cancel.

Step 1 - Create an S3 bucket
In your AWS tenancy, create an S3 bucket (e.g. my-bucket) and a CORS policy as
follows.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-meta-custom-header</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>Authorization</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>


You can revoke all public access to the bucket, as we will be writing to the
bucket using an OpenID Connect user that has assumed an IAM Role with the
correct policy to access the bucket.

Step 2 - Create an OpenID Connector Provider in AWS for Auth0
This allows us to use AWS as an intermediary for validating JWT OpenID tokens
against an oAuth service, demonstrated here with Auth0.
Your web app users will authenticate (login) with Auth0 - The oAuth response
from Auth0 authorize endpoint delivers an OpenID JWT token to the browser, which
you typically store in localStorage for securing subsequent requests from the
client to your backend.
To setup the AWS OpenID Connect provider you need the oAuth service root
endpoint, and the oAuth application clientID. In Auth0 login to your dashboard
and in your application settings you can get this info.

The oAuth endpoint is : https://{your-auth0-domain}/ (note the trailing slash),
e.g. https://appname.auth0.com/

The application Client ID is a 32 character Base62 encoded string, e.g. 
6sR4LF3UpznU4T26S3xRheGhjEKdni4H

Now login to your AWS tenancy, navigate to IAM account management > Identity
Providers. Click on Create Provider, and enter the Auth0 details from above as
follows, ensuring you add the trailing slash on the provider URL:

Then click Next and Create the provider.

Step 3 - Create an IAM Role for the OpenID Provider
Navigate to Policies in IAM, and create a new Policy as follows using the JSON
editor, setting the correct name for {my-bucket}:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::{my-bucket}",
                "arn:aws:s3:::{my-bucket}/*"
            ]
        }
    ]
}


Note, this policy will grant the authenticated user assuming the role all
actions on the bucket.. you may want to make this more restrictive. You could
also add other AWS grants here too, so your user can access other AWS services
if need be. Name your policy to something sensible, eg OpenIdS3Policy.

Now navigate to IAM Roles, and create a new Role. Choose Web Identity as below
and choose your OpenID Provider you just created, and the Auth0 Client ID as the
audience in the drop down options :

Next, Set the permissions of the role by choosing the OpenIdS3Policy created
previously. It is easy to find your policy by selecting policy type Customer
managed from the drop down. Save the Role with a suitable name eg OpenIdS3Role.

Step 4 - Grant access to the S3 bucket
Now we need to grant access to the S3 bucket for the IAM Role just created.
Copy the ARN for the Role, and navigate to your bucket setttings, and choose 
Bucket Policy. Add a Policy as follows, using the ARN for the role, and the
correct bucket name :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permissions",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:GetBucketLocation",
          "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}"
    },
    {
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}/*"
    }
  ]
}


That is all the AWS work needed. An authenticated OpenID Connect client can now
access your S3 bucket by assuming the role of an IAM user. When the AWS
connection is made, AWS authenticates the supplied JWT token against Auth0 using
the AWS Identity Provider. Thus a user has to be logged into an Auth0 session to
be able to access the bucket. AWS matches the aud value in the decoded JWT token
with the trust relationship condition specified in the role we created. This is
where the all important trailing slash I mentioned comes into play. This is an
artefact of using Auth0 as an OpenID identity service. Auth0 adds a trailing
slash to the aud value in the JWT token it generates. If we dont add the same,
AWS cannot match the role with the token when it tries to assume the identity of
the connecting user. If you use a different oAuth provider than Auth0 you may
not need this trailing slash. Think of it as an Auth0 'gotcha'.

Step 5 - Integrate with Dropzone
Dropzone [http://www.dropzonejs.com] is one of the better, and widely used open
source file upload JS libs. It is very customizable and mostly event driven so
is easy to modify. Out of the box Dropzone uses XHR requests to send files to a
remote server. We are going to override Dropzone's file uploading mechanism to
use the AWS SDK Managed Upload API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property] 
instead.

First, install (or make available) the AWS SDK in your web app, e.g.

npm install aws-sdk --save


Now define an accept callback function, that Dropzone invokes when a file has
been accepted for upload. This lets us add an S3.ManagedUpload instance to the
file object being queued by Dropzone for upload :

import S3 from 'aws-sdk/clients/s3';

function acceptCallback(file, done) {
  // options for the managed upload for this accepted file
  // define the bucket, and the S3 key the file will be stored as
  var params = {Bucket: 'my-bucket', Key: file.name, Body: file};
  // add an S3 managed upload instance to the file
  file.s3upload = new S3.ManagedUpload({params: params});
  done();
}


Now we need to override the Dropzone uploadFiles function to upload via the AWS
SDK instead of XHR. Note we are using the file.s3upload instance we added to the
file object when the file was accepted by Dropzone. We emit Dropzone events on
success and failure, and we also listen to the AWS httpUploadProgress event, and
emit the equivalent Dropzone event, so Dropzone can render upload progress for
each file :

// override the uploadFiles function to send via AWS SDK instead of xhr
Dropzone.prototype.uploadFiles = function (files) {
  for (var j = 0; j < files.length; j++) {
    var file = files[j];
    sendFile(file);
  }
};

function sendFile(file) {
  file.s3upload.send(function(err, data) {
    if (err) {
      theDropzone.emit("error", file, err.message);
    } else {
      theDropzone.emit("complete", file);
    }
  });

  // listen to the AWS httpUploadProgress event, and emit an equivalent Dropzone event
  file.s3upload.on('httpUploadProgress', function(progress) {
    if (progress.total) {
      var percent = ((progress.loaded * 100) / progress.total);
      theDropzone.emit('uploadprogress', file, percent, progress.loaded);
    }
  });
}


If we want our users to be able to Cancel uploads mid-progress we need to
implement an abort function that can be invoked on the Dropzone canceled event :

function abortUpload(file) {
  if (file.s3upload) file.s3upload.abort();
}


Finally you need to initialize your Dropzone on some DOM element on your page
and
setup AWS Credentials for the S3 managed uploads. To do this we create an 
AWS.WebIdentityCredentials object using the ARN of the IAM role we created in
Step 3, and also inject the JWT ID token from your local storage/session - the
token returned by the Auth0 client side authentication workflow. When we
initialize Dropzone we also set up the accept and canceled event callbacks.

import AWS from 'aws-sdk/global';
import S3 from 'aws-sdk/clients/s3';

function initializeDropzone(dropzoneElement) {

  var idToken =  'your-jwt-openId-token'; // get from local storage/session
  AWS.config.credentials = new AWS.WebIdentityCredentials({
    RoleArn: 'arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role',
    WebIdentityToken: idToken
  });

  theDropzone = new Dropzone(dropzoneElement, {
    autoQueue: false, // if you want to manually trigger uploads
    accept: acceptCallback,
    canceled : abortUpload,
    init: function () {
      this.on('removedfile', function (file) {
        abortUpload(file)
      })
    }
  })
};


All done. In these examples, I have shown the minimum you need to do to get S3
Managed Uploads working with, for example, the Boostrap Dropzone demo here
[http://www.dropzonejs.com/bootstrap.html]. There are of course lots more
features of Dropzone you can implement, but this should be sufficient to for you
to build fully resilient secure S3 uploads using Dropzone and OpenID
authentication for very large files.

Using S3 Managed Uploads with Dropzone and OpenID to support very large file upload

JSON-LD has been among us for several years now, its adoption is increasing, but
I suspect not as fast as it probably should. JSON-LD arose out of the RDF
community, subsequently adopted as a W3C standard, as a pattern for representing
RDF and linked data (hence 'LD') as JSON, with a key aim of making RDF much
easier to consume for developers.

While Google has raised awareness of JSON-LD considerably as it’s preferred
pattern for marking up web pages with schema.org [https://schema.org] structured
data, JSON-LD however has only made relatively small inroads into mainstream web
development. There are number of reasons for this, not least that the W3C and
JSON-LD working group did not do a great job of communicating its worth and
explaining how to use it effectively, although this has improved a lot more
recently, the excellent (and long overdue) JSON-LD API best practices guide
[https://json-ld.org/spec/latest/json-ld-api-best-practices/] published in
February 2018 is a very good example.

I am going to attempt to help the cause too, and make a convincing case of why
JSON-LD should be used in mainstream software development, regardless of whether
you are working with RDF.

A Very Brief Overview
I want to avoid making this post a full-on lesson about JSON-LD and RDF, so if
you are not entirely sure what JSON-LD is, here is super brief 101.
Alternatively get stuck into the full JSON-LD 1.0 specification here
[https://www.w3.org/TR/json-ld], and the draft 1.1 spec here
[https://json-ld.org/spec/latest/json-ld/].

Roots in RDF :
JSON-LD is a W3C standard for representing RDF [https://www.w3.org/RDF/] in JSON
format.
It is a standard pattern (schema) for representing the key elements of RDF
resources and the statements about them, such as the resource identifier (the
IRI) of the object being represented and the rdf type (class) of object being
described. For example, you will often see snippets of JSON-LD looking like this
:

{
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple"
}


If you are familiar with RDF, you will be aware that most RDF formats typically
involve a proliferation of IRIs/URIs everywhere. JSON-LD however alleviates this
by abstracting base URIs into a context. For the example above, if this was
written without a context in valid JSON-LD it would look more like this :

{
  "@id": "http://data.example.com/organizations/1",
  "@type": "http://data.example.com/some-model/Organization",
  "http://data.example.com/some-model/name": "Apple"
}


While verbose and ugly, in RDF this can be represented as, or transformed into
long form statements (triples) like this :


<http://data.example.com/organizations/1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.example.com/some-model/Organization> .
<http://data.example.com/organizations/1> <http://data.example.com/some-model/name> "Apple" .



You can test the validity and see how JSON-LD can be expanded into RDF using the 
JSON-LD playground here
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22%40type%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%3A%22Apple%22%7D]
.

The Context :
Introducing the ‘context’ into the JSON-LD makes this much more palatable. The
context defines the fields in the data payload :

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple"
}


In this example, the original data from my first snippet, the type and name
properties of the resource are explicitly identified in some schema or ontology,
so as a consumer of this JSON you are aware of the model that this data was
described with. Without the context the JSON cannot be transformed to RDF
statements. As a consumer of course you can choose whether to ignore the
context, or use it depending on your own need.

This same JSON-LD is often written such that that the context and payload are
separated using the @graph construct as follows. This is in fact identical data
to that above, it just identifies the data payload and the context that
describes it as separate JSON objects.

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


Back in RDF world, this also lets us create a named-graph which would then allow
us to generate RDF quad statements instead of triples through the addition of an
identifier for the graph :

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@id" : "urn:my:organizations:1",
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


Which if you try this
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40context%22%3A%7B%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%7D%2C%22%40id%22%3A%22urn%3Amy%3Aorganizations%3A1%22%2C%22%40graph%22%3A%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22%40type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%7D%7D] 
in the json-ld playground will give you RDF statements :

<http://data.example.com/organizations/1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.example.com/some-model/Organization> <urn:my:organizations:1> .
<http://data.example.com/organizations/1> <http://data.example.com/some-model/name> "Apple" <urn:my:organizations:1> .


To make this more concise the context does not need to be explicit in each
payload, it can be remote, so a consumer can choose whether to dereference it or
not :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id" : "urn:my:organizations:1",
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


This also means contexts can be cached for higher performance.
So there you have it, RDF in JSON form. JSON-LD has brought RDF out of the labs,
and made it presentable with a highly consumable JSON representation for
developers.
Or has it...

Mythbusters
Myth 1 - frequently spoken by developers : “it’s not like normal JSON”
While the introduction of JSON-LD is undoubtedly a massive deal for the RDF
development community, it has not been widely embraced by software developers.
Most developers will not have heard of RDF, and even if they are consuming
JSON-LD from some RDF or Graph Database backend, they won’t (and probably don’t
need to be) concerned with it.

A key design goal
[https://json-ld.org/spec/latest/json-ld/#design-goals-and-rationale] in the
JSON-LD specification is :

> Zero Edits, most of the time - “JSON-LD ensures a smooth and simple transition
from existing JSON-based systems. In many cases, zero edits to the JSON document
and the addition of one line to the HTTP response should suffice“


However, when developers first come across JSON-LD it is rarely like this. The
majority of published JSON-LD is littered with JSON-LD nuance. Developers do not
like consuming many of the JSON-LD constructs, and when you hear the complaint
“it’s not like normal JSON” it is typically for good reason. For example, two of
the most common complaints are :

The "@" prefixes :
The “@” prefix on the JSON properties is not a valid ES5 or ES6 prefix for
javascript variables, meaning when dereferencing JSON as a JS object, you cannot
do things like

const id = myObject.@id ;

Instead you need to code

const id = myObject['@id']

Arrays vs Singletons :
JSON-LD output from some SPARQL query or RDF transformation typically does not
confer any indication if a property is an array or a singleton. The result of
one request to some back-end API serving JSON-LD might give :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple",
  "employs": {
    "@id" : "http://data.example.com/people/1",
    "@type": "Person",
    "name" : "Tim Cook"
  }
}


A future request may return an array for the employs property :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "@id" : "http://data.example.com/people/1",
    "@type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "@id" : "http://data.example.com/people/2",
    "@type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


The software developer consuming this output then has to code around this when
evaluating the value of the employs property, and check whether the property
contains an array or a singleton.

I imagine these are possibly the two most common JSON-LD complaints from devs.
Even Google publish their schema.org JSON-LD with @id and @type field names on
the Google Knowledge Graph API [https://developers.google.com/knowledge-graph/].
This doesn't help, and yet it doesn’t have to be like this. Instead of telling
developers to “get over it”, as publishers of JSON-LD we can make JSON-LD look
and behave like normal JSON using the context, meeting the design goal of the
specification. However, it is not always obvious how to make this happen, here
are some simple examples on how to address the two common issues above.

Addressing the “@” issue
Use context aliasing to abstract the “@” symbol away from the data payload. By
creating aliases in the context for those properties prefixed with @, they can
be appear in the data payload fully JS compatible. The context does not just
allow us to provide definitions of our domain-specific JSON properties, but we
can also alias the JSON-LD standard fields. Try this
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40context%22%3A%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%7D%2C%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%2C%22employs%22%3A%5B%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F1%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Tim%20Cook%22%7D%2C%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F2%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Jony%20Ive%22%7D%5D%7D] 
in the JSON-LD playground :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : "http://data.example.com/some-model/employs"
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "id" : "http://data.example.com/people/2",
    "type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


Handling collections :
Use the context to define which fields are arrays, and output them as such :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : {
      "@id" : "http://data.example.com/some-model/employs",
      "@container" : "@set"
    }
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "id" : "http://data.example.com/people/2",
    "type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


Forcing the set (or list) property to be marshalled as an array in JSON even
when it is a singleton. See this in action
[https://json-ld.org/playground/#startTab=tab-compacted&json-ld=%7B%22%40context%22%3A%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%7D%2C%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%2C%22employs%22%3A%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F1%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Tim%20Cook%22%7D%7D&frame=%7B%7D&context=%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%2C%22%40container%22%3A%22%40set%22%7D%7D] 
in the JSON-LD playground :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : {
      "@id": "http://data.example.com/some-model/employs",
      "@container": "@set"
    }
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
    }
  ]
}


Using a remote context (that most web developers can safely ignore), we now we
have JSON-LD that looks just the way your developers would prefer, and meeting
the design goal of the JSON-LD specification :

{
  "@context": "https://data.example.com/contexts/organization", 
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
      "id" : "http://data.example.com/people/1",
      "type": "Person",
      "name" : "Tim Cook"
    }
  ]
}


Myth 2 - “JSON-LD is only needed if you are working with RDF or Graph Databases”
If you are building a web application or micro-service which is using RDF, then
JSON-LD is clearly a no-brainer. However if you don’t care about RDF why bother
?

I argue that you should bother. If you are publishing an API you may not know
how your consumers are using your data. They may well be using a graph database,
or dealing with RDF or linked data and be very keen to consume your API in an
RDF compatible format. Furthermore, publishing JSON-LD requires you to publish a
context. You are thus constraining your JSON to a schema. This is a good thing.
It communicates a model of your domain, provides a payload contract for your
consumers (or a contract between your own micro-service APIs and applications),
and a validation mechanism.

Why we should all adopt JSON-LD
JSON Entropy
Being inherently a schemaless message format, applications built with components
that use JSON to serialize/deserialize data moving between them have a tendency
over time to create JSON property soup. As requirements evolve, and developers
are left to their own devices to modify features and adapt the codebase,
properties are introduced into JSON on an ad-hoc basis. JSON entropy increases.
What may have started out as a simple, clean JSON model, tends towards one of
mess and complexity.
Adopting JSON-LD as your message format reduces this entropy escalation, as :

a) it makes you as a developer think about the schema (structure) of the JSON
from the perspective of your consumers, and the wider application as a whole.
b) once we start publishing schemas for our data, we start to think like a data
architect rather than as a developer.
c) we are more likely to collaborate on the schema design, and at the very
minimum, ask for a peer review of a proposed change.

While this may seem like an overhead, it is a relatively small investment into
your data architecture that pays dividends over the long term.

Schema
The JSON-LD context is a schema for your data, not only defining the property
datatypes but also the classes of json resources and, via referenced ontologies,
the semantics and relationships between properties and classes in your data.
The json-schema [http://json-schema.org/] project has attempted to fill this
space for JSON, but has not really gained too much traction. While it is a
little richer in schema-definition features than what a JSON-LD context
provides, JSON-LD is an official W3C standard, and offers a lot more than
json-schema can with respect to linked data and RDF compatibility.

We all can publish linked data
If you are using linked data or RDF, then you will no doubt already be familiar
with JSON-LD, and hopefully using JSON-LD as a message format in your own
applications. If you are publishing linked data then it is very likely you are
publishing it as JSON-LD. If you are serving public APIs as regular JSON, then
publishing JSON-LD instead makes a lot of sense. As discussed, you should be
able to transition with little or zero change to your data payload. All it takes
to make the transition is to publish a context. For little additional overhead,
you and your consumers benefit. It is essential however, that we, as publishers
of JSON-LD make it as consumer friendly as possible (and this includes Google).

Software Development

Software Development Blog Posts