Steve Liles

Maps in web applications are a pretty common feature these days, and with
amazing libraries like Leaflet [https://leafletjs.com/] its now pretty easy to
build a great mapping experience.

Typically these maps use the Web Mercator
[https://en.wikipedia.org/wiki/Web_Mercator_projection] projection, a variant of
the Mercator projection most people are familiar with from paper maps and which
has been in common usage for hundreds of years.

The Mercator projection takes our spherical world and maps (projects) it to a
rectangle, which is very convenient for viewing on a rectangular sheet of paper
or screen, but produces a distorted view of the world that inflates the size of
objects near the poles.

Greenland is NOT actually the same size as the entire African continent!
Likewise, things near the pole that are close together in reality can appear to
be far apart, and it is very difficult to visualise the proximity of objects in
the Arctic Circle when viewed on Mercator projected maps.

Luckily, it is possible to render maps on the web using alternative projections,
although it can be tricky to get up and running. In this post we’ll have a look
at setting up an Arctic projection with React, Leaflet, and Proj4js
[http://proj4js.org/] (via Proj4Leaflet
[https://kartena.github.io/Proj4Leaflet/]).

First lets get a map up and running with the default Web Mercator projection,
then we’ll walk through the modifications necessary to display an Arctic
projection.

Mapping with React + Leaflet
Getting started with the default projection is straight-forward — we just need
to set up a container for our map and render a TileLayer with a suitable tile
provider url. As we’re just testing we’ll use tiles from openstreetmap using the
following url pattern.

const TILES = https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png


The code to render the map with React-Leaflet is short and sweet:

import { MapContainer, TileLayer } from "react-leaflet";

const App = () => (  
  <MapContainer
    style={{
      width: 1000,
      height: 1000
    }}
    center={[0, 0]}
    zoom={2}
    minZoom={1}
  >      
    <TileLayer url={TILES} />
  </MapContainer>
)

export default App;

That’s all we need to render a world map in Mercator projection. Here it is in 
codesandbox [https://codesandbox.io/s/mercator-projection-s1v59]:

Note: it is important to give your map a size (width & height) otherwise you
won’t see anything!Now that we have a map rendering in Mercator projection, lets
convert it to a polar projection, specifically the Arctic Lambert Azimuthal
Equal Area projection (Arctic LAEA for short). To do that we need to provide
Leaflet with two things:

 1. a coordinate reference system (CRS) to enable it to project and unproject
    coordinates, and
 2. a set of tile images are appropriate for that CRS

The CRS implementation comes courtesy of Proj4Leaflet, and must be parameterised
with a proj4 configuration string and details of the map tile bounds and
resolutions.

Proj4 configuration
We can find the proj4 configuration string at epsg.io [https://epsg.io] simply
by searching for the projection we want to use. There are several Arctic LAEA
projections we could use, I’m going to go ahead with epsg:3575
[https://epsg.io/3575]. The configuration string we need can be found under the
section titled “Export” for the proj4js link, which gives the following:

proj4.defs(
  "EPSG:3575",
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"
);

Map Tile Configuration
Next, we’ll need a compatible set of tiles for the EPSG:3575 projection. I’m
going to use tiles hosted by the Global Biodiversity Information Facility
[https://tile.gbif.org/ui/]. The map tile url pattern is given on that page as:

https://tile.gbif.org/{srs}/{tileset}/{z}/{x}/{y}{format}{params}

Here, {srs} is the spatial reference system (in our case 3575), {tileset} can
only be omt, {format} allows for a choice of vector or various resolutions of
png (i’ll use @4x.png), and {params} allows us to choose a tile style - only for
png tiles - where i’ll use ?style=gbif-classic, which altogether gives us a tile
url like this:

https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en

We need to provide the CRS with a little configuration related to the map tiles,
which we can derive from the extents provided by Global Biodiversity Information
Facility (see the table in the usage section). The tiles are 512x512 and the
provided extents for the northern hemisphere are:

–6371007.2 √2, 6371007.2 √2, 6371007.2 √2, –6371007.2 √2

So we can derive the extent and resolutions like this:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1)  
  .fill()  
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));

Complete Proj4Leaflet Configuration
Putting it all together we get the following:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1) 
  .fill() 
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));
  
export const ARCTIC_LAEA = new Proj.CRS(
  "EPSG:3575", 
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs", 
  { 
    origin: [-extent, extent], 
    bounds: L.bounds(
      L.point(-extent, extent), 
      L.point(extent, -extent)
    ), 
    resolutions: resolutions 
  });
  
const ARCTIC_TILES_URL = 
  "https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en";

All that remains is to update MapContainer to refer to the new CRS and center on
the pole.

crs={ARCTIC_LAEA}
center={[90, 135]}

... modify TileLayer to use the arctic tiles ...

url={ARCTIC_TILES_URL}

And we’re done! Check out the complete working code on codesandbox
[https://codesandbox.io/s/react-leaflet-arctic-bxw7d].

Boom! Arctic Map!Welcome to the Arctic!

Software Development

Arctic Projections with Leaflet and React

In this mini-series we’re looking at rendering interactive graphical UI’s using
components built with React and SVG. Throughout the series we’re putting
together an interactive floor-plan while working through SVG concepts, starting
from the basics and working through some more complicated areas.

If you’re new to React and/or SVG, or you just want to get some context on our
ongoing project, you might want to start from the beginning with part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics]
, Declarative Graphics, where we laid out our basic floor-plan blueprint.

Inpart 2
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations] 
we added cameras to our plan and showed their field of view, but there’s a
problem: the camera field of view passes through walls and around corners,
giving a false impression of what the camera can see.

In this, part 3, we’ll use clip paths to constrain the field of view to the room
that the camera is in, preventing it from overflowing into other rooms or
outside the floor-plan entirely.

Our cameras can see through walls and around corners!Boundaries
Let’s start with the easy bit: anything in the room should respect the room’s
boundaries. We already have geometry for the room that defines the constraints
that we want to apply to anything in the room, so how can we apply those
constraints to prevent overflow?

The answer, of course, is clip paths. A clip path (defined with theclipPath 
element) is just what it sounds like: a path that clips any elements to which it
is applied, such that only content that falls inside the defined path is
rendered.

In SVG we can compose clip paths from any drawable SVG elements, e.g. circle, 
rect, polygon, path, and even text!

Binocular Vision
For a simple but fun first example, let’s bring back the Pac-Man ghosts from 
part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics] 
and use a clip-path to pretend we’re looking at them through binoculars.

First, here’s the scene without clipping, showing our four ghosts on a blue
background:

<g>  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />
  <Ghost x={400} y={350} scale={2} colour="red" />
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

un-clipped ghostiesNow let's create our first clip path to constrain the drawing
to two adjoining circles in the old movie trope binocular shape.

<clipPath id="binoculars">
  <circle cx={500} cy={400} r={200} />
  <circle cx={800} cy={400} r={200} />
</clipPath>

A couple of things worth highlighting here:

 1. As you can see, we’ve given the clipPath an id, which is necessary in order
    to apply the clip path to another SVG element.
 2. We didn’t stroke or fill the circles — we don’t need to — because clipPath 
    doesn’t pay any attention to stroke and fill, it simply takes the entirety
    of the shapes added to it and allows content to be rendered if it falls
    within those shapes. In that sense, clipPath is all-or-nothing.

Now all we need to do is set the clipPath attribute on the element we want to
clip with the valueurl(#binoculars). In our case we just need to add that to the 
<g> element containing the ghosts.

<g clipPath="url(#binoculars)">  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />  
  <Ghost x={400} y={350} scale={2} colour="red" />  
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

The black background of the SVG shows through outside the clip pathWhat goes on
in the room, stays in the room
Back to the floor-plan. Our Room component currently renders as a group<g> 
containing walls, corners, and cameras (and I added the room name as red text in
the corner of each room). Here’s the current code and its output:

The room component from part 2 - walls, corners, cameras, and labelsWe want to
modify this component so that the contents of a Room can no longer render
outside the bounds of the room. To do that we want to do two things:

 1. Change the render order such that the contents of the room (currently really
    just the camera’s) are drawn first, and therefore are rendered beneath the
    walls and corners (in z order).
 2. Render the room contents inside a <g> element that is constrained by a clip
    path along the room’s boundaries, so that contents cannot overflow the room.

Step 1 just means pulling the camera’s up to be the first thing rendered inside
the Room’s <g> element, but why do we even need step 1? Well, our walls and
corners are rendered along the boundary of the room, but they have some
thickness and so half of the thickness of the wall falls inside the boundary of
the room.

The clip path will prevent contents of the room from being rendered outside the
boundary, but would not stop them from rendering over the inner part of the
walls, so this is a simple workaround to prevent that. A more satisfying — but
more complicated — alternative would be to adjust the clip path to account for
the wall thickness, but for now we’ll keep things simple.

Room geometry as clip path
Having pulled our camera’s up to the first-child position in the room, we can
nest it in a new group <g> element, to which we will later apply our clip path.

<g>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} />  
  ))}
</g>

Now let’s create the clip path. We need to give it an id so that we can refer to
it from the clipPath attribute of the camera group element. Our room data
includes an id so we can use that to make up the clip path id, e.g.

<clipPath id={`${id}-content-clip`}></clipPath>

To define the shape of the clip path we need to use the room boundary
coordinates to construct a solid shape. Our room geometries are just a series of
coordinates with no flashy curves or arcs, so while we could use the
all-powerful path element we have a simpler alternative in the polygon element.

The SVG polygon element gives us a convenient way to convert our coordinates
into a rendered shape: polygon takes a points attribute, which is a
space-separated list of comma-separated coordinates, like this:

<polygon points="20,20 40,30 30,30 ..."/>

This is perfect for us, because we can simply map our room coordinates array to
a string for the points attribute and we’re done!

<clipPath id={`${id}-content-clip`}>  
  <polygon points={coords.map(({x,y}) => `${x},${y}`).join(" ")}/></clipPath>

Now all we need to do is add the clipPath attribute to our camera group to apply
the clip-path.

<g clipPath={`url(#${id}-content-clip)`}>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} /> 
  ))}
</g>

Win! The camera fields of view are now clipped to the room boundary!This is
great, with some very simple changes we’ve clipped the content of the room so
that it doesn’t spill outside the boundary!

Lines of Sight
Look closely at the floor-plan image above though, and you’ll see something’s
not quite right. Our cameras can’t see through walls any more, but they can see
around corners!

To solve this we’re going to need the camera itself to clip the field of view
according to its lines of sight. This will involve creating a more intricate
clip-path shape, but most of what follows is just a fun bit of problem solving,
so if you’re not into algorithms feel free to skip to the bottom to see the
results.

The Disjoint Segment Visibility Problem
To clip the camera field of view to what the camera would actually be able to
see accounting for occlusion due to walls blocking lines of sight, we need to
solve the Disjoint Segment Visibility Problem where the segments are our walls.

When dealing with large numbers of segments this is typically done with a
sweep-line algorithm that projects rays out from the view-point (our camera) at
discrete intervals around a full 360 degree sweep, checking for the nearest
intersection with a segment at each interval.

sweep-line algorithm at 10° intervals, red dots show nearest intersection to
cameraOne problem with the sweep-line approach is that the use of discrete
intervals (e.g. 1 degree) means that the precision is only as high as the chosen
interval allows for, and we might produce a field of view that visibly cuts
through corners of walls or misses the corner and leaves a small gap.

corners cut through or cut off due to lack of precision (interval too large)One
way to deal with this would be to use smaller intervals until we get the
precision we want, with the trade-off that the smaller the interval the more
calculations we need to make.

In our case, however, we know that each room is composed of a very small number
of segments (walls) so we can take a different approach: rather than sweep
through 360 degrees, we can check only the angles to known segment end-points,
which reduces the number of calculations we need to make, and simultaneously
allows us to achieve very high precision at little cost.

The Visibility Polygon
Our task, then, is to create a “visibility polygon” describing the critical
lines of sight from the camera position to the nearest walls of the containing
room. We’ll use a modified  sweep-line algorithm that boils down to some pretty
simple steps:

 1. Find the angle to each wall end-point from the camera
 2. Reduce to the set of unique angles
 3. Sort the unique angles into clockwise order
 4. Insert two new angles θ- and θ+ to either side of θ, where θ- is a fraction
    smaller than (CCW of)θ and θ+ is the same fraction larger than (CW of) θ.
 5. For each angle in the resulting list, find the nearest wall intersection and
    add it to our visibility polygon

At step 1, we can find the angle to each wall end-point simply by converting
each coordinate from the Cartesian coordinate system to a polar coordinate
system with the camera as the origin. The angle part of the polar coordinate is
thus the angle to the wall end-point.

Step 2 eliminates any duplicate work, for example where a camera might be
looking directly along the line of a wall such that both wall end-points are at
the same angle, or in more complex room shapes where distant corners might align
with nearer ones along the cameras line of sight.

distant corners that align with nearer ones can be eliminated from further
processingBecause rooms can potentially be composed of complicated arrangements
of walls, Step 3 is important to ensure that the points of our visibility
polygon are arranged clockwise around its boundary to produce a continuous shape
from start to finish.

It doesn’t strictly matter whether we sort now or at the end, but it is more
efficient to do it now because step 4 triples the number of points that would
need to be sorted.

wall coordinates for complex room shapes may not be clockwise ordered from the
camera perspectiveStep 4 gives us the high precision we want — the angles
fractionally before and after the angle to each wall end-point are the critical
line of sight intervals we need to test in order to get very precise results
with no visible corner cutting.

critical line-of-sight angles fractionally before and after the angle to the
wall corner pointsStep 5 converts the collected angles of critical lines of
sight into the Cartesian coordinates of our visibility polygon, by finding the
nearest intersection of the line of sight with a wall.

In javascript the whole thing looks like this:

const getVisibilityPolygon = (from, cartesianCoords) => {
  // wall corners as indexes into the cartesian coords array
  let walls = cartesianCoords.map((_, i) => [    
    i, (i + 1) % cartesianCoords.length  
  ]);  
  
  // get all points in polar coords, giving us angles from camera  
  let polar = cartesianCoords.map((c) => cartesianToPolar(c, from));  
    
  // reduce to the set of unique angles that we need to test  
  const unique = Array.from(    
    polar.reduce((r, { angle }) => r.add(angle), new Set())  
  );  
    
  // sort into clockwise order  
  const clockwise = unique.sort((a, b) => a - b);  
    
  // produce angles slightly CCW and CW of each angle in the list
  const checkPoints = clockwise    
    .map((angle) => [angle - 0.01, angle, angle + 0.01])    
    .flat();  
    
  // find the nearest intersection with a wall  
  return checkPoints.map((angle) => {
    return nearestIntersection(from, angle, walls, cartesianCoords);     
  });
}

For this to work we’ll need to add a few more utilities to our geometry toolkit:
First we’ll need to modify our polarToCartesian function so that we can pass the
camera’s Cartesian coordinates as the origin for the polar coordinates. Second,
we need to add the inverse function cartesianToPolar to convert in the other
direction.

export const REFERENCE_DIRECTION = -90;

export const degreesToRadians = (degrees) => 
  (degrees * Math.PI) / 180.0;

export const radiansToDegrees = (radians) => 
  radians * (180.0 / Math.PI);

export const vector = (from, to) => ({ 
  x: from.x - to.x, 
  y: from.y - to.y
});

export const magnitude = (vector) =>  
  Math.sqrt(Math.pow(vector.x, 2) + Math.pow(vector.y, 2));

export const distance = (from, to) => magnitude(vector(from, to));

export const polarToCartesian = (
  { angle, range }, orig = { x: 0, y: 0 }
) => {  
  let angleInRadians = degreesToRadians(angle + REFERENCE_DIRECTION);
  let cartesian = {    
    x: range * Math.cos(angleInRadians),    
    y: range * Math.sin(angleInRadians)  
  };  
  return { x: orig.x + cartesian.x, y: orig.y + cartesian.y };
};

export const cartesianToPolar = (point, orig = { x: 0, y: 0 }) => {  
  let v = vector(point, orig);  
  let range = magnitude(v);  
  let angle = radiansToDegrees(Math.atan2(v.y, v.x));  
  return { range, angle: angle - REFERENCE_DIRECTION };
};

Finally, we need a functionnearestIntersection that takes a Cartesian coordinate
of a point of interest such as our camera location, an angle, and a list of
segments (walls) to test for intersection at that angle, and returns the nearest
intersection to the point of interest as a Cartesian coordinate.

const nearestIntersection = (from, angle, walls, coords) => { ... }

This is going to come down to finding intersections between line segments, where
one segment is a wall, and the other is the line of sight — an imagined ray
projected out at angle from the camera position — so we will need a function to
test whether and where two segments intersect.

Finding the intersection of two line equations (think lines of infinite length)
is well-known mathematics, where one possible solution in code looks like this:

export const lineIntersectsLine = ([a, b], [c, d]) => {  
  let a1 = b.y - a.y;  
  let b1 = a.x - b.x;  
  let c1 = a1 * a.x + b1 * a.y;  
  let a2 = d.y - c.y;  
  let b2 = c.x - d.x;  
  let c2 = a2 * c.x + b2 * c.y;  
  let determinant = a1 * b2 - a2 * b1;  
  if (determinant === 0) {    
    // parallel lines never intersect    
    return false;  
  } else {    
    let x = (b2 * c1 - b1 * c2) / determinant;    
    let y = (a1 * c2 - a2 * c1) / determinant;    
    return { x, y };  
  }
};

This gives us the intersection point of the two line equations unless the lines
are parallel, and is a step in the right direction, but we need to know that the
point of intersection actually lies somewhere on both finite length sub-segments 
of the line equation (the wall, and the ray from the camera). We can check this
by testing that the intersection point lies within the bounds of both line
segments:

export const MARGIN_OF_ERROR = 1e-9;

const pointIsOnSegment = ({ x, y }, [a, b]) =>  
  x - MARGIN_OF_ERROR <= Math.max(a.x, b.x) &&  
  x + MARGIN_OF_ERROR >= Math.min(a.x, b.x) &&  
  y - MARGIN_OF_ERROR <= Math.max(a.y, b.y) &&  
  y + MARGIN_OF_ERROR >= Math.min(a.y, b.y);

Notice that we’re allowing a tiny margin of error here to account for errors in
floating point math. And now we can write a function that tests whether and
where two line segments intersect:

const segmentIntersectsSegment = (l1, l2) => {  
  let pointOfIntersection = lineIntersectsLine(l1, l2);  
  if (    
    pointOfIntersection &&    
    pointIsOnSegment(pointOfIntersection, l1) &&     
    pointIsOnSegment(pointOfIntersection, l2)  
  ) {    
    return pointOfIntersection;  
  } else {
    return false;  
  }
};

Which gives us everything we need to complete our implementation of 
nearestIntersection as follows:

const LARGE_DISTANCE = 1e9;

const nearestIntersection = (from, angle, walls, coords) => {  
  let to = polarToCartesian({ range: LARGE_DISTANCE, angle }, from);  
  let intersections = walls    
    .map(([a, b]) => 
      segmentIntersectsSegment([from, to], [coords[a], coords[b]])    
    )    
    .filter((intersection) => !!intersection)    
    .map((intersection) => ({      
      intersection,      
      distance: distance(intersection, from)    
    }))    
    .sort((a, b) => a.distance - b.distance);  
  return intersections?.[0]?.intersection;
};

When projecting the ray from the camera i’m using a very large distance instead
of the camera’s range, which will give us a visibility polygon defined as points
along the  walls of the room, regardless whether the camera can see that far.
 This slightly simplifies the algorithm because it means there will always be at
least one intersection for each angle.

Below you can see what happens when we test this with a more complex room shape
and a bunch of different cameras. To illustrate the visibility polygon vs the
camera field of view I have not clipped the field of view — instead I’ve drawn
both the field of view (green) and the visibility polygon (red) with some
transparency so that the intersection (orange) and difference are clearly
visible. Once the clip is applied, only the orange part will be visible.

When clipped to the visibility polygon, only the orange part of the field of
view would remainFeeding this back into our Camera component is straight-forward
and gives us the following:

Overlapping fields of view, clipped along lines of sightWhew, that got a little
heavy, but we now have a clean, simple, and fast mechanism for finding the
visibility polygon from any given point in a room, and we can use that
visibility polygon as a clip-path to constrain the camera field of view in a
realistic manner.

Recap
In this post we learned what clip-paths do, how to compose them from arbitrary
SVG shapes and paths, and how to apply them to elements or groups to constrain
what gets drawn.

We then took a bit of a dive into some fun algorithms that once again take
advantage of alternative coordinate systems to make our lives easier.

In part 4 we’ll look at masks — a more sophisticated method of constraining the
rendered output that gives us more control than clip-paths.

Graphical UI's with SVG and React, Part 3 - Clip Paths

Graphical UI’s with SVG and React, part 2 — Arcs, Angles, and Transformations

React and SVG make a powerful combination: Declarative UI component library
meets declarative graphics language — a match made in front-end developer
heaven.

In this mini-series we’ll be looking at rendering interactive graphical UI’s
using components built with React and SVG.

These days SVG icons are ubiquitous, but those are not the sort of components
we’re talking about.

We’re going to put together the interactive floor-plan UI using simple React
code that renders SVG elements instead of HTML. We’ll take it relatively slowly,
building through the basics into more advanced topics.

 * Part 1 — Declarative Graphics (this post)
   In which we look at the match made in heaven that is React and SVG
 * Part 2 — Arcs, Angles, and Transformations
   [https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations]
   Keeping geometry simple with groups and transformations
 * Part 3 — Clip-paths
   An adventure in composite geometry
 * Part 4 — Masks
   When clip-paths don’t cut it: masks to the rescue
 * Part 5 — Animating components 
   Where we animate individual components
 * Part 6 — Animating the viewBox 
   In which we fluidly zoom and pan our entire UI

Let’s dive straight in!

Declarative Graphics
React developers enjoy first-class support for HTML elements in JSX:

const SomeComponent = () => (  
  <div>
    <p>hi!</p>
  </div>
);

But did you know that this applies to inline SVG elements too?

const SomeOtherComponent = () => (
  <svg viewBox="0 0 100 100">
    <circle cx={50} cy={50} r={30} fill="red"/>
  </svg>
);

Out of the box we can write  components that compose SVG primitives to create
complex interactive UI’s, just as you would with more typical HTML based
components.

(Note for React-Native folks: you’ll need to install react-native-svg, which was
looking pretty good last time I worked with it. Here
[https://stackoverflow.com/a/39041997/473338] is a simple but pretty powerful
example I created a few years ago).

Simple composition of declarative graphical components (https://62zqd.csb.app/)
Notice that we did all this drawing without a single line of imperative code,
and the declarative markup of our ghosts is very readable and accessible to
anyone familiar with React — a far cry from what this would look like if we
tried to do it with the Canvas API.

All your elements are belong to us
To draw anything with SVG, we first need an <svg>element to define the drawing
context within which all descendant elements are rendered.

We can have many <svg> elements in our page, each describing an isolated drawing
context — e.g. one <svg> per icon, chart, or other snazzy graphical widget.

Alternatively, we can root many graphical components within the same <svg> and
create an entire UI in a single drawing context.

Viewport and ViewBox
At a minimum an <svg> element should specify how much screen-space it will
occupy — its viewport. This can be done with CSS styles or width and height
attributes.

<svg width="200px" height="200px">...</svg>

We’ll typically want to specify a viewBox too, to explicitly define the
“user-space” portion of our drawing that is made visible in the viewport.

Think of the viewport as a window into the world defined by the SVG
(“user-space”), and the viewBox as zoom and pan settings that control what parts
of the user-space you can see through the window.

The viewBox attribute is a string which specifies, in order, the user-space x
and y coordinates of the top-left visible corner, and the width and height of
the viewBox.

<svg viewBox="x y width height">...</svg>

For example, “-100 -100 200 200” describes a square viewBox 200 units high and
200 units wide, centred around the origin at (0,0).

<svg width="10" height="10" viewBox="-100 -100 200 200">...</svg>

If we don’t specify it explicitly we get a viewBox that maps width and height
1:1 to the viewport, with the user-space origin (0,0) at the top-left corner.

To re-iterate: the viewport describes how much space the <svg> element takes up
within your HTML page. The viewBox describes the portion of the image that is
visible within that viewport. By changing the viewBox we can zoom in or out, or
show entirely different portions of the SVG image.

The only difference between these two SVG’s is the viewBox and the background
colour (https://0oy6p.csb.app/)If the proportions of your SVG viewport in pixels
don’t match the proportions of the viewBox, the default behaviour is to preserve
the aspect ratio and centre and fit the viewBox within the available viewport.
This behaviour can be controlled by varying the preserveAspectRatio
[https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/preserveAspectRatio] 
attribute.

By default the viewBox will be centred and “fit”within the SVG viewport (
https://52fbc.csb.app/)The separation of coordinate systems between the viewBox
and the viewport means that within our SVG we can work with coordinates that
makes sense for our specific drawing.

For example, if we’re drawing an interactive widget we can make the maths easy
for ourselves by working in percentages and specifying a viewBox of “0 0 100
100”.

Or if our widget has radial symmetry and we want to easily draw from the centre,
we can use a viewBox of, say, “-50 -50 100 100” centred at the origin (0,0).

In this series we’ll be drawing floor-plans, so we could choose to work in
metres, or feet-and-inches, or whatever system of measurement our plans use.
We’re actually going to use millimetres, just for the simplicity of working in
whole numbers.

We’ll describe our floor-plan with some simple JSON, specifying the room
geometry as coordinates of corners working clockwise around the room.

Now we can set up our viewport and viewBox, allowing a little padding around the
floor-plan so it isn’t too snug in the viewport.

import data from './floorplan-data.json';
const App = () => (
  <svg 
    width="500px"
    height="500px"
    viewBox="-1000 -1000 14000 11000"
    style={{backgroundColor:'blue'}}
  >
    // ... floorplan components here ...  
  </svg>
);

With our viewport and viewBox sorted, we can get on with drawing.

Declarative Drawing Primitives
SVG includes a number of drawing primitives ranging from the simple <line> and 
<circle> to the more flexible <path>. Mozilla Developer Network has a good 
reference [https://developer.mozilla.org/en-US/docs/Web/SVG/Element] of the
available elements.

Shapes can be filled and/or stroked, where fill is the colour applied within the
bounds defined by the shape, and stroke is the colour applied to the outline of
the shape. The width and other styles of the stroke can be controlled with
attributes or CSS.

We’ll be drawing our initial floor-plans using two simple primitives: <line> and 
<circle>.

<line> describes a line between a pair of (x, y) coordinates.

<line x1={100} y1={100} x2={200} y2={200} stroke="red"/>

<circle> describes a circle of radius r centred at (cx, cy).

<circle cx={50} cy={50} r={20} fill="green" stroke="black"/>

Room with a View
To draw our building we’ll pass the data into a Floorplan component which will
render out the various elements of the floor-plan — initially just the rooms.

const Floorplan = ({ data: { rooms } }) => (
  rooms.map(r => <Room {...r} />)
);

For the Room component to draw the walls we’ll need to map over pairs of
consecutive corner coordinates, linking them together to form walls. For example
a rectangular room with corners a, b, c, and d, will have four walls
(a-b),(b-c), (c-d), (d-a).

We can pull out those pairs with a simple function, perhaps wrapped in a useMemo 
hook for efficient re-renders.

const walls = useMemo(() =>  
  coords.map((_, i) => {
    const a = coords[i];
    const b = coords[(i + 1) % coords.length];
    return [a, b];
  }),
  [coords]
);

Now drawing the walls is as simple as using the SVG <line> element to describe a
line between the pairs of connected corners.

That works, but it isn’t very pretty. Let’s extract a Wall component and a 
Corner component, and get a bit more creative.

In our wall component we’re now drawing two lines per wall — first a thick white
line, then over-drawing the white line with a thinner, dark blue, dashed line.

Corner is very simple — just a circle, stroked white and filled dark blue.

The Room component now simply composes these higher-level Wall and Corner 
components instead of SVG primitives.

https://codesandbox.io/s/react-svg-basics-rk1t3You may have noticed that we
nested our Walls and Corners in an SVG <g> element. This is a logical grouping
element — the <g> on its own doesn’t render anything visible — but it does give
us a neat way of performing transformations such as rotations and translations
on entire sub-hierarchies of our SVG, and a means of capturing events from those
sub-hierarchies by attaching event handlers to the <g> element.

Recap
In this post we saw that:

 1. We can easily draw arbitrary shapes using SVG primitives and
    straight-forward, declarative React code — no plugins, libraries, or
    imperative code required.
 2. Complex graphical UI’s can be built by composing together increasingly
    higher level components, just like any other React application.
 3. The separation of viewBox and viewport lets you draw in a coordinate space
    that suits you, then render the result in a viewport appropriate to the
    final display requirements.

In “Part 2 — Arcs, Angles, and Transformations
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations]
”, we’ll dig further into those <g> elements and look at paths and
transformations while adding camera’s to our floor-plan.

Until next time, happy drawing!

Graphical UI's with SVG and React,  part 1 - Declarative Graphics

Julian Everett

One the most important tasks for software architects in the early stages of any
technology delivery program is to reach “architecture conceivability”, or
consensus agreement with senior developers that the technical proposition could
actually conceivably be workable. Starting from a position of defensive
scepticism, it typically entails an outside-in narrative exploration of the
potential end-to-end program execution flows at a level of detail between the
Component and Code levels in the C4 Architecture Model [https://c4model.com/] 
for key use cases.

The granularity of the discussion will switch depending on the level of
technical risk entailed by the component. If it is providing a standard
capability that is self-evident to to those present how to code, then there is
no point doing the more detailed code discussion. If it is encoding anything
with a non-trivial level of technical risk, then dropping down to the next level
will be required until everyone is roughly on the same page about how it needs
to work. In this way it acts a bit like a code preview (as opposed to a code
review), functioning in the same way that a delivery prespective
[https://www.malotaux.eu/nrmc.php?id=prespectives] is more efficient than a
retrospective in terms of mitigating risk.

Once and only once this has been completed, you will have established baseline
viability for the program. The narrative described will be unquestionably wrong,
inefficient, full of flaws – that’s not the point. The point is that it will
represent a first, stuttering, throwaway sketch out which a simple, resilient
system design can eventually be evolved. Gaping holes and pointless layers of
cruft will already be visible. Beyond that, most of the selective pressures
driving that evolution will be created later by writing tests and the feedback
from coding itself. 

That’s all fine. However, until you’ve established architecture conceivability,
in reality you’ve got nothing. It is hard to over-emphasise the importance of
this point. Context and Container-level views are meaningless slideware (or at
very best, vague indicators of business process alignment) until they are backed
up by Component and Code-level views. That is solely how they gain their value,
by proxy as maps to decomposing and navigating the the Code-level views of the
platform. Without the code, they are essentially maps to nothing.

Technical Architecture

Architecture Conceivability

Fabio Colasanti

For Data Science to truly live up to expectations, it must embrace software craftsmanship.

Data Science is about building software

Earlier this month I was chatting with my old friend Pierluigi Casale
[https://www.linkedin.com/in/pierluigi-casale-41271430/] (Data Scientist at
TomTom, AI lecturer and researcher) about what seems to be the question of the
century: "What is Data Science"? Tongue in cheek, I offered my very own answer:
"Data Science is that thing you get when someone with a PhD plays with a Jupyter
Notebook, and a technical architect refuses to let it anywhere near production
because it doesn't have a single unit test". 
Although sporting a PhD himself, Pierluigi agreed that my slightly caustic
definition was indeed something he observed himself, and that he too was wary
of. He mentioned how he often concludes the many talks he gives at conferences
around the world with a diagram from the notorious paper "Hidden Technical Debt
in Machine Learning Systems
[https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf]
" by D. Scully et al.


There is indeed a widespread assumption that Data Science and Machine Learning
belong into a hazy limbo somewhere between data analysis, analytics, and R&D.
Within this mystical space, a set of practices otherwise regarded as inefficient
seem to be accepted as the norm: manual mining and manipulation of data; one-off
transformations virtually impossible to replicate; casual scripting of code with
total disregard of any development standard; artefacts laying around and code
running here or there...
When Scully talks about "glue code" and "plumbing", he is pointing at the right
problems but possibly underplaying the issue by being too humble about it. So
forgive my bluntness and let me put that straight: Data Science is about
building software.

Whether the context is academia or enterprise, research or commercial alike, the
adoption of Software Development best practices, tools and methodologies is the
key for efficient delivery of outcomes. At Data Language we have worked with a
number of clients at different levels of "enlightenment", and witnessed first
hand how injecting software craftsmanship into Data Science can lead to
profoundly transformational effects. We can also testify how the development of
our Tagmatic [https://datalanguage.com/tagmatic] service has been indeed 10%
Machine Learning and 90% software engineering: not much we could have done with
our shiny model without things like containerisation, high availability, high
throughput, load balancing, scaling, automation, security, governance,
monitoring, and many others. 

The adoption of established best practices in Software Development will boost
productivity, shorten the feedback cycles, and produce more robust code that is
easier to change, maintain, and ship.

Six things we at Data Language do and you should too
1. Ditch the darn notebooks
I'm fully aware this is highly controversial bordering sacrilegious for many,
but the truth is these type of editors foster bad habits and are holding you
back. They are like comfy, safe, cute trikes you got used to riding when you
were three; but if you actually want to go somewhere other than your garden,
you’ve got to move on and get yourself a bike. 
At this year's JupyterCon in NY, the talk widely regarded as best of the lot was
"I don't like notebooks.
[https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68282]"
by Joel Grus. I wholeheartedly invite you to check it out.

2. Version control
A common refrain in favour of notebooks, is that they enable collaboration.
Oddly enough, when further engaged on this topic and asked about which version
control tool they use, what's their branching strategy, how often they merge,
etc... it all unravels. 
Collaboration is mighty important, so make sure you get on top of your version
control strategy. (And ditch the darn notebooks)

3. Test automation and CI
Arguments for Test Driven Development and Continuous Integration are widely
available and universally recognised as the single crucial factor that enables
faster, safer, better code. The very same arguments apply to Data Science. The
ability to move faster with your code enables quicker feedback and shorter
iterations: you will be able to experiment and prove ideas quicker, boosting
productivity. You will also end up with code that is designed better, and it can
be transitioned into a production environment more easily.

4. Infrastructure as code
In order to take full advantage of the ongoing democratisation of computing
power, it is crucial to write code that is portable and can be deployed wherever
is faster to run it. In a typical development setting, we want be able to
provision computing resources in the cloud, deploy our code there, execute it,
collect outputs and dispose of the remote resources. And we want to be able to
do this in a fully automated fashion, at the push of a button.
Many ways to skin this cat, but if I were to single out one thing you should do,
and learn to do well, is containerisation. Plenty of great Docker learning
resources around: get stuck in!

5. Data management
As in any other scientific setting, it is mighty important for experiments to be
repeatable under the same conditions. We want to make sure immutability of
inputs is preserved through transformations. Versioning and artefacts management
tools play an invaluable role in the development of Data Science solutions. This
goes hand in hand with practices discussed in points 2) and 3), and falls
naturally from there.

6. Metadata management
Often the journey is just as valuable as the destination. As much as possible
(and to the extent that is useful, of course!) we want to be able to retain key
information about the process at hand, so that we can better interpret, analyse
and enhance its outputs. The spectrum here is broad, from basic logging to
systematic capturing and analysis of metadata. The point is: we want to be able
to fulfil this function in the way that maximises the insights offered by the
data being collected. 
Metadata management is at the core of many activities that are either routine
(eg compare performance of models), or required for production (eg backout
strategy for a failing data pipeline), or further reaching into the realms of
ethics and accountability (eg quantitative justification of a Machine Learning
prediction). It is important we do it well.

Conclusions
I fully appreciate some of the recommendations above may feel daunting to the
more traditional Data Scientists, due to such activities not (yet!) being
typically associated with their profession. To some extent, we witnessed
something similar in the past with network and infrastructure specialists, now
evolved in DevOps that design and build infrastructure solutions through code,
fully embedded in the fabric of development teams from the onset, sharing tools
and methodologies. 
Similarly, the key for enabling the evolution of Data Science from R&D practice
to fully fledged enterprise function, lays in bringing data scientists and
developers closer: before long the distinction will blur, and specialist
knowledge will thrive on a common substrate of working practices. 

At the end of the day, remember, Data Science is about building software.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. We deliver practical data science implementations for both our own SaaS products, and for our clients' bespoke platforms and products, for the analysis of data and media at scale.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. Find updates, News and insights on Data Science here.

Data Science - our Work, Features, News and Updates

Data Science

Paul Wilton

The problem: flaky connections cause upload of large files to S3 to fail
Uploading files to S3 securely direct from a browser is a common pattern. There
is plenty of information available on how to do this. Typically most examples
documented online use standard S3 signed POST requests (either from direct from
a html form or via an XHR request). This often means using some server side API
call (which you need to build) to generate an AWS signed request, which can then
be used in your web app client side. While being fairly easy to implement, for
very large files a flaky connection or a dropped packet often means a failed
upload and retries. What you need is multipart, chunked uploads that are
resilient. However this is not trivial to achieve with S3.

The solution: secure chunked multipart managed uploads
If you use the AWS SDK there are essentially two routes. Use the S3 REST API and
manage file chunks yourself. This is the low-level approach and is complex.
Alternatively you can use the AWS SDK Managed Uploads API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property]
, in which the SDK does all the hard work for you to manage file chunking,
upload, resilience, and the reconstruction of the file parts into a single S3
file at the end. However to use this you have to supply AWS credentials, and
this is not something you typically want to expose client side. However, if your
upload feature is behind some oAuth or OpenID authentication layer (as in my
case, and I suspect is often the case if you are allowing uploads to your
backend) then you are in business. You can setup AWS to work as OpenID Connect
provider. Finally you need to integrate this with some client side file uploader
library, such as Dropzone [http://www.dropzonejs.com/]. I am going to
demonstrate step-by-step how to :

a. Setup an S3 bucket to receive the uploads with a suitable access policy and
CORS settings.
b. Setup AWS as an OpenID Connect provider for some external oAuth service - in
this case Auth0 [https://auth0.com].
c. Integrate AWS S3 Managed Uploads securely with Dropzone to handle multiple
parallel resilient uploads for very large files, with progress bars, and cancel.

Step 1 - Create an S3 bucket
In your AWS tenancy, create an S3 bucket (e.g. my-bucket) and a CORS policy as
follows.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-meta-custom-header</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>Authorization</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>


You can revoke all public access to the bucket, as we will be writing to the
bucket using an OpenID Connect user that has assumed an IAM Role with the
correct policy to access the bucket.

Step 2 - Create an OpenID Connector Provider in AWS for Auth0
This allows us to use AWS as an intermediary for validating JWT OpenID tokens
against an oAuth service, demonstrated here with Auth0.
Your web app users will authenticate (login) with Auth0 - The oAuth response
from Auth0 authorize endpoint delivers an OpenID JWT token to the browser, which
you typically store in localStorage for securing subsequent requests from the
client to your backend.
To setup the AWS OpenID Connect provider you need the oAuth service root
endpoint, and the oAuth application clientID. In Auth0 login to your dashboard
and in your application settings you can get this info.

The oAuth endpoint is : https://{your-auth0-domain}/ (note the trailing slash),
e.g. https://appname.auth0.com/

The application Client ID is a 32 character Base62 encoded string, e.g. 
6sR4LF3UpznU4T26S3xRheGhjEKdni4H

Now login to your AWS tenancy, navigate to IAM account management > Identity
Providers. Click on Create Provider, and enter the Auth0 details from above as
follows, ensuring you add the trailing slash on the provider URL:

Then click Next and Create the provider.

Step 3 - Create an IAM Role for the OpenID Provider
Navigate to Policies in IAM, and create a new Policy as follows using the JSON
editor, setting the correct name for {my-bucket}:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::{my-bucket}",
                "arn:aws:s3:::{my-bucket}/*"
            ]
        }
    ]
}


Note, this policy will grant the authenticated user assuming the role all
actions on the bucket.. you may want to make this more restrictive. You could
also add other AWS grants here too, so your user can access other AWS services
if need be. Name your policy to something sensible, eg OpenIdS3Policy.

Now navigate to IAM Roles, and create a new Role. Choose Web Identity as below
and choose your OpenID Provider you just created, and the Auth0 Client ID as the
audience in the drop down options :

Next, Set the permissions of the role by choosing the OpenIdS3Policy created
previously. It is easy to find your policy by selecting policy type Customer
managed from the drop down. Save the Role with a suitable name eg OpenIdS3Role.

Step 4 - Grant access to the S3 bucket
Now we need to grant access to the S3 bucket for the IAM Role just created.
Copy the ARN for the Role, and navigate to your bucket setttings, and choose 
Bucket Policy. Add a Policy as follows, using the ARN for the role, and the
correct bucket name :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permissions",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:GetBucketLocation",
          "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}"
    },
    {
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}/*"
    }
  ]
}


That is all the AWS work needed. An authenticated OpenID Connect client can now
access your S3 bucket by assuming the role of an IAM user. When the AWS
connection is made, AWS authenticates the supplied JWT token against Auth0 using
the AWS Identity Provider. Thus a user has to be logged into an Auth0 session to
be able to access the bucket. AWS matches the aud value in the decoded JWT token
with the trust relationship condition specified in the role we created. This is
where the all important trailing slash I mentioned comes into play. This is an
artefact of using Auth0 as an OpenID identity service. Auth0 adds a trailing
slash to the aud value in the JWT token it generates. If we dont add the same,
AWS cannot match the role with the token when it tries to assume the identity of
the connecting user. If you use a different oAuth provider than Auth0 you may
not need this trailing slash. Think of it as an Auth0 'gotcha'.

Step 5 - Integrate with Dropzone
Dropzone [http://www.dropzonejs.com] is one of the better, and widely used open
source file upload JS libs. It is very customizable and mostly event driven so
is easy to modify. Out of the box Dropzone uses XHR requests to send files to a
remote server. We are going to override Dropzone's file uploading mechanism to
use the AWS SDK Managed Upload API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property] 
instead.

First, install (or make available) the AWS SDK in your web app, e.g.

npm install aws-sdk --save


Now define an accept callback function, that Dropzone invokes when a file has
been accepted for upload. This lets us add an S3.ManagedUpload instance to the
file object being queued by Dropzone for upload :

import S3 from 'aws-sdk/clients/s3';

function acceptCallback(file, done) {
  // options for the managed upload for this accepted file
  // define the bucket, and the S3 key the file will be stored as
  var params = {Bucket: 'my-bucket', Key: file.name, Body: file};
  // add an S3 managed upload instance to the file
  file.s3upload = new S3.ManagedUpload({params: params});
  done();
}


Now we need to override the Dropzone uploadFiles function to upload via the AWS
SDK instead of XHR. Note we are using the file.s3upload instance we added to the
file object when the file was accepted by Dropzone. We emit Dropzone events on
success and failure, and we also listen to the AWS httpUploadProgress event, and
emit the equivalent Dropzone event, so Dropzone can render upload progress for
each file :

// override the uploadFiles function to send via AWS SDK instead of xhr
Dropzone.prototype.uploadFiles = function (files) {
  for (var j = 0; j < files.length; j++) {
    var file = files[j];
    sendFile(file);
  }
};

function sendFile(file) {
  file.s3upload.send(function(err, data) {
    if (err) {
      theDropzone.emit("error", file, err.message);
    } else {
      theDropzone.emit("complete", file);
    }
  });

  // listen to the AWS httpUploadProgress event, and emit an equivalent Dropzone event
  file.s3upload.on('httpUploadProgress', function(progress) {
    if (progress.total) {
      var percent = ((progress.loaded * 100) / progress.total);
      theDropzone.emit('uploadprogress', file, percent, progress.loaded);
    }
  });
}


If we want our users to be able to Cancel uploads mid-progress we need to
implement an abort function that can be invoked on the Dropzone canceled event :

function abortUpload(file) {
  if (file.s3upload) file.s3upload.abort();
}


Finally you need to initialize your Dropzone on some DOM element on your page
and
setup AWS Credentials for the S3 managed uploads. To do this we create an 
AWS.WebIdentityCredentials object using the ARN of the IAM role we created in
Step 3, and also inject the JWT ID token from your local storage/session - the
token returned by the Auth0 client side authentication workflow. When we
initialize Dropzone we also set up the accept and canceled event callbacks.

import AWS from 'aws-sdk/global';
import S3 from 'aws-sdk/clients/s3';

function initializeDropzone(dropzoneElement) {

  var idToken =  'your-jwt-openId-token'; // get from local storage/session
  AWS.config.credentials = new AWS.WebIdentityCredentials({
    RoleArn: 'arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role',
    WebIdentityToken: idToken
  });

  theDropzone = new Dropzone(dropzoneElement, {
    autoQueue: false, // if you want to manually trigger uploads
    accept: acceptCallback,
    canceled : abortUpload,
    init: function () {
      this.on('removedfile', function (file) {
        abortUpload(file)
      })
    }
  })
};


All done. In these examples, I have shown the minimum you need to do to get S3
Managed Uploads working with, for example, the Boostrap Dropzone demo here
[http://www.dropzonejs.com/bootstrap.html]. There are of course lots more
features of Dropzone you can implement, but this should be sufficient to for you
to build fully resilient secure S3 uploads using Dropzone and OpenID
authentication for very large files.

Using S3 Managed Uploads with Dropzone and OpenID to support very large file upload

JSON-LD has been among us for several years now, its adoption is increasing, but
I suspect not as fast as it probably should. JSON-LD arose out of the RDF
community, subsequently adopted as a W3C standard, as a pattern for representing
RDF and linked data (hence 'LD') as JSON, with a key aim of making RDF much
easier to consume for developers.

While Google has raised awareness of JSON-LD considerably as it’s preferred
pattern for marking up web pages with schema.org [https://schema.org] structured
data, JSON-LD however has only made relatively small inroads into mainstream web
development. There are number of reasons for this, not least that the W3C and
JSON-LD working group did not do a great job of communicating its worth and
explaining how to use it effectively, although this has improved a lot more
recently, the excellent (and long overdue) JSON-LD API best practices guide
[https://json-ld.org/spec/latest/json-ld-api-best-practices/] published in
February 2018 is a very good example.

I am going to attempt to help the cause too, and make a convincing case of why
JSON-LD should be used in mainstream software development, regardless of whether
you are working with RDF.

A Very Brief Overview
I want to avoid making this post a full-on lesson about JSON-LD and RDF, so if
you are not entirely sure what JSON-LD is, here is super brief 101.
Alternatively get stuck into the full JSON-LD 1.0 specification here
[https://www.w3.org/TR/json-ld], and the draft 1.1 spec here
[https://json-ld.org/spec/latest/json-ld/].

Roots in RDF :
JSON-LD is a W3C standard for representing RDF [https://www.w3.org/RDF/] in JSON
format.
It is a standard pattern (schema) for representing the key elements of RDF
resources and the statements about them, such as the resource identifier (the
IRI) of the object being represented and the rdf type (class) of object being
described. For example, you will often see snippets of JSON-LD looking like this
:

{
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple"
}


If you are familiar with RDF, you will be aware that most RDF formats typically
involve a proliferation of IRIs/URIs everywhere. JSON-LD however alleviates this
by abstracting base URIs into a context. For the example above, if this was
written without a context in valid JSON-LD it would look more like this :

{
  "@id": "http://data.example.com/organizations/1",
  "@type": "http://data.example.com/some-model/Organization",
  "http://data.example.com/some-model/name": "Apple"
}


While verbose and ugly, in RDF this can be represented as, or transformed into
long form statements (triples) like this :


<http://data.example.com/organizations/1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.example.com/some-model/Organization> .
<http://data.example.com/organizations/1> <http://data.example.com/some-model/name> "Apple" .



You can test the validity and see how JSON-LD can be expanded into RDF using the 
JSON-LD playground here
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22%40type%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%3A%22Apple%22%7D]
.

The Context :
Introducing the ‘context’ into the JSON-LD makes this much more palatable. The
context defines the fields in the data payload :

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple"
}


In this example, the original data from my first snippet, the type and name
properties of the resource are explicitly identified in some schema or ontology,
so as a consumer of this JSON you are aware of the model that this data was
described with. Without the context the JSON cannot be transformed to RDF
statements. As a consumer of course you can choose whether to ignore the
context, or use it depending on your own need.

This same JSON-LD is often written such that that the context and payload are
separated using the @graph construct as follows. This is in fact identical data
to that above, it just identifies the data payload and the context that
describes it as separate JSON objects.

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


Back in RDF world, this also lets us create a named-graph which would then allow
us to generate RDF quad statements instead of triples through the addition of an
identifier for the graph :

{
  "@context": {
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name"
  },
  "@id" : "urn:my:organizations:1",
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


Which if you try this
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40context%22%3A%7B%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%7D%2C%22%40id%22%3A%22urn%3Amy%3Aorganizations%3A1%22%2C%22%40graph%22%3A%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22%40type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%7D%7D] 
in the json-ld playground will give you RDF statements :

<http://data.example.com/organizations/1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://data.example.com/some-model/Organization> <urn:my:organizations:1> .
<http://data.example.com/organizations/1> <http://data.example.com/some-model/name> "Apple" <urn:my:organizations:1> .


To make this more concise the context does not need to be explicit in each
payload, it can be remote, so a consumer can choose whether to dereference it or
not :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id" : "urn:my:organizations:1",
  "@graph" : {
    "@id": "http://data.example.com/organizations/1",
    "@type": "Organization",
    "name": "Apple"
  }
}


This also means contexts can be cached for higher performance.
So there you have it, RDF in JSON form. JSON-LD has brought RDF out of the labs,
and made it presentable with a highly consumable JSON representation for
developers.
Or has it...

Mythbusters
Myth 1 - frequently spoken by developers : “it’s not like normal JSON”
While the introduction of JSON-LD is undoubtedly a massive deal for the RDF
development community, it has not been widely embraced by software developers.
Most developers will not have heard of RDF, and even if they are consuming
JSON-LD from some RDF or Graph Database backend, they won’t (and probably don’t
need to be) concerned with it.

A key design goal
[https://json-ld.org/spec/latest/json-ld/#design-goals-and-rationale] in the
JSON-LD specification is :

> Zero Edits, most of the time - “JSON-LD ensures a smooth and simple transition
from existing JSON-based systems. In many cases, zero edits to the JSON document
and the addition of one line to the HTTP response should suffice“


However, when developers first come across JSON-LD it is rarely like this. The
majority of published JSON-LD is littered with JSON-LD nuance. Developers do not
like consuming many of the JSON-LD constructs, and when you hear the complaint
“it’s not like normal JSON” it is typically for good reason. For example, two of
the most common complaints are :

The "@" prefixes :
The “@” prefix on the JSON properties is not a valid ES5 or ES6 prefix for
javascript variables, meaning when dereferencing JSON as a JS object, you cannot
do things like

const id = myObject.@id ;

Instead you need to code

const id = myObject['@id']

Arrays vs Singletons :
JSON-LD output from some SPARQL query or RDF transformation typically does not
confer any indication if a property is an array or a singleton. The result of
one request to some back-end API serving JSON-LD might give :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple",
  "employs": {
    "@id" : "http://data.example.com/people/1",
    "@type": "Person",
    "name" : "Tim Cook"
  }
}


A future request may return an array for the employs property :

{
  "@context": "https://data.example.com/contexts/organization",
  "@id": "http://data.example.com/organizations/1",
  "@type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "@id" : "http://data.example.com/people/1",
    "@type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "@id" : "http://data.example.com/people/2",
    "@type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


The software developer consuming this output then has to code around this when
evaluating the value of the employs property, and check whether the property
contains an array or a singleton.

I imagine these are possibly the two most common JSON-LD complaints from devs.
Even Google publish their schema.org JSON-LD with @id and @type field names on
the Google Knowledge Graph API [https://developers.google.com/knowledge-graph/].
This doesn't help, and yet it doesn’t have to be like this. Instead of telling
developers to “get over it”, as publishers of JSON-LD we can make JSON-LD look
and behave like normal JSON using the context, meeting the design goal of the
specification. However, it is not always obvious how to make this happen, here
are some simple examples on how to address the two common issues above.

Addressing the “@” issue
Use context aliasing to abstract the “@” symbol away from the data payload. By
creating aliases in the context for those properties prefixed with @, they can
be appear in the data payload fully JS compatible. The context does not just
allow us to provide definitions of our domain-specific JSON properties, but we
can also alias the JSON-LD standard fields. Try this
[https://json-ld.org/playground/#startTab=tab-nquads&json-ld=%7B%22%40context%22%3A%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%7D%2C%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%2C%22employs%22%3A%5B%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F1%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Tim%20Cook%22%7D%2C%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F2%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Jony%20Ive%22%7D%5D%7D] 
in the JSON-LD playground :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : "http://data.example.com/some-model/employs"
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "id" : "http://data.example.com/people/2",
    "type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


Handling collections :
Use the context to define which fields are arrays, and output them as such :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : {
      "@id" : "http://data.example.com/some-model/employs",
      "@container" : "@set"
    }
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
  	},
    {
    "id" : "http://data.example.com/people/2",
    "type": "Person",
    "name" : "Jony Ive"
  	}
  ]
}


Forcing the set (or list) property to be marshalled as an array in JSON even
when it is a singleton. See this in action
[https://json-ld.org/playground/#startTab=tab-compacted&json-ld=%7B%22%40context%22%3A%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%7D%2C%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Forganizations%2F1%22%2C%22type%22%3A%22Organization%22%2C%22name%22%3A%22Apple%22%2C%22employs%22%3A%7B%22id%22%3A%22http%3A%2F%2Fdata.example.com%2Fpeople%2F1%22%2C%22type%22%3A%22Person%22%2C%22name%22%3A%22Tim%20Cook%22%7D%7D&frame=%7B%7D&context=%7B%22type%22%3A%7B%22%40id%22%3A%22%40type%22%7D%2C%22id%22%3A%7B%22%40id%22%3A%22%40id%22%7D%2C%22Organization%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2FOrganization%22%2C%22name%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Fname%22%2C%22employs%22%3A%7B%22%40id%22%3A%22http%3A%2F%2Fdata.example.com%2Fsome-model%2Femploys%22%2C%22%40container%22%3A%22%40set%22%7D%7D] 
in the JSON-LD playground :

{
  "@context": {
    "type" : {
      "@id" : "@type"
    },
    "id" : {
      "@id" : "@id"
    },
    "Organization" : "http://data.example.com/some-model/Organization", 
    "name" : "http://data.example.com/some-model/name",
    "employs" : {
      "@id": "http://data.example.com/some-model/employs",
      "@container": "@set"
    }
  },
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
    "id" : "http://data.example.com/people/1",
    "type": "Person",
    "name" : "Tim Cook"
    }
  ]
}


Using a remote context (that most web developers can safely ignore), we now we
have JSON-LD that looks just the way your developers would prefer, and meeting
the design goal of the JSON-LD specification :

{
  "@context": "https://data.example.com/contexts/organization", 
  "id": "http://data.example.com/organizations/1",
  "type": "Organization",
  "name": "Apple",
  "employs": [
    {
      "id" : "http://data.example.com/people/1",
      "type": "Person",
      "name" : "Tim Cook"
    }
  ]
}


Myth 2 - “JSON-LD is only needed if you are working with RDF or Graph Databases”
If you are building a web application or micro-service which is using RDF, then
JSON-LD is clearly a no-brainer. However if you don’t care about RDF why bother
?

I argue that you should bother. If you are publishing an API you may not know
how your consumers are using your data. They may well be using a graph database,
or dealing with RDF or linked data and be very keen to consume your API in an
RDF compatible format. Furthermore, publishing JSON-LD requires you to publish a
context. You are thus constraining your JSON to a schema. This is a good thing.
It communicates a model of your domain, provides a payload contract for your
consumers (or a contract between your own micro-service APIs and applications),
and a validation mechanism.

Why we should all adopt JSON-LD
JSON Entropy
Being inherently a schemaless message format, applications built with components
that use JSON to serialize/deserialize data moving between them have a tendency
over time to create JSON property soup. As requirements evolve, and developers
are left to their own devices to modify features and adapt the codebase,
properties are introduced into JSON on an ad-hoc basis. JSON entropy increases.
What may have started out as a simple, clean JSON model, tends towards one of
mess and complexity.
Adopting JSON-LD as your message format reduces this entropy escalation, as :

a) it makes you as a developer think about the schema (structure) of the JSON
from the perspective of your consumers, and the wider application as a whole.
b) once we start publishing schemas for our data, we start to think like a data
architect rather than as a developer.
c) we are more likely to collaborate on the schema design, and at the very
minimum, ask for a peer review of a proposed change.

While this may seem like an overhead, it is a relatively small investment into
your data architecture that pays dividends over the long term.

Schema
The JSON-LD context is a schema for your data, not only defining the property
datatypes but also the classes of json resources and, via referenced ontologies,
the semantics and relationships between properties and classes in your data.
The json-schema [http://json-schema.org/] project has attempted to fill this
space for JSON, but has not really gained too much traction. While it is a
little richer in schema-definition features than what a JSON-LD context
provides, JSON-LD is an official W3C standard, and offers a lot more than
json-schema can with respect to linked data and RDF compatibility.

We all can publish linked data
If you are using linked data or RDF, then you will no doubt already be familiar
with JSON-LD, and hopefully using JSON-LD as a message format in your own
applications. If you are publishing linked data then it is very likely you are
publishing it as JSON-LD. If you are serving public APIs as regular JSON, then
publishing JSON-LD instead makes a lot of sense. As discussed, you should be
able to transition with little or zero change to your data payload. All it takes
to make the transition is to publish a context. For little additional overhead,
you and your consumers benefit. It is essential however, that we, as publishers
of JSON-LD make it as consumer friendly as possible (and this includes Google).

Software Development

Software Development Blog Posts