Paul Wilton

Data Graphs is the latest Data Language baby. Data Graphs
[https://datagraphs.com] was born out of need, a need for really easy structured
data management. Our thoughts on this evolved to eventually produce a SaaS
product that allows a team to curate a knowledge graph for any domain, from
simple flat concept collections to complex structured domain data. This is all
wrapped up in an extremely intuitive user experience, combined with a fully
comprehensive API and web-hooks to give you extremely efficient integration with
applications and business systems. You can check it out here : 
https://datagraphs.com

What does it do ?
Put simply it provides a dead simple way to create knowledge graph data models,
populate them with data, and interrogate (query) and integrate that data with
your applications using a flexible and easy API.  In essense, it is Knowledege
Graphs as a Service (KGaaS).

Building your own knowledge graph platform can take months, requites highly
specialised expertise and lot of $$, and that is before you start populating it
with data. Even when you build your own KG platform, the information management
tools dont come for free, and are typically sorely lacking in utlity.

We have short-circuited that complex engineering process, and done all the hard
yards for you. Futhermore we have added a stunning information management user
experience. With Data Graphs you can be up and running in minutes, start
creating your graph data models, and managing your data.

You don’t have to worry about scaling, engineering, backups, it’s all handled
for you. But best of all is the super-intuitive user interface for managing,
searching, and maintaining data over time. The API lets you search, query,
filter, and facet on your data - slice and dice it. The API is rocket fast and
easy to use, and also geospatially aware.

We think Data Graphs addresses a lot of use-cases, from straight-forward data
catalogs or vocabulary management, to easy to setup and maintain knowledge
graphs. You could use it for managing structured tagging data for a publishing
system, integrating it with your CMS, or even for maintaining time series
structured data powering charts in WordPress for data journalism. It is ideal as
a central metadata or reference data repository for your enterprise business
systems. You could even use it for maintaining a nicely categorised and faceted
product catalog to integrate with your e-commerce store.

Schemas, Concepts, and Datasets
The Data Graphs model is based around three fundamental ideas - 

Datasets - these are collections of Concepts - organised into groupings of your
choice. Datasets can hold concepts of different types, or concepts all of the
same type. Datasets can also be public or private -

 * a public dataset means the data it holds is accessible via API using only an
   API key, so it could be directly accessible via anyone on the internet, who
   has access to your API key.
 * a private dataset means the data it holds can only be accessed using Open ID
   / oAuth security credentials

Schemas - this is your data model, definitions of the things (or concepts) in
your domain, their properties, and how they are linked together, the
relationships between the things. Data Graphs has an intuitive user interface
that lets you quickly design a data structure that reflects your own domain
model.  Each Concept type in your domain has its own schema definition where you
define the properties of a concept and the data types of those properties
(dates, integers, text etc). For example, a simple schema of a Person might have
a givenName (text), familyName (text), and a dateOfBirth (date), and a headShot 
(image).  Rather than explain it in depth here, if you need to know more, we
have made a short video
[https://datagraphs.com/demos/how-to-setup-your-data-sets] that shows how
quickly you can create a schema for your domain model.

Concepts - the data itself. Concepts are the data instances that you load into
Data Graphs conforming to your schemas. You can create and edit concepts via the
user interface, bulk import them from for example CSV files or external sources,
or load them via the Data Graphs UI. Once created, you can search , discover,
and maintain the data via the UI.

SaaSification
Data Graphs is fully SaaSified - you can sign up online [https://datagraphs.com]
, frictionless, and get started immediately with a free trial. You can manage
your own users, roles, your own backups, your own API keys and oAuth
credentials. 

The User Experience
Delivering a great data management experience was our primary goal when creating
Data Graphs. In almost all the knowledge graph and information management
projects we have worked on, data management and curation tooling was always an
area that was a challenge. We set out to build something extremely intuitive to
use, with a UI that would be self-aware of the data types, the relationship
semantics, and make it super easy for users to explore, find, and navigate data.

Dog Food
We are great fans of the “Eat your own dog food” approach to product
development. If our APIs are not good enough for ourselves to build an
application upon, how can we expect them to be good enough for others? With this
in mind, all the features in the Data Graphs Application are built upon the same
APIs exposed to the Data Graphs client, and with that in mind, let’s talk about
integration..

Integration
There is no point having a fabulous user experience for ingesting and managing
your data if you cannot integrate it. As with most SaaS products Data Graphs
comes with an API. A set of REST endpoints for searching, filtering, querying,
faceting, retrieving and creating data. It is fast, it is flexible.

With the API you can access data only in specified datasets, or you can federate
a query across all your data.

If used with only your API key, you access only the data in public dataset. If
used with your oAuth credentials, you can access both public or private
datasets.

Web Hooks - Triggered
The API gives us pull integration, web hooks provide us with push. You can
configure web hooks for Concept create, update, delete, and also candidate
(concept) approval. You define optional endpoints (typically in your  own
platform) that are invoked whenever a specified Data Graphs event is triggered.
For example you could trigger some downstream system when a Concept of a
specific type is created or updated. For example if an Organization concept is
updated, trigger some workflow in an existing business system. The web-hook
events supply the entire concept data in its payload, which means you could use
this to push data into another database or application.

Querying data - The Data Graphs Filter Expression Language
As well as features such as sort and facet (by concept property) the API lets
you query your data using a flexible filter expression language (often called
NQL).  In simple terms, this lets us filter data like this 
filter=property1:value1,property2:value2

A filter expression is a string which provides a (Concept)  property, operator
and value in the form where property is a JSON path representing the field to
filter on (using JSON dot notation) and operator is optional , and tells the API
how to compare values. value is the value to match against.

Operators can be -(not), >, >=,<<=, and also a test if a value is in a group of
values [value,value,...]

Boolean expressions are also supported including parentheses for grouping
expressions together, and overriding operator precedence. 

Full text search is supported using the traditional q parameter. We index all
the text fields in concepts, and boost the canonical labels.

Combining all these together lets us build extremely high utility data queries,
and very flexible integrations.

You can query within a specified dataset, or you can federate across all your
data, and using the filter expression language it means you can ask extremely
useful questions of your data.

The API is fast - you can expect all requests to be returned in less than a few
hundred milliseconds.

Geospatial
We also support geospatial data. We have primitive datatypes for latitude, 
longitude and geohash - so both our API and UI can be geospatially aware for
concepts with location information. As well as being able to explore geospatial
data in the UI on maps, you can also query geospatially using point-radius,
bounding-box, and bounding -polygon queries. Any geospatial query can be
combined with other API filters and full text search.

This means you can ask questions of your data such as “Find me all the concepts
within 50km of New York, classified by X, with property P greater than Y, and
the word “hello” in their label”. Nice.

For example, given a geospatial data set of Airports with relationship to
airlines , get all airports within a 50km radius of London, serviced by EasyJet,
we might use an API request like :

/airports?filter=_near((51.51, -0.12), 50km),airlines:urn:demo:airlines:easyjet

Client Library
For developers we have published a Javascript client library for Data Graphs to
make integration even easier. The client library is particularly useful for
consuming the API using OpenID credentials, as the machine-machine OpenID
authorization flow is all taken care of.  The client library documentation is
published in NPM here : 
https://www.npmjs.com/package/@datalanguage/datagraphs-client

Sub-classing, Inheritance, and Transitive Inference
While Data Graphs is not a full blown graph database, it is a fully managed
structured data solution that can be used to build a high-utility highly
performant knowledge graph for your organization without having to manage any
infrastructure.  There are some key features in knowledge graphs that add
significant value. 

Subclassing and class inheritance is one of those features. In our (extensive)
experience of working with graph databases, inference is a feature that is
widely misunderstood. The set of potential inference rules available in the full
RDF and OWL specification is considerable. However in most practical
implementations of knowledge graphs (outside of academia), there are just one or
two core inferences that add significant value in most business applications.
Subclassing and transitivity is top of that list. Data Graphs has full support
for this.

When you create a new Concept schema definition, you have the option of
declaring a parent type (class) and inheriting its properties. 

Create a Concept with a parent typeOnce you have created your concept schemas,
we have a neat class structure management tool like this :

Concept relationships (and the User Experience) then become class structure
aware, with transitive inference.  When a concept property defines a
relationship to another concept type/class the UI allows you to select concept
instances of that type and all its subclasses.

Transitive relationship propertiesAPI queries also obey the usual semantics of
transitive inference across a class hierarchy. 

Linked Data and Graph Databases
Data Graphs has support for JSON-LD. API responses can be requested with an 
application/ld+json mime-type such that the data is returned in JSON-LD with a
context dynamically constructed from your schema.

This lets you transform data directly to RDF for loading into your own graph
database, or another RDF compliant system. 

So while functioning capably as a standlone fully managed knowledge graph, Data
Graphs could thus also be used as a tool-suite for editing data in a downstream
enterprise graph database / triple store. Using the web-hooks, it is fairly
trivial to build a service that updates entities in a graph database when the
web-hook is triggered from a concept create/update operation in Data Graphs.

The Road Ahead
Data Graphs already has a plethora of great features
[https://datagraphs.com/features], but we have some great new ones lined up in
our road map. Not least is data and model visualisation.

Building on the class structure visualisation we plan to render detailed domain
model / ontology diagrams dynamically from your schema. We also plan a dynamic,
navigable graph visualisation of the instance data itself. On the back-end, we
plan to offer regional data storage options.

We think we have produced something really special with Data Graphs, something
that really delivers utility and value for information management in your
organisation - but don’t take my word for it, go ahead try it out for free
[https://datagraphs.com].

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

Data Graphs - Fully Managed Knowledge Graphs

Steve Liles

In this mini-series we’re looking at rendering interactive graphical UI’s using
components built with React and SVG. Throughout the series we’re putting
together an interactive floor-plan while working through SVG concepts, starting
from the basics and working through some more complicated areas.

If you’re new to React and/or SVG, or you just want to get some context on our
ongoing project, you might want to start from the beginning with part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics]
, Declarative Graphics, where we laid out our basic floor-plan blueprint.

Inpart 2
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations] 
we added cameras to our plan and showed their field of view, but there’s a
problem: the camera field of view passes through walls and around corners,
giving a false impression of what the camera can see.

In this, part 3, we’ll use clip paths to constrain the field of view to the room
that the camera is in, preventing it from overflowing into other rooms or
outside the floor-plan entirely.

Our cameras can see through walls and around corners!Boundaries
Let’s start with the easy bit: anything in the room should respect the room’s
boundaries. We already have geometry for the room that defines the constraints
that we want to apply to anything in the room, so how can we apply those
constraints to prevent overflow?

The answer, of course, is clip paths. A clip path (defined with theclipPath 
element) is just what it sounds like: a path that clips any elements to which it
is applied, such that only content that falls inside the defined path is
rendered.

In SVG we can compose clip paths from any drawable SVG elements, e.g. circle, 
rect, polygon, path, and even text!

Binocular Vision
For a simple but fun first example, let’s bring back the Pac-Man ghosts from 
part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics] 
and use a clip-path to pretend we’re looking at them through binoculars.

First, here’s the scene without clipping, showing our four ghosts on a blue
background:

<g>  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />
  <Ghost x={400} y={350} scale={2} colour="red" />
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

un-clipped ghostiesNow let's create our first clip path to constrain the drawing
to two adjoining circles in the old movie trope binocular shape.

<clipPath id="binoculars">
  <circle cx={500} cy={400} r={200} />
  <circle cx={800} cy={400} r={200} />
</clipPath>

A couple of things worth highlighting here:

 1. As you can see, we’ve given the clipPath an id, which is necessary in order
    to apply the clip path to another SVG element.
 2. We didn’t stroke or fill the circles — we don’t need to — because clipPath 
    doesn’t pay any attention to stroke and fill, it simply takes the entirety
    of the shapes added to it and allows content to be rendered if it falls
    within those shapes. In that sense, clipPath is all-or-nothing.

Now all we need to do is set the clipPath attribute on the element we want to
clip with the valueurl(#binoculars). In our case we just need to add that to the 
<g> element containing the ghosts.

<g clipPath="url(#binoculars)">  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />  
  <Ghost x={400} y={350} scale={2} colour="red" />  
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

The black background of the SVG shows through outside the clip pathWhat goes on
in the room, stays in the room
Back to the floor-plan. Our Room component currently renders as a group<g> 
containing walls, corners, and cameras (and I added the room name as red text in
the corner of each room). Here’s the current code and its output:

The room component from part 2 - walls, corners, cameras, and labelsWe want to
modify this component so that the contents of a Room can no longer render
outside the bounds of the room. To do that we want to do two things:

 1. Change the render order such that the contents of the room (currently really
    just the camera’s) are drawn first, and therefore are rendered beneath the
    walls and corners (in z order).
 2. Render the room contents inside a <g> element that is constrained by a clip
    path along the room’s boundaries, so that contents cannot overflow the room.

Step 1 just means pulling the camera’s up to be the first thing rendered inside
the Room’s <g> element, but why do we even need step 1? Well, our walls and
corners are rendered along the boundary of the room, but they have some
thickness and so half of the thickness of the wall falls inside the boundary of
the room.

The clip path will prevent contents of the room from being rendered outside the
boundary, but would not stop them from rendering over the inner part of the
walls, so this is a simple workaround to prevent that. A more satisfying — but
more complicated — alternative would be to adjust the clip path to account for
the wall thickness, but for now we’ll keep things simple.

Room geometry as clip path
Having pulled our camera’s up to the first-child position in the room, we can
nest it in a new group <g> element, to which we will later apply our clip path.

<g>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} />  
  ))}
</g>

Now let’s create the clip path. We need to give it an id so that we can refer to
it from the clipPath attribute of the camera group element. Our room data
includes an id so we can use that to make up the clip path id, e.g.

<clipPath id={`${id}-content-clip`}></clipPath>

To define the shape of the clip path we need to use the room boundary
coordinates to construct a solid shape. Our room geometries are just a series of
coordinates with no flashy curves or arcs, so while we could use the
all-powerful path element we have a simpler alternative in the polygon element.

The SVG polygon element gives us a convenient way to convert our coordinates
into a rendered shape: polygon takes a points attribute, which is a
space-separated list of comma-separated coordinates, like this:

<polygon points="20,20 40,30 30,30 ..."/>

This is perfect for us, because we can simply map our room coordinates array to
a string for the points attribute and we’re done!

<clipPath id={`${id}-content-clip`}>  
  <polygon points={coords.map(({x,y}) => `${x},${y}`).join(" ")}/></clipPath>

Now all we need to do is add the clipPath attribute to our camera group to apply
the clip-path.

<g clipPath={`url(#${id}-content-clip)`}>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} /> 
  ))}
</g>

Win! The camera fields of view are now clipped to the room boundary!This is
great, with some very simple changes we’ve clipped the content of the room so
that it doesn’t spill outside the boundary!

Lines of Sight
Look closely at the floor-plan image above though, and you’ll see something’s
not quite right. Our cameras can’t see through walls any more, but they can see
around corners!

To solve this we’re going to need the camera itself to clip the field of view
according to its lines of sight. This will involve creating a more intricate
clip-path shape, but most of what follows is just a fun bit of problem solving,
so if you’re not into algorithms feel free to skip to the bottom to see the
results.

The Disjoint Segment Visibility Problem
To clip the camera field of view to what the camera would actually be able to
see accounting for occlusion due to walls blocking lines of sight, we need to
solve the Disjoint Segment Visibility Problem where the segments are our walls.

When dealing with large numbers of segments this is typically done with a
sweep-line algorithm that projects rays out from the view-point (our camera) at
discrete intervals around a full 360 degree sweep, checking for the nearest
intersection with a segment at each interval.

sweep-line algorithm at 10° intervals, red dots show nearest intersection to
cameraOne problem with the sweep-line approach is that the use of discrete
intervals (e.g. 1 degree) means that the precision is only as high as the chosen
interval allows for, and we might produce a field of view that visibly cuts
through corners of walls or misses the corner and leaves a small gap.

corners cut through or cut off due to lack of precision (interval too large)One
way to deal with this would be to use smaller intervals until we get the
precision we want, with the trade-off that the smaller the interval the more
calculations we need to make.

In our case, however, we know that each room is composed of a very small number
of segments (walls) so we can take a different approach: rather than sweep
through 360 degrees, we can check only the angles to known segment end-points,
which reduces the number of calculations we need to make, and simultaneously
allows us to achieve very high precision at little cost.

The Visibility Polygon
Our task, then, is to create a “visibility polygon” describing the critical
lines of sight from the camera position to the nearest walls of the containing
room. We’ll use a modified  sweep-line algorithm that boils down to some pretty
simple steps:

 1. Find the angle to each wall end-point from the camera
 2. Reduce to the set of unique angles
 3. Sort the unique angles into clockwise order
 4. Insert two new angles θ- and θ+ to either side of θ, where θ- is a fraction
    smaller than (CCW of)θ and θ+ is the same fraction larger than (CW of) θ.
 5. For each angle in the resulting list, find the nearest wall intersection and
    add it to our visibility polygon

At step 1, we can find the angle to each wall end-point simply by converting
each coordinate from the Cartesian coordinate system to a polar coordinate
system with the camera as the origin. The angle part of the polar coordinate is
thus the angle to the wall end-point.

Step 2 eliminates any duplicate work, for example where a camera might be
looking directly along the line of a wall such that both wall end-points are at
the same angle, or in more complex room shapes where distant corners might align
with nearer ones along the cameras line of sight.

distant corners that align with nearer ones can be eliminated from further
processingBecause rooms can potentially be composed of complicated arrangements
of walls, Step 3 is important to ensure that the points of our visibility
polygon are arranged clockwise around its boundary to produce a continuous shape
from start to finish.

It doesn’t strictly matter whether we sort now or at the end, but it is more
efficient to do it now because step 4 triples the number of points that would
need to be sorted.

wall coordinates for complex room shapes may not be clockwise ordered from the
camera perspectiveStep 4 gives us the high precision we want — the angles
fractionally before and after the angle to each wall end-point are the critical
line of sight intervals we need to test in order to get very precise results
with no visible corner cutting.

critical line-of-sight angles fractionally before and after the angle to the
wall corner pointsStep 5 converts the collected angles of critical lines of
sight into the Cartesian coordinates of our visibility polygon, by finding the
nearest intersection of the line of sight with a wall.

In javascript the whole thing looks like this:

const getVisibilityPolygon = (from, cartesianCoords) => {
  // wall corners as indexes into the cartesian coords array
  let walls = cartesianCoords.map((_, i) => [    
    i, (i + 1) % cartesianCoords.length  
  ]);  
  
  // get all points in polar coords, giving us angles from camera  
  let polar = cartesianCoords.map((c) => cartesianToPolar(c, from));  
    
  // reduce to the set of unique angles that we need to test  
  const unique = Array.from(    
    polar.reduce((r, { angle }) => r.add(angle), new Set())  
  );  
    
  // sort into clockwise order  
  const clockwise = unique.sort((a, b) => a - b);  
    
  // produce angles slightly CCW and CW of each angle in the list
  const checkPoints = clockwise    
    .map((angle) => [angle - 0.01, angle, angle + 0.01])    
    .flat();  
    
  // find the nearest intersection with a wall  
  return checkPoints.map((angle) => {
    return nearestIntersection(from, angle, walls, cartesianCoords);     
  });
}

For this to work we’ll need to add a few more utilities to our geometry toolkit:
First we’ll need to modify our polarToCartesian function so that we can pass the
camera’s Cartesian coordinates as the origin for the polar coordinates. Second,
we need to add the inverse function cartesianToPolar to convert in the other
direction.

export const REFERENCE_DIRECTION = -90;

export const degreesToRadians = (degrees) => 
  (degrees * Math.PI) / 180.0;

export const radiansToDegrees = (radians) => 
  radians * (180.0 / Math.PI);

export const vector = (from, to) => ({ 
  x: from.x - to.x, 
  y: from.y - to.y
});

export const magnitude = (vector) =>  
  Math.sqrt(Math.pow(vector.x, 2) + Math.pow(vector.y, 2));

export const distance = (from, to) => magnitude(vector(from, to));

export const polarToCartesian = (
  { angle, range }, orig = { x: 0, y: 0 }
) => {  
  let angleInRadians = degreesToRadians(angle + REFERENCE_DIRECTION);
  let cartesian = {    
    x: range * Math.cos(angleInRadians),    
    y: range * Math.sin(angleInRadians)  
  };  
  return { x: orig.x + cartesian.x, y: orig.y + cartesian.y };
};

export const cartesianToPolar = (point, orig = { x: 0, y: 0 }) => {  
  let v = vector(point, orig);  
  let range = magnitude(v);  
  let angle = radiansToDegrees(Math.atan2(v.y, v.x));  
  return { range, angle: angle - REFERENCE_DIRECTION };
};

Finally, we need a functionnearestIntersection that takes a Cartesian coordinate
of a point of interest such as our camera location, an angle, and a list of
segments (walls) to test for intersection at that angle, and returns the nearest
intersection to the point of interest as a Cartesian coordinate.

const nearestIntersection = (from, angle, walls, coords) => { ... }

This is going to come down to finding intersections between line segments, where
one segment is a wall, and the other is the line of sight — an imagined ray
projected out at angle from the camera position — so we will need a function to
test whether and where two segments intersect.

Finding the intersection of two line equations (think lines of infinite length)
is well-known mathematics, where one possible solution in code looks like this:

export const lineIntersectsLine = ([a, b], [c, d]) => {  
  let a1 = b.y - a.y;  
  let b1 = a.x - b.x;  
  let c1 = a1 * a.x + b1 * a.y;  
  let a2 = d.y - c.y;  
  let b2 = c.x - d.x;  
  let c2 = a2 * c.x + b2 * c.y;  
  let determinant = a1 * b2 - a2 * b1;  
  if (determinant === 0) {    
    // parallel lines never intersect    
    return false;  
  } else {    
    let x = (b2 * c1 - b1 * c2) / determinant;    
    let y = (a1 * c2 - a2 * c1) / determinant;    
    return { x, y };  
  }
};

This gives us the intersection point of the two line equations unless the lines
are parallel, and is a step in the right direction, but we need to know that the
point of intersection actually lies somewhere on both finite length sub-segments 
of the line equation (the wall, and the ray from the camera). We can check this
by testing that the intersection point lies within the bounds of both line
segments:

export const MARGIN_OF_ERROR = 1e-9;

const pointIsOnSegment = ({ x, y }, [a, b]) =>  
  x - MARGIN_OF_ERROR <= Math.max(a.x, b.x) &&  
  x + MARGIN_OF_ERROR >= Math.min(a.x, b.x) &&  
  y - MARGIN_OF_ERROR <= Math.max(a.y, b.y) &&  
  y + MARGIN_OF_ERROR >= Math.min(a.y, b.y);

Notice that we’re allowing a tiny margin of error here to account for errors in
floating point math. And now we can write a function that tests whether and
where two line segments intersect:

const segmentIntersectsSegment = (l1, l2) => {  
  let pointOfIntersection = lineIntersectsLine(l1, l2);  
  if (    
    pointOfIntersection &&    
    pointIsOnSegment(pointOfIntersection, l1) &&     
    pointIsOnSegment(pointOfIntersection, l2)  
  ) {    
    return pointOfIntersection;  
  } else {
    return false;  
  }
};

Which gives us everything we need to complete our implementation of 
nearestIntersection as follows:

const LARGE_DISTANCE = 1e9;

const nearestIntersection = (from, angle, walls, coords) => {  
  let to = polarToCartesian({ range: LARGE_DISTANCE, angle }, from);  
  let intersections = walls    
    .map(([a, b]) => 
      segmentIntersectsSegment([from, to], [coords[a], coords[b]])    
    )    
    .filter((intersection) => !!intersection)    
    .map((intersection) => ({      
      intersection,      
      distance: distance(intersection, from)    
    }))    
    .sort((a, b) => a.distance - b.distance);  
  return intersections?.[0]?.intersection;
};

When projecting the ray from the camera i’m using a very large distance instead
of the camera’s range, which will give us a visibility polygon defined as points
along the  walls of the room, regardless whether the camera can see that far.
 This slightly simplifies the algorithm because it means there will always be at
least one intersection for each angle.

Below you can see what happens when we test this with a more complex room shape
and a bunch of different cameras. To illustrate the visibility polygon vs the
camera field of view I have not clipped the field of view — instead I’ve drawn
both the field of view (green) and the visibility polygon (red) with some
transparency so that the intersection (orange) and difference are clearly
visible. Once the clip is applied, only the orange part will be visible.

When clipped to the visibility polygon, only the orange part of the field of
view would remainFeeding this back into our Camera component is straight-forward
and gives us the following:

Overlapping fields of view, clipped along lines of sightWhew, that got a little
heavy, but we now have a clean, simple, and fast mechanism for finding the
visibility polygon from any given point in a room, and we can use that
visibility polygon as a clip-path to constrain the camera field of view in a
realistic manner.

Recap
In this post we learned what clip-paths do, how to compose them from arbitrary
SVG shapes and paths, and how to apply them to elements or groups to constrain
what gets drawn.

We then took a bit of a dive into some fun algorithms that once again take
advantage of alternative coordinate systems to make our lives easier.

In part 4 we’ll look at masks — a more sophisticated method of constraining the
rendered output that gives us more control than clip-paths.

Software Development

Graphical UI's with SVG and React, Part 3 - Clip Paths

Graphical UI’s with SVG and React, part 2 — Arcs, Angles, and Transformations

BOTT is a data ethics framework for the practical assessment of AI and predictive analytics models within their ecosystems; Bias, Outcome, Target, Trust

BOTT Framework for Practical AI Ethics - Bias, Outcome, Target, Trust

As a data company, data ethics has always been important to us, but in the last
few years, as Machine learning and AI have become front and centre as a
significant part of digital transformation in the enterprise, data ethics has
become a hot topic. It bears such a considerable weight in predictive analytics,
not just in conveying trust to consumers of machine intelligence, but also in
ensuring predictive models deliver genuine value, make unbiased predictions, and
reflect the true state of the environment within which they live. When data
ethics go wrong, it can go badly wrong, potentially bringing down an entire
organisation, with substantial direct and indirect collateral damage,
potentially even on national democracy and social cohesion - the Cambridge
Analytica incident a case in point.

We all need to take it seriously.

In 2018 Data Language was privileged to have been accepted onto the Digital
Catapult “Machine Intelligence Garage
[https://www.digicatapult.org.uk/for-startups/acceleration-programmes/machine-intelligence-garage]
” programme. As well as supporting us with a terrific programme including
compute resource, strategic help and industry engagement, Digital Catapult also
took Data Language through their ethics programme. This was like a cold-shower,
really waking us up to depth and breadth of data ethics. A realisation that data
ethics is not just about data (technically), but about the way you approach your
entire business. 

With all this in mind, we were motivated to develop a data ethics framework that
we could use internally at Data Language to assess all our predictive analytics
projects, and that could guide us practically during implementation. We call
this framework BOTT.

The BOTT Framework
BOTT (all good frameworks need a memorable acronym) stands for Bias, Outcome,
Target and Trust. Through these four pillars, it allows our data science team to
examine, and assess a predictive analytics model within its local and external
ecosystem for quality, bias, and accuracy, and ensure it meets high standards of
data ethics.

Bias
An analysis of potential bias that may be baked into the model during its
life-cycle, either literally through data or code, or intangibly through for
example, some conscious or subconscious human bias. We examine bias from the
following perspectives :

Training data bias - is the training data diverse enough and comprehensive
enough to accurately reflect the environment of the predictive model. What
potential biases might skew predictive results either for good or bad outcomes -
is there any chance these biases might exist in the training data we have used.

Algorithmic bias - is the model algorithm, its mathematical basis, or code
potentially biased?

Real-time input data - if real-time (predict time) input data is involved in
making some prediction, what might these data biases be, and how can we prevent
them from impacting outputs.

Team bias - has or could conscious or unconscious bias of the model development
team contribute to model bias, and have we mitigated against it. Often just an
open and frank team discussion around bias (with respect to the project under
consideration) is enough to flush out these sort of issues

Political, environmental, and social bias - similar to the team bias but looking
at potential external factors through a similar lens - for both ideally the team
needs to be very comfortable talking about potentially politically or socially
sensitive topics in an open, friendly and non-defensive manner.

Output sampling - essentially QA - have you done effective output sampling and
for quality assurance - does the risk of bias in the predictive model call for
independent third party reviews in QA workflows, etc

Outcome
An analysis and survey of the full range of possible outcomes of the model:

Catalogue - construct a catalogue of the potential outcome scenarios

Impact - What does success and failure of the model mean :

 * in the context of the target use-cases
 * best and worst case scenario modelling
 * model failure consequences, e.g. could somebody die if the model fails ?

Target
Analysis of who or what the model is targeted at and the context in which these
users operate.

Users -  Who are target users of the model  - who or what is the direct consumer
of the model ?

Purpose - What problem is the model solving?

Context: Is it part of an ensemble of models, is there a wider predictive
ecosystem the model sits within - what impact does this model have within this
context ?

Life span -  What is the life expectancy of the model, and how does its
environment change over time

Subversion - Can other users or consumers subvert the model for
unintended/undesirable ends?

Catalogue - Construct  a catalogue of potential unintended uses/consequences,
and ascertain if those vulnerabilities can be designed out of the architecture?

Trust
Providing some visibility of the "behind the scenes" implementation can raise
confidence in the models and foster a trust relationship between consumers and
predictive services.

Explainable AI (XAI) is becoming a field in its own right, and an important part
of machine learning and predictive analytics. Codable frameworks are emerging to
aid explainability (for example Google Model Cards
[https://modelcards.withgoogle.com/about]). This codability will be key in being
able to compare and assess models through some normative evaluation mechanism.

Transparency - what should we be exposing of the lineage and decision making
pipeline of the model prediction algorithm :

 * training cycles / training samples
 * outputs of intermediate predictions
 * decision inputs/outputs

Impact - Why should these be exposed ? - perform an impact analysis of the
possible effects of not exposing model lineage, with respect to legal,
environmental, and model improvement consequences.

Data Ethics

The BOTT Framework for Practical Data Ethics

Jim Howard

Get top tips and insights from Jim Howard on valuable use cases for your Sports Data, and avoid the common pitfalls.

Getting the most from your Sports Data - Insights and Best Practice

My name is Jim Howard and I am a freelance Product Manager specialising in
developing products using data and content. After working at the BBC for a
number of years, I have worked on projects for a number of clients including
Government Digital Service, Eurosport and Stats Perform.

I got to know Data Language after working with some of the team on big
publishing and audience-facing Sport products (World Cups, Olympics etc), when I
was at the BBC and The Telegraph.

We have been discussing some of the issues that we have had to deal with over
the years and the opportunities that are still there for companies to make more
of the data that they consume. In this post, I will try and summarise some of
the key areas we have identified and some areas that could still be developed
further.

Sport, data and content in publishing

Now that the restrictions on Sport are gradually being eased, we thought it
would be an opportunity to review some of the areas where Sport-related products
have been developed and business issues have been addressed with a more
innovative approach to the use of data (not analytics) as a core ‘building
block’ to development. 

As always in organisations, there are lots of great ideas around innovation but
the business needs a focus - and sport is an ideal test subject. As we have
found out, you can look at a major sporting event in isolation (Olympics,
Football World Cup etc), review the outcomes, decide on what areas worked and
then integrate into wider product development strategies. 

I will try and break this down into areas that will be relevant to other
products, not just Sport.

Buying in sports data 
First off, you should look at refocusing Sport as a data proposition and not
just a content one.

Most major publishers typically buy in fixtures, results and tables from a
specialist sports data company. Users, and editorial teams expect that.
Depending on the amount and scale of data, this contract is usually paid
annually and is an ongoing cost. So how do you get maximum value from the data
you are buying?

The data usually contains unique IDs for teams, players and competitions, as
well as additional metadata such as dates, times, locations, rounds
(semi-finals, round 13 etc) and venues. These data feeds also contain names of
players, teams and competitions. A whole wealth of information that we could
exploit much better than we were. 

When we started looking in a lot more detail at the data that was being bought,
you could start to see the opportunities to properly join the dots between the
content proposition, the technical challenges, and what we wanted to build as a
Product. 

It wasn’t perfect the first time round we tried this, although we did get to the
stage very quickly of having a process to tag content and create automated pages
for all players, teams and groups at the 2010 World Cup
[https://www.bbc.co.uk/blogs/bbcinternet/2010/07/bbc_world_cup_2010_dynamic_sem.html]
.

We then realised that as Sport is so consistent, once you had a basic domain
model in place, then you could approach development and products in a different
way. 

Working up a Sport domain modelUsing the data to help your tagging 
Workflow is always a challenge, whether you are introducing a whole new system
or just tweaking an existing one. 

From a product perspective, I want consistent and accurately tagged content to
avoid issues with SEO, duplication, URLs and historical inconsistency across
different systems within a business. 

Other business aspects to consider were ensuring we could align the editorial
style guide, improving editorial team efficiency, enabling consistent reporting
of traffic and ad revenue, allow for cleaner syndication of content and at
times, manage tags in multiple languages (Bayern Munich and Bayern München are
the same team), and managing alternate names and synonyms for your tags.

When we first introduced manual tagging of content, we had issues with
consistency and accuracy. Even with guidance, individuals approached relevance
and accuracy in slightly different ways. 

Taking feedback from this, our next iteration was a semi-automated tagging
process using NLP techniques, where we focussed tagging only on headlines and
first 4 or 5 paragraphs of the content. The copy was cross referenced with the
sport vocabulary we had created from the bought in data and tags suggested to
journalists. 

This created a much quicker and accurate workflow for the editorial teams. 

There were unexpected challenges too, such as when a new CMS allowed creation of
multiple tags by content teams, with no editorial oversight on tag quality. The
CMS allowed audience-facing pages to be created automatically once a tag had
been used several times - even if they were incorrect. This caused major issues
for the editorial team, the SEO strategy and User Experience. 

We revisited the key terms around sport that the data supplier had delivered,
used that as a starting point for a controlled vocabulary for the editorial
team, and then used machine intelligence to automatically retag the archive of
existing articles, video and other content in the CMS, to ensure consistency. 

Linking a content model and data 
Another key part of this process is understanding the editorial workflow around
fixtures and events and where a level of automation can help save time and
effort. 

There is a considerable amount of consistency around how events are approached
within newsrooms, with editorial teams creating match previews, live coverage,
reports, photo galleries, post-match analysis etc, all of which fit within a 
content model
[https://www.cmswire.com/content-strategy/content-modeling-what-it-is-and-how-to-get-started/] 
and data strategy approach. This thinking equally applies to video and audio,
live and clips. 

As part of a workflow review we could see that the editorial teams were taking
time to manually create pages in the CMS in the days before a big weekend of
sport. 

We suggested using the bought in fixture list to drive automated creation of
pages in the CMS for journalists. We identified what templates needed creating
around a match - which links nicely with the content model work - and spun-up
pre-tagged templates in the CMS when the fixtures were announced. 

Before a match starts, we already know the teams, competition, venue, historical
records etc.  Depending on the sport, we will also know the athletes taking part
in a match (football tends to announce starting line-ups about 45 mins before
kick off) although other sports make announcements further in advance. 

Silver Oliver’s workshop flipchart diagram of editorial workflowScaling Up
This data-driven approach allows you to scale quickly. New competitions or
events can be approached from a ‘we just need some new data’ perspective and
this should fit with your ongoing data management strategy. 

This is especially relevant around major events where you may want to cover new
teams or events as part of the Olympics, Commonwealth Games or research has
shown you that your audience wants more detailed club level NFL or NRL coverage. 

If you want to create a new type of written content, such as a match report,
that is an extension of your existing content model. 

When you start adding in multiple media types used for coverage such as
photographs and social media, you can begin to see where consistency of tagging
and annotation is key to creating product propositions. 

These methods can also be used for content gathering as well - from external
suppliers to content supplied by correspondents and social media sources. 

Data visualisation
One of the challenges every media business has, is how do you stand out from the
crowd around Sport? Everyone has the same fixture list (that you have no control
over) therefore a focus on the user experience and a unique proposition helps
you stand out. 

Depending on the amount of data that you are buying, there is a lot of mileage
in offering a unique insight, and deeper analysis into the events or
competition. 

I worked on the Roboblogger project
[https://www.niemanlab.org/2016/06/the-telegraph-is-trying-to-streamline-soccer-live-blogging-with-an-automated-graphic-system/]
, part funded by the Google Digital News Innovation fund
[https://newsinitiative.withgoogle.com/dnifund/]. We really dug into the details
of what was in the football data that we bought and pushed the automated
publishing of infographics as far as we could.

It helped that we had already put in place a controlled vocabulary of sport
terms, which made the consistency between systems much easier. 

We ensured that any graphics created fitted with the visual style guide of The
Telegraph and were getting them published in under a few minutes after we
received data about the events. 

It’s not all about the website or app
Once you start digging into how a publishing business works, you can start to
see how progress could be made by thinking about an ‘open data strategy’ within
your organisation and allowing different systems to use the same vocabulary to
make life easier, although that does start changing the business focus to manage
data better - still a very overlooked area at the moment.

There are still multiple opportunities to join the data dots in businesses.
Especially linking back end systems and archiving with audience facing products.

 * Unlocking content archives. Always handy for Olympics, World Cups, and major
   events.
 * Ensuring you can map multiple, external suppliers to a single tag or ID.
 * Advertising and commercial consistency for teams and competitions and even
   individuals athletes.
 * Providing consistency between content and tags after mergers between
   companies.
 * Electronic Programme Guides (EPG).
 * Video tagging - enhancing video with structured data - both your own video
   and inbound social media video
 * Analysis and reporting.

Sport - or even an individual sport such as football - is a very consistent
domain which makes it easier to integrate into your business model, or at the
very least, look at Sport, as your first step towards utilising data and content
in a structured and consistent way.

The M word
I have avoided writing metadata all the way through this so far. 

Data suppliers provide very consistent products and can help you solve some
business and product issues. Linking together data and content to create better
products both for internal processes and audience-facing or syndication
purposes.

For example, you shouldn't have to rely on editorial teams to rebuild a section
or feature. We can do this based on the data we have. If google enables a new
rich snippet type we can add this markup etc. 

Most media organisations buy in content feeds, and although the assets are
tagged, they won’t be quite the same as the vocabulary that you use within your
organisation. Clarifying that vocabulary will save time and money between
systems and your own team.

The key to structured data is a well structured content model, a data management
strategy and metadata.

If you want to extend the model, then you can have a list of your contributors
and journalists as well. This would enable me to create a feed or even a
subscription based-product based on a question such as:

> “Give me all match reports (content type) written by Joe Bloggs (Author) about
Melchester Rovers (team) in the past 6 months (date range)”
Are you making the most of your bought data and content?
Data management is a challenge and unfortunately it does tend to get lumped in
with data science and analytics within a business.

I prefer to look at the reference data as a core building block for the Product.
I want to look at the details of the data contract to see what is being bought.
Does it fit with the current editorial strategy or is it just a historical add
on? Is this what the audience is looking for or expecting? Do they come to your
site or buy your paper for a specific type of coverage?

The cost of the data contract varies from company to company and of course also
depends on the amount of data bought. One thing in my experience that is
consistent though, is that usually only a small amount of the data is being used
by the business.

The coronavirus pandemic has caused a raft of fixture cancellations,
rearrangements, postponements and economic challenges across the board for
industries associated with sport. 

Meeting new business challenges is going to be about agility and speed of
response to new opportunities. 

Companies will have to be more agile in the future and the key to this agility
is having scalable and well marked up content that can be repurposed. 

The future
Hopefully 2021 will be a big year for Sport with a still to be defined weekly
sporting calendar, a rescheduled Olympics in Japan, a rescheduled European
football championships, a Lions Tour to South Africa and lots more. 

Companies will want to make the most of these opportunities for engaging
audiences and hopefully these examples can start some processes we have been
involved in:

 * Advertising and commercial consistency - for teams and competitions and even
   individual athletes
 * Unlocking under utilised content archives
 * Providing consistency across content and data after mergers between
   organisations
 * Electronic programme guides - a key syndication product and internal tool for
   some organisations
 * Video clips and live coverage

Around major events, other departments within your organisation that don’t
usually do sport coverage, may well want to start creating content about the
event such as documentaries etc that you may want to leverage.

Many of these ideas are not just relevant to Sport, the patterns I describe can
be applied to many domains and publishing any form of content, whether that is
in news, sport, entertainment, healthcare, science, or finance. Adding
geospatial metadata into the mix adds another dimension, and more avenues for
adding value.

Every domain has its own challenges but Sport is a very nice place to start. If
you would like to know more, please do get in touch.

Data Language is an industry leader in Information Architecture, ontology design and practical knowledge graph implementation. We use Domain Driven Design practises and user-centred design to focus information and data design on core business needs.

Find Information Architecture news and product innovation updates here, including ontology design, vocabulary management, domain & data modelling and more.

Information Architecture - News, Product Innovation and Technology

Information Architecture

Getting the most from your Sports Data

A few weeks ago Cochrane launched their COVID-19 Study Register and I described
the data and technical architecture in detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
.

I described how PICO graphs [https://data.cochrane.org/ontologies/] are used to
augment the Studies with structured linked data that describe the Population,
Intervention, Comparator and Outcome of the trial. Cochrane have now started in
earnest annotating PICO graphs onto the Studies, and these are now being
surfaced in the Study Register application [https://covid-19.cochrane.org/].

Here, we can see the labels of the PICO concepts rendered in the UI, but in the
data itself, these are arranged as a PICO graph structure conforming to the 
PICO
ontology [https://data.cochrane.org/ontologies/pico/index-en.html].

We have also added faceted search by PICO concept into the sidebar. This lets
users quickly filter down their search results using the Cochrane vocabulary
terms.

Once we have a critical mass of PICOs annotated in the register, each vocabulary
concept will be searchable in the search bar at the top of the app, extending
the previously full-text search into a controlled-vocabulary search. A user can
now find studies using a combination of terms chosen from the Cochrane linked
data vocabulary, to deliver extremely accurate results. Moreover we can combine
a metadata driven search with full-text where needed.

Schema.org
Each Study page rendered in the application is also marked up with schema.org
[https://schema.org]structured data. So while internally we are describing our
content and PICO data with the Cochrane bounded context ontologies, at render
time, we are transforming our model to the schema.org MedicalEntity model to
provide machine readable content for search engines and other content consumers.
You can read more about the schema.org medical schema here.
[https://schema.org/docs/meddocs.html]

As an example, for this Studyhttps://covid-19.cochrane.org/studies/crs-13266121 
when viewing the source of the page, you can see we are marking up the clinical
trial as a MedicalStudy in the schema.org ontology, and identifying the 
healthCondition terms, and the interventions, including the Drug Chloroquine.
Each term has associated medical codes mapped to bioinformatics taxonomies
including SNOMED-CT, MeSH, RxNorm and the ATC codes for Drugs where available.
The resulting schema.org JSON-LD for this study is :

{
  "@context": "http://schema.org",
  "@graph": [
    {
      "@type": "MedicalStudy",
      "id": "http://data.cochrane.org/studies/crs-13266121",
      "name": "NCT04333628",
      "identifier": "NCT04333628",
      "description": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
      "mainEntityOfPage": {
        "@type": "WebPage",
        "url": "https://covid-19.cochrane.org/studies/crs-13266121",
        "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
        "description": "Clinical study : Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial"
      },
      "healthCondition": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        }
      ],
      "studySubject": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        },
        {
          "type": "MedicalProcedure",
          "id": "http://data.cochrane.org/concepts/r4hp5z141f1t",
          "url": "https://data.cochrane.org/concepts/r4hp5z141f1t",
          "name": "Usual Care",
          "alternateName": [
            "Routine Prenatal Care",
            "Routine Care",
            "Standard Care",
            "Providing care according to standard",
            "SOC"
          ],
          "code": [
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "372921003"
            }
          ]
        },
        {
          "type": "Drug",
          "id": "http://data.cochrane.org/concepts/r4hp13k8jxft",
          "url": "https://data.cochrane.org/concepts/r4hp13k8jxft",
          "name": "Chloroquine",
          "alternateName": [
            "Chloroquine",
            "Product containing chloroquine"
          ],
          "code": [
            {
              "codingSystem": "MeSH",
              "codeValue": "D002738"
            },
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "14728000"
            },
            {
              "codingSystem": "RxNorm",
              "codeValue": "2393"
            },
            {
              "codingSystem": "ATC",
              "codeValue": "P01BA01"
            }
          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "name": "Pharmacological Interventions"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "name": "Other"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        }
      ],
      "subjectOf": [
        {
          "@type": "MedicalScholarlyArticle",
          "datePublished": "2020-01-01",
          "url": "https://clinicaltrials.gov/show/NCT04333628",
          "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19",
          "abstract": "19 COVID is a deadly viral disease that has been spreading around the world for several months, and is caused by a CORONA family virus (COVID-19). Following IN-VITRO evidence of the antiviral effect of CHLOROQUINE in CORONA viruses, this drug has been used empirically for COVID-19 patients and is currently recommended in Israel for the treatment of intermediate and severity disease. The mechanism of action of chloroquine is in part by inhibiting the virus distribution, and changing the intracellular acidity, the virus distribution site. The intracellular chloroquine concentration is determined by a pump called PGP that removes the drug from the cell and is activated by the drug. In the treatment of malaria, the benefit of low dosage of the drug has been shown to be effective due to the fact that the intracellular concentration of the drug is probably higher, and therefore the logic to examine this issue in COVID-19 treatment. The purpose of this study is to test whether a low dose of Chloroquine will reduce the duration of the viral shedding and prevent the disease from worsening",
          "author": [
            {
              "type": "Person",
              "name": "HaEmek Medical Center"
            }            
          ]
        }
      ]
    }
  ]
}

PICO and Schema.org
The PICO model and process (https://en.wikipedia.org/wiki/PICO_process) is now
widely adopted for framing research questions in evidence based healthcare with
the BMJ citing this model as best practice
[https://bestpractice.bmj.com/info/toolkit/learn-ebm/how-to-clarify-a-clinical-question/]
.

PICO data is applied by clinicaltrials.gov in a semi-structured tabular format
on most of their published trials and PubMed also has a PICO search tool for
querying MEDLINE, the  database from the US National Library of Medicine.

Supporting PICO would be a welcome addition to the schema.org model, allowing
any publisher of medical literature to mark up their content in a standard way
with machine readable structured PICO data as Cochrane and others are now. I can
imagine a future where Google indexes the schema.org PICO markup and provides
the ultimate PICO based search tool across all the medical literature citations
at their disposal.

At Data Language, as co-authors of the PICO Ontology, we would be more than
happy to help define some simple extensions to the schema.org MedicalEntity
model to make this happen.

PICO, Cochrane and Schema.org

Our global expertise in knowledge graph engineering are helping Cochrane respond to COVID-19. We are using linked data and semantic technologies in this mission.

Knowledge Graph Technology vs. COVID-19 | Cochrane & Data Language

How knowledge graph technology is helping Cochrane respond to COVID-19

Julian Everett

Over the past five years it has been our privilege to be a technology partner of 
Cochrane [https://www.cochrane.org/]. For anyone unfamiliar with their work,
they are globally recognised as the gold standard in evidence-based healthcare.
Historically their primary published output has been systematic reviews, which
evaluate and summarise the current state of human knowledge about specific
clinical questions. These then inform both healthcare policy and clinical
guidelines for point-of-care decision making around the world. For example, over
90% of WHO guidelines are informed by Cochrane reviews. 

The review production process starts with the formalisation of the clinical
question of interest in terms of the Population it relates to (e.g. age, sex,
condition), the Interventions being investigated (e.g. drugs and/or procedures),
the Comparators against which the Interventions are being evaluated (e.g.
placebo), and the Outcomes against which effects are measured. This is then
translated into search protocol and run against trial registries and journal
databases to identify all relevant clinical study information in the public
domain. The matches are then evaluated for risk of bias, and the subset of
results which meet Cochrane’s stringent data quality thresholds are subsequently
aggregated and analysed to determine the cumulative effect estimates. 

In essence, this is about the identification of causal relationships between
interventions and outcomes. To that end, up until now Cochrane has focussed on
double-blind randomised controlled trials. They provide the highest quality
input data: randomisation greatly reduces the likelihood of alternative
explanations for the observed results as the number of participants increases;
blinding of field researchers and participants (i.e. hiding information about
specific interventions) removes the cognitive and behavioural bias risks that
might arise if, for example, placebos are known. 

Finally, review production is not a static process. New information is being
released into the public domain all the time both in the form of additional
journal articles about completed studies as well as information about new
studies. As a result, ongoing evidence surveillance and timely meta-analysis
updates are important additional requirements.

During our time working with Cochrane, we have built a new data platform using
domain-driven design techniques which has an explicit information model encoded
into a knowledge graph at its core. The model is published here
[https://data.cochrane.org/ontologies/]. This is complemented by a loosely
coupled, service-based architecture which ensures each business capability in
the value stream is cohesively encapsulated. In a follow up article, my
colleague Paul Wilton will review how the technical architecture hangs together
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
. The platform design allows services to be very quickly deployed in new
choreographies as the business context changes - something which has proved
invaluable over the last few weeks. 

So, what have been the business function and information flow impacts of the
Covid-19 pandemic on Cochrane? Firstly it has created an unprecedented critical
demand for the most up-to-date, reliable evidence to inform frontline clinical
decision-making around the world in order to save as many lives as possible. To
that end, they have been asked by the WHO to produce a precisely scoped,
authoritative register of Covid-19 studies. This essentially inverts their
business model from one based on a wide coverage of clinical questions using
only the highest quality primary evidence to one based on a single clinical
question which urgently extracts every bit of informational value from every
piece of evidence available. That means using all types of RCTs and
interventional studies, as well as observational studies, modelling studies,
diagnostic/prognostic studies and anything else that might improve outcomes. The
surveillance requirement for updates and new evidence switched from months to
days. The cycle time for systematic review production needed to be cut, and a
new Rapid Review format has been developed as a result to ensure baseline
datasets can be pulled together as quickly as possible and then augmented with
streamlined editorial content to meet clinical need. At the same time, the
feedback loop to frontline workers was closed to ensure that their highest
priority clinical questions are the ones being addressed first.

The Covid-19 Study Register [https://covid-19.cochrane.org/] was launched
internally on Monday 30th March and publicly on Monday 6th April. In all, two
back-end platforms were integrated together and a new front-end web application
was designed, built and shipped in just under 3 weeks. It has necessarily been
our fastest ever delivery by some distance, and was only even remotely possible
because of three core enablers:

 1. A data architecture, technical architecture and continuous integration
    infrastructure which minimised the technical risks and impacts of adapting
    to the changing organisational context.
 2. A small number of individually, amazingly talented developers.
 3. The tacit/emergent team structure and pre-existing relationships between
    those developers, established over a long track record of successful
    delivery on other demanding client projects, which meant that as a delivery
    unit they functioned at a truly optimal level from day one.

Since internal launch, our focus has been on getting as much data as possible
into the information model by any and every means we can think of, as quickly as
possible and at the highest possible quality levels. This is covering expert
crowd annotation, machine learning, and other text analysis and informational
reverse engineering techniques. One of the biggest automation/ML challenges
currently is the lack of training data due to the rapid emergence and rate of
evolution of the Covid-19 domain. We have previous experience of this class of
problem from our work with media newsrooms and automated classification of
breaking news stories. It is likely the crisis will drive extensive further
innovation in this space. 

At the same time, we have started receiving comments from stakeholders and end
users across the organisation. It has been some of the most inspiring and
touching feedback we have ever received both as an organisation and as
individuals, especially in light of the wider situation right now. The high
point so far was one very senior Cochrane staff member who, in wonderfully
memorable terms, said he couldn't overstate how useful the register is to all
aspects of his work and that he's now going to shout about it to everyone who
will listen.

A beautiful moment in the midst of demanding times.

Technical Strategy

Covid-19: Rapid Response Technical Delivery at Cochrane

React and SVG make a powerful combination: Declarative UI component library
meets declarative graphics language — a match made in front-end developer
heaven.

In this mini-series we’ll be looking at rendering interactive graphical UI’s
using components built with React and SVG.

These days SVG icons are ubiquitous, but those are not the sort of components
we’re talking about.

We’re going to put together the interactive floor-plan UI using simple React
code that renders SVG elements instead of HTML. We’ll take it relatively slowly,
building through the basics into more advanced topics.

 * Part 1 — Declarative Graphics (this post)
   In which we look at the match made in heaven that is React and SVG
 * Part 2 — Arcs, Angles, and Transformations
   [https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations]
   Keeping geometry simple with groups and transformations
 * Part 3 — Clip-paths
   An adventure in composite geometry
 * Part 4 — Masks
   When clip-paths don’t cut it: masks to the rescue
 * Part 5 — Animating components 
   Where we animate individual components
 * Part 6 — Animating the viewBox 
   In which we fluidly zoom and pan our entire UI

Let’s dive straight in!

Declarative Graphics
React developers enjoy first-class support for HTML elements in JSX:

const SomeComponent = () => (  
  <div>
    <p>hi!</p>
  </div>
);

But did you know that this applies to inline SVG elements too?

const SomeOtherComponent = () => (
  <svg viewBox="0 0 100 100">
    <circle cx={50} cy={50} r={30} fill="red"/>
  </svg>
);

Out of the box we can write  components that compose SVG primitives to create
complex interactive UI’s, just as you would with more typical HTML based
components.

(Note for React-Native folks: you’ll need to install react-native-svg, which was
looking pretty good last time I worked with it. Here
[https://stackoverflow.com/a/39041997/473338] is a simple but pretty powerful
example I created a few years ago).

Simple composition of declarative graphical components (https://62zqd.csb.app/)
Notice that we did all this drawing without a single line of imperative code,
and the declarative markup of our ghosts is very readable and accessible to
anyone familiar with React — a far cry from what this would look like if we
tried to do it with the Canvas API.

All your elements are belong to us
To draw anything with SVG, we first need an <svg>element to define the drawing
context within which all descendant elements are rendered.

We can have many <svg> elements in our page, each describing an isolated drawing
context — e.g. one <svg> per icon, chart, or other snazzy graphical widget.

Alternatively, we can root many graphical components within the same <svg> and
create an entire UI in a single drawing context.

Viewport and ViewBox
At a minimum an <svg> element should specify how much screen-space it will
occupy — its viewport. This can be done with CSS styles or width and height
attributes.

<svg width="200px" height="200px">...</svg>

We’ll typically want to specify a viewBox too, to explicitly define the
“user-space” portion of our drawing that is made visible in the viewport.

Think of the viewport as a window into the world defined by the SVG
(“user-space”), and the viewBox as zoom and pan settings that control what parts
of the user-space you can see through the window.

The viewBox attribute is a string which specifies, in order, the user-space x
and y coordinates of the top-left visible corner, and the width and height of
the viewBox.

<svg viewBox="x y width height">...</svg>

For example, “-100 -100 200 200” describes a square viewBox 200 units high and
200 units wide, centred around the origin at (0,0).

<svg width="10" height="10" viewBox="-100 -100 200 200">...</svg>

If we don’t specify it explicitly we get a viewBox that maps width and height
1:1 to the viewport, with the user-space origin (0,0) at the top-left corner.

To re-iterate: the viewport describes how much space the <svg> element takes up
within your HTML page. The viewBox describes the portion of the image that is
visible within that viewport. By changing the viewBox we can zoom in or out, or
show entirely different portions of the SVG image.

The only difference between these two SVG’s is the viewBox and the background
colour (https://0oy6p.csb.app/)If the proportions of your SVG viewport in pixels
don’t match the proportions of the viewBox, the default behaviour is to preserve
the aspect ratio and centre and fit the viewBox within the available viewport.
This behaviour can be controlled by varying the preserveAspectRatio
[https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/preserveAspectRatio] 
attribute.

By default the viewBox will be centred and “fit”within the SVG viewport (
https://52fbc.csb.app/)The separation of coordinate systems between the viewBox
and the viewport means that within our SVG we can work with coordinates that
makes sense for our specific drawing.

For example, if we’re drawing an interactive widget we can make the maths easy
for ourselves by working in percentages and specifying a viewBox of “0 0 100
100”.

Or if our widget has radial symmetry and we want to easily draw from the centre,
we can use a viewBox of, say, “-50 -50 100 100” centred at the origin (0,0).

In this series we’ll be drawing floor-plans, so we could choose to work in
metres, or feet-and-inches, or whatever system of measurement our plans use.
We’re actually going to use millimetres, just for the simplicity of working in
whole numbers.

We’ll describe our floor-plan with some simple JSON, specifying the room
geometry as coordinates of corners working clockwise around the room.

Now we can set up our viewport and viewBox, allowing a little padding around the
floor-plan so it isn’t too snug in the viewport.

import data from './floorplan-data.json';
const App = () => (
  <svg 
    width="500px"
    height="500px"
    viewBox="-1000 -1000 14000 11000"
    style={{backgroundColor:'blue'}}
  >
    // ... floorplan components here ...  
  </svg>
);

With our viewport and viewBox sorted, we can get on with drawing.

Declarative Drawing Primitives
SVG includes a number of drawing primitives ranging from the simple <line> and 
<circle> to the more flexible <path>. Mozilla Developer Network has a good 
reference [https://developer.mozilla.org/en-US/docs/Web/SVG/Element] of the
available elements.

Shapes can be filled and/or stroked, where fill is the colour applied within the
bounds defined by the shape, and stroke is the colour applied to the outline of
the shape. The width and other styles of the stroke can be controlled with
attributes or CSS.

We’ll be drawing our initial floor-plans using two simple primitives: <line> and 
<circle>.

<line> describes a line between a pair of (x, y) coordinates.

<line x1={100} y1={100} x2={200} y2={200} stroke="red"/>

<circle> describes a circle of radius r centred at (cx, cy).

<circle cx={50} cy={50} r={20} fill="green" stroke="black"/>

Room with a View
To draw our building we’ll pass the data into a Floorplan component which will
render out the various elements of the floor-plan — initially just the rooms.

const Floorplan = ({ data: { rooms } }) => (
  rooms.map(r => <Room {...r} />)
);

For the Room component to draw the walls we’ll need to map over pairs of
consecutive corner coordinates, linking them together to form walls. For example
a rectangular room with corners a, b, c, and d, will have four walls
(a-b),(b-c), (c-d), (d-a).

We can pull out those pairs with a simple function, perhaps wrapped in a useMemo 
hook for efficient re-renders.

const walls = useMemo(() =>  
  coords.map((_, i) => {
    const a = coords[i];
    const b = coords[(i + 1) % coords.length];
    return [a, b];
  }),
  [coords]
);

Now drawing the walls is as simple as using the SVG <line> element to describe a
line between the pairs of connected corners.

That works, but it isn’t very pretty. Let’s extract a Wall component and a 
Corner component, and get a bit more creative.

In our wall component we’re now drawing two lines per wall — first a thick white
line, then over-drawing the white line with a thinner, dark blue, dashed line.

Corner is very simple — just a circle, stroked white and filled dark blue.

The Room component now simply composes these higher-level Wall and Corner 
components instead of SVG primitives.

https://codesandbox.io/s/react-svg-basics-rk1t3You may have noticed that we
nested our Walls and Corners in an SVG <g> element. This is a logical grouping
element — the <g> on its own doesn’t render anything visible — but it does give
us a neat way of performing transformations such as rotations and translations
on entire sub-hierarchies of our SVG, and a means of capturing events from those
sub-hierarchies by attaching event handlers to the <g> element.

Recap
In this post we saw that:

 1. We can easily draw arbitrary shapes using SVG primitives and
    straight-forward, declarative React code — no plugins, libraries, or
    imperative code required.
 2. Complex graphical UI’s can be built by composing together increasingly
    higher level components, just like any other React application.
 3. The separation of viewBox and viewport lets you draw in a coordinate space
    that suits you, then render the result in a viewport appropriate to the
    final display requirements.

In “Part 2 — Arcs, Angles, and Transformations
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations]
”, we’ll dig further into those <g> elements and look at paths and
transformations while adding camera’s to our floor-plan.

Until next time, happy drawing!

Graphical UI's with SVG and React,  part 1 - Declarative Graphics

A discussion on the synergies between GraphQL and Graph Databases, data architecture and how ontology models provide the special ingredients.

The synergies between GraphQL and Graph Databases

5 latin characters
Until recently, my understanding was that apart from 5 latin characters GraphQL
has got little or nothing to do with Graph Databases. But things are changing,
Graph Database vendors have started bringing these two technologies together. A
recent project has given me some insights.

For some background, GraphQL is a contemporary API interface standard used as an
alternative to REST. GraphQL was originally developed by Facebook. It provides a
structured query language for querying (and mutating) data, along with a runtime
implementation to invoke those queries upon your data. In pretty much all
implementations it uses JSON as its serialization format.

REST (I suspect most readers know this) is a lightweight low-level API standard
for performing uniform and a predefined set of stateless operations on data
aligned with the HTTP verbs. With REST you have to design and  build your API
interface contract for that set of operations - and while of course there are
common patterns in REST APIs, this is very much arbitrary. REST makes no
assumptions about the high level API contract or the underlying data model. The
developer defines both. With GraphQL you only define the data model (the GraphQL
schema), exposing to the GraphQL runtime and your consumers what the shape of
the data is.

With a GraphQL query you can request only the minimum data required to fulfil
your current need. Of course you can do this with a REST API too - in fact it is
a common pattern with JSON REST APIs to filter response data by JSON fields
using dot notation. But of course you need to build this yourself.

GraphQL does a lot more for you to boot, as well as out of the box query
patterns for mutating data, it also supports subscriptions (pub/sub style) - so
your GraphQL client can subscribe to server side events, and have data pushed to
it.

The developer ecosystem around GraphQL is superb now - The Apollo GraphQL
[https://www.apollographql.com] implementation works beautifully with React -
integrating with React Hooks for query and subscriptions. Apollo also has
in-browser developer tools akin to the redux-devtools for GraphQL debugging, and
you can manage local app state via Apollo using the GraphQL schemas / queries
etc.

Caching and performance
Requests to GraphQL are typically all POST requests - not great for caching you
ask ? Indeed, this is where REST has excelled for years. But the GraphQL server
implementations have come to the rescue. The Apollo GraphQL server has an
internal caching layer that supports full response caching, and fine grained
field-level caching (read more here
[https://www.apollographql.com/docs/apollo-server/performance/caching/] )

Amazon’s AWS AppSync is a fully managed GraphQL service built upon the Apollo
technologies - and can source data out of the box from other services such as
DynamoDB, ElasticSearch, REST APIs etc. With fully managed caching and real-time
data subscriptions, it really does make your developer life easy.

On the surface it still appears GraphQL has little to do with Graph Databases.
Yet increasingly we are hearing about how they should go together like bread and
butter, sausage and mash.

Not all graph databases are equal
I won't go into depth about what a Graph Database is, but am going to explore
some of the options available, and the relationship between Graph Databases and
GraphQL.

Graph databases come in several flavour combinations. First up you have RDF
graphs vs Labelled-Property graphs. The first big choice. Both are very
different [https://en.wikipedia.org/wiki/Graph_database#Graph_models]. Your
primary use case is important here. Essentially if you primarily need to walk,
query and analyse paths through your graph, then Labelled-Property graphs are
probably the way to go. If you are more interested in the nature of the
relationships between things, and creating rich semantic representations of
things, then RDF graphs hit the spot. Both types have their own well-developed
query language. Labelled-Property graphs are typically queried with Cypher or
Gremlin (eg Neo4J), RDF graphs are queried with SPARQL.

While most mainstream RDF based graph databases are fully compliant to the RDF
and SPARQL W3C standards, not all RDF graph databases are equal. Each has its
own implementation nuances and come with their own added extras (decisions,
decisions!):

 * An RDF graph database may or may not support reasoning (to some degree of the
   RDFS and OWL standards).
 * Reasoning may be at query time (backward chaining) or at write-time
   (forward-chaining).
 * The way reasoning is implemented can dramatically impact performance.
 * Add-ons and extensions for GeoSPARQL, Path analysis, Full text search,
   connections to other datastores etc.

To give some examples (DISCLAIMER - apols to those vendors not mentioned, I am
only referencing those I have worked with properly hands-on, and this is not an
exhaustive features list - it’s just a flavour) :

GraphDB [https://www.ontotext.com/products/graphdb/] (Ontotext)

 * Fully compliant with RDF/SPARQL standard;
 * Highly compliant and configurable OWL / RDFS forward-chaining reasoning
   (materialises inferred knowledge at write time);
 * Unparallelled full-text-search capability using lucene, or via bi-drectional
   connectors to external SOLR or ElasticSearch clusters.
 *  Ability to materialise post-inference fully structured JSON or JSON-LD
   objects into ElasticSearch indices or MongoDB
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

Neptune [https://aws.amazon.com/neptune/] (Amazon AWS)

 * Fully compliant with RDF/SPARQL standards;
 * No reasoning capability
 * Fully managed cloud service
 * Highly, highly horizontally scalable (primarily due to lack of reasoning)
 * Full text search capability via connector to ElasticSearch - materialisation
   into elastic is constrained (you cannot materialise user-modelled JSON
   objects into an elastic index), but delivers good full-text search utility.
 * You use this either as a Labelled-Property graph (using Gremlin) or an RDF
   graph (with SPARQL)
 * No query workbench/admin UI - you need to install your own 

Stardog [https://stardog.com]

 * Fully compliant with RDF/SPARQL standards;
 * Highly compliant and configurable OWL / RDFS backward-chaining reasoning
   (materialises inferred knowledge at query time);
 * Extremely fast online write times (due to backward-chaining model)
 * Adequate full-text-search capability using internal indexing
 * An outstanding SPARQL extension for PATH analysis (Pathfinder) - you can
   query for paths, shortest paths with weighted edges, normally only possible
   with labelled-property graphs
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

So you can see that each of these three are fully W3C compliant RDF / SPARQL
graph databases. But each has one or more killer features, that the others do
not. 

GraphDB - the post-inference connectors to ElasticSearch and MongoDB unlock some
outstanding technical and data architecture patterns out of the box.

Stardog - combines the semantics of RDF with sophisticated Path querying and
analysis.

Neptune - fully managed cloud service with unparalleled horizontal scalability,
but no reasoning.

The upshot - choose wisely based on your requirements now, and what is likely to
matter most in the medium term.

..and then enter stage right, GraphQL

Sausage and Mash or Pineapple on Pizza ?
How do GraphQL and Graph Databases fit together? As we learned earlier, GraphQL
is a REST API alternative with a structured query language combined with a
runtime for query processing and serving data to GraphQL clients (and it is
almost always implemented with JSON in mind).

So why are Graph Database vendors getting involved and what do you get for your
money?

The cynical marketer in me says those 5 latin characters definitely count for
something. Which sales manager wouldn’t love the potential energy pressure in
the ready-to-explode singularity of Graph Databases and GraphQL (while humming
‘Don’t stop me now’).

However, GraphQL does only provide another query mechanism for your graph
database alongside SPARQL or Gremlin.  Moreover it is much less expressive, and
most definitely not designed for exploiting the rich semantics of RDF and OWL. 

In fact if you go all in on a michelin-starred Graph Database, and only use
GraphQL to query it, I would argue you really probably don’t need a Graph
Database - a NOSQL JSON store would probably suffice. Of course, if you have
forward-chaining inference in your graph database (as with GraphDB), then your
GraphQL queries will benefit from the RDFS/OWL semantics you have modelled your
data with, which does add weight to the value-proposition.

Still hungry ?
Ontologies are an important part of the picture. In the RDF world an ontology
semantically describes the model and nature of your domain, and ultimately the
data structures stored in your knowledge graphs, and if inference-aware, they
describe the meaning of the data, and how we reason upon it. Typically these
models are communicated up and down our tech stack (and the responsible
developers), and often all the way out to consumers of your APIs (public or
enterprise-internal).  While we typically move data around using JSON the
linked-data aficionados will know well that if we do this using JSON-LD, we can
use the same ontology models from the knowledge graph (or a subset) for
describing the JSON-LD payloads we ship around outside of our graph database.

So do we need GraphQL to take advantage of these ontological graph models ?.  No
we don’t, we can do it with normal REST APIs (querying our graph with SPARQL)
and using JSON-LD payloads - but of course we have to build more API code to do
this.

With GraphQL though we can (we must) formally describe our data structures as
GraphQL schemas, and so we have a great opportunity here to align our GraphQL
schemas with those same ontology models that describe the data in our knowledge
graph -  this could be either a subset of our ontology landscape, or possibly
the superset, if we are using other GraphQL (served) data sources aside from the
data in the graph database. This is starting to look like very nice decoupled
data architecture with well structured cohesive data models, with possibly the
most efficient mechanism for consuming data in our web/mobile applications (a
GraphQL client).

It is looking more tasty
Adding GraphQL client interfaces to a Graph Database does make life easier - if
you are rapidly prototyping straight from the graph, or federating data into an
external GraphQL server there is a nice pattern for making a common set of well
described data models.

The Graph Database vendors have indeed jumped on-board the GraphQL sushi train
(and it’s not all about those 5 latin characters). Stardog and GraphDB (from
Ontotext) both offer GraphQL clients now.

At this point I was going to show an example of how GraphQL might be used to
query a Graph Database, but Jem Rayfield at Ontotext has already done the job in
his superb Star Wars blog post
[https://www.ontotext.com/blog/the-rise-of-the-knowledge-graph/] (if you have
got this far, then this is a must read).

Ontotext has taken the next logical step and created a ‘Semantic Object Service’
 that automatically generates GraphQL schemas bound to the underlying RDF data
model  in GraphDB, moving us towards the ontology ubiquity I described above,
where your domain ontologies are expressed in the knowledge graph, and across
your wider data-estate, including your GraphQL schemas. Nice work.

Jem concludes by saying :

> GraphQL is a very popular graph query language, well used, known and liked by
developers. The Ontotext Platform uses GraphQL to lower the barrier of entry to
knowledge graph data, whilst still providing the full expressivity and power of
SPARQL.
and this is a key message, which equally applies to Stardog and any other graph
database that may yet offer a GraphQL client. While GraphQL lowers the barrier
to entry, GraphQL gives you only as much as it is able to with respect to
querying and graph traversal;  to really unlock the power of the graph database,
you will need to use SPARQL.

But if you never plan to, and you are not interested in reasoning either, then
possibly you should ask yourself if you really need a graph database ?

What if you want to use GraphQL as the primary API technology in your stack, but
want to use SPARQL too to access the graph database (amongst other tech)?

A mixed data technology case study
In a very recent engagement, the Data Language team have been working on a Smart
Cities IoT project. The nature of the domain meant the type of data we needed to
deal with was very ‘graphy’ in nature, and the ability to describe our data
semantically and traverse a graph with SPARQL was of high utility. Inference was
less of a concern. The proof of concept for the project was brought together in
AWS with :

 * AWS MSK - a high performance Kafka message bus to consume, orchestrate, and
   distribute IoT sensor events
 * AWS Neptune to give us our scalable RDF based smart city knowledge graph
 * A set of JSON-LD REST API endpoints invoking SPARQL queries on the graph
   running in Kubernetes (EKS)
 * A suite of Kafka event processor consumers and producers deployed into
   Kubernetes mutating the graph, and emitting new events based on our business
   logic rules.
 * AWS AppSync - the GraphQL engine sourcing events and the JSON-LD REST APIs
   from Neptune, providing the primary developer interface to our client
   applications
 * A ReactJS built UI using GraphQL react-hooks to subscribe to real time events
   data, and to query data from the sensors and our knowledge graph (ultimately
   via expressive SPARQL queries).

A simplified view of the architecture looks essentially like this :


The kicker was modeling data across the entire stack using a suite of
ontologies. These ontology models describe our knowledge graph data, our IoT
message payloads in JSON-LD and were all federated in the primary application
API expressed as GraphQL schemas in AppSync. Building the ReactJS front end
using the Apollo GraphQL client and react-hooks was a very nice developer
experience.

Dessert
While GraphQL technically has little to do with Graph Databases, they are not a
perfect match like sausages and mash, GraphQL is just another API / query
technology, and not as expressive as the native graph database query languages
like SPARQL, but in itself is a high-utlity, very sweet technology.

There are some clear synergies, and do provide some of that warm comfort food
feeling when used together. GraphQL does give you an easily understood and very
developer friendly way into a graph database. Once you add the secret sauce of
domain ontologies describing both your knowledge graph and possibly even your
entire data-estate, expressed through your GraphQL schemas, then you have a very
nice pattern for a decoupled data architecture with well structured, nicely
communicated data models across your technology stack, and out to your consumers
too - making for efficient application development using the GraphQL ecosystem.
Get stuck in.