Steve Liles

Maps in web applications are a pretty common feature these days, and with
amazing libraries like Leaflet [https://leafletjs.com/] its now pretty easy to
build a great mapping experience.

Typically these maps use the Web Mercator
[https://en.wikipedia.org/wiki/Web_Mercator_projection] projection, a variant of
the Mercator projection most people are familiar with from paper maps and which
has been in common usage for hundreds of years.

The Mercator projection takes our spherical world and maps (projects) it to a
rectangle, which is very convenient for viewing on a rectangular sheet of paper
or screen, but produces a distorted view of the world that inflates the size of
objects near the poles.

Greenland is NOT actually the same size as the entire African continent!
Likewise, things near the pole that are close together in reality can appear to
be far apart, and it is very difficult to visualise the proximity of objects in
the Arctic Circle when viewed on Mercator projected maps.

Luckily, it is possible to render maps on the web using alternative projections,
although it can be tricky to get up and running. In this post we’ll have a look
at setting up an Arctic projection with React, Leaflet, and Proj4js
[http://proj4js.org/] (via Proj4Leaflet
[https://kartena.github.io/Proj4Leaflet/]).

First lets get a map up and running with the default Web Mercator projection,
then we’ll walk through the modifications necessary to display an Arctic
projection.

Mapping with React + Leaflet
Getting started with the default projection is straight-forward — we just need
to set up a container for our map and render a TileLayer with a suitable tile
provider url. As we’re just testing we’ll use tiles from openstreetmap using the
following url pattern.

const TILES = https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png


The code to render the map with React-Leaflet is short and sweet:

import { MapContainer, TileLayer } from "react-leaflet";

const App = () => (  
  <MapContainer
    style={{
      width: 1000,
      height: 1000
    }}
    center={[0, 0]}
    zoom={2}
    minZoom={1}
  >      
    <TileLayer url={TILES} />
  </MapContainer>
)

export default App;

That’s all we need to render a world map in Mercator projection. Here it is in 
codesandbox [https://codesandbox.io/s/mercator-projection-s1v59]:

Note: it is important to give your map a size (width & height) otherwise you
won’t see anything!Now that we have a map rendering in Mercator projection, lets
convert it to a polar projection, specifically the Arctic Lambert Azimuthal
Equal Area projection (Arctic LAEA for short). To do that we need to provide
Leaflet with two things:

 1. a coordinate reference system (CRS) to enable it to project and unproject
    coordinates, and
 2. a set of tile images are appropriate for that CRS

The CRS implementation comes courtesy of Proj4Leaflet, and must be parameterised
with a proj4 configuration string and details of the map tile bounds and
resolutions.

Proj4 configuration
We can find the proj4 configuration string at epsg.io [https://epsg.io] simply
by searching for the projection we want to use. There are several Arctic LAEA
projections we could use, I’m going to go ahead with epsg:3575
[https://epsg.io/3575]. The configuration string we need can be found under the
section titled “Export” for the proj4js link, which gives the following:

proj4.defs(
  "EPSG:3575",
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"
);

Map Tile Configuration
Next, we’ll need a compatible set of tiles for the EPSG:3575 projection. I’m
going to use tiles hosted by the Global Biodiversity Information Facility
[https://tile.gbif.org/ui/]. The map tile url pattern is given on that page as:

https://tile.gbif.org/{srs}/{tileset}/{z}/{x}/{y}{format}{params}

Here, {srs} is the spatial reference system (in our case 3575), {tileset} can
only be omt, {format} allows for a choice of vector or various resolutions of
png (i’ll use @4x.png), and {params} allows us to choose a tile style - only for
png tiles - where i’ll use ?style=gbif-classic, which altogether gives us a tile
url like this:

https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en

We need to provide the CRS with a little configuration related to the map tiles,
which we can derive from the extents provided by Global Biodiversity Information
Facility (see the table in the usage section). The tiles are 512x512 and the
provided extents for the northern hemisphere are:

–6371007.2 √2, 6371007.2 √2, 6371007.2 √2, –6371007.2 √2

So we can derive the extent and resolutions like this:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1)  
  .fill()  
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));

Complete Proj4Leaflet Configuration
Putting it all together we get the following:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1) 
  .fill() 
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));
  
export const ARCTIC_LAEA = new Proj.CRS(
  "EPSG:3575", 
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs", 
  { 
    origin: [-extent, extent], 
    bounds: L.bounds(
      L.point(-extent, extent), 
      L.point(extent, -extent)
    ), 
    resolutions: resolutions 
  });
  
const ARCTIC_TILES_URL = 
  "https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en";

All that remains is to update MapContainer to refer to the new CRS and center on
the pole.

crs={ARCTIC_LAEA}
center={[90, 135]}

... modify TileLayer to use the arctic tiles ...

url={ARCTIC_TILES_URL}

And we’re done! Check out the complete working code on codesandbox
[https://codesandbox.io/s/react-leaflet-arctic-bxw7d].

Boom! Arctic Map!Welcome to the Arctic!

Software Development

Arctic Projections with Leaflet and React

Paul Wilton

In my last blog post, I introduced our new SaaS product Data Graphs
[https://datalanguage.com/blog/data-graphs-fully-managed-knowledge-graphs] - a
cloud hosted Knowledge Graphs as a Service platform - effectively lowering the
barriers to entry into knowledge graph adoption. Data Graphs
[https://datagraphs.com] not only makes it trivially easy to spin up a knowledge
graph for a domain or project, but provides a very intuitive user interface for
curating and managing your data within, along with search, discovery and
integration using a richly featured API.

Since then we have been busy rolling out new features, the most striking of
which are the new visualisations. Data Graphs now lets you automatically
visualize your domain model, and lets you navigate your graph data visually too.

The Domain Model
This visualization is a superb way to verify the domain (or ontology) model you
have created in Data Graphs accurately reflects the true nature of your domain
and the semantic relationships between classes of concepts are as you intended.
This visualisation lets you see the structure of your schema, including all
primitive data properties, relationship properties, and any subclasses you have
defined. 

A typical pattern is to draw your domain model first in some external tool, or
on paper or whiteboard, then construct the schema in Data Graphs. Now you can
visually verify if the models align.

The domain model is visualized using a force-directed graph. If the automatic
render is not perfect,  you can use the controls to adjust the layout. Each node
can be pinned and moved simply by dragging it to the position you prefer.  You
can pin all nodes using one of the control bar buttons on the left of the
diagram. Subclass relationships can be toggle on / off, and you can also
remove/prune classes using the sidebar that lists all the concept types in your
project. When happy with the layout, you can save it, so next time you view it,
it will retain the visual changes you have made.

Graph Explorer
You can explore your knowledge graph visually starting from any concept in your
datasets.

There is an option in the concept view screen to “explore graph”. Clicking this
will open the data visualization using the concept selected as the starting
node.

The graph opens and you then have the ability to explore the graph. Clicking on
a node in the graph opens all its inbound and outbound relationships. It also
renders the data about the node selected in the sidebar on the right.

If a concept has a preview image defined, these will be rendered as the
canonical image for each node.

To navigate at a more granular level, right clicking on a node will open a
context menu where you have the option to open or close specific relationships,
or prune the node itself from the graph visualization.

Like in the domain model, holding and dragging a node pins it the place you move
it to.

I hope these are helpful. We have plans to continually improve these
visualization features over the coming months.

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

Knowledge Graph Visualization in Data Graphs

Data Graphs is the latest Data Language baby. Data Graphs
[https://datagraphs.com] was born out of need, a need for really easy structured
data management. Our thoughts on this evolved to eventually produce a SaaS
product that allows a team to curate a knowledge graph for any domain, from
simple flat concept collections to complex structured domain data. This is all
wrapped up in an extremely intuitive user experience, combined with a fully
comprehensive API and web-hooks to give you extremely efficient integration with
applications and business systems. You can check it out here : 
https://datagraphs.com

What does it do ?
Put simply it provides a dead simple way to create knowledge graph data models,
populate them with data, and interrogate (query) and integrate that data with
your applications using a flexible and easy API.  In essense, it is Knowledege
Graphs as a Service (KGaaS).

Building your own knowledge graph platform can take months, requites highly
specialised expertise and lot of $$, and that is before you start populating it
with data. Even when you build your own KG platform, the information management
tools dont come for free, and are typically sorely lacking in utlity.

We have short-circuited that complex engineering process, and done all the hard
yards for you. Futhermore we have added a stunning information management user
experience. With Data Graphs you can be up and running in minutes, start
creating your graph data models, and managing your data.

You don’t have to worry about scaling, engineering, backups, it’s all handled
for you. But best of all is the super-intuitive user interface for managing,
searching, and maintaining data over time. The API lets you search, query,
filter, and facet on your data - slice and dice it. The API is rocket fast and
easy to use, and also geospatially aware.

We think Data Graphs addresses a lot of use-cases, from straight-forward data
catalogs or vocabulary management, to easy to setup and maintain knowledge
graphs. You could use it for managing structured tagging data for a publishing
system, integrating it with your CMS, or even for maintaining time series
structured data powering charts in WordPress for data journalism. It is ideal as
a central metadata or reference data repository for your enterprise business
systems. You could even use it for maintaining a nicely categorised and faceted
product catalog to integrate with your e-commerce store.

Schemas, Concepts, and Datasets
The Data Graphs model is based around three fundamental ideas - 

Datasets - these are collections of Concepts - organised into groupings of your
choice. Datasets can hold concepts of different types, or concepts all of the
same type. Datasets can also be public or private -

 * a public dataset means the data it holds is accessible via API using only an
   API key, so it could be directly accessible via anyone on the internet, who
   has access to your API key.
 * a private dataset means the data it holds can only be accessed using Open ID
   / oAuth security credentials

Schemas - this is your data model, definitions of the things (or concepts) in
your domain, their properties, and how they are linked together, the
relationships between the things. Data Graphs has an intuitive user interface
that lets you quickly design a data structure that reflects your own domain
model.  Each Concept type in your domain has its own schema definition where you
define the properties of a concept and the data types of those properties
(dates, integers, text etc). For example, a simple schema of a Person might have
a givenName (text), familyName (text), and a dateOfBirth (date), and a headShot 
(image).  Rather than explain it in depth here, if you need to know more, we
have made a short video
[https://datagraphs.com/demos/how-to-setup-your-data-sets] that shows how
quickly you can create a schema for your domain model.

Concepts - the data itself. Concepts are the data instances that you load into
Data Graphs conforming to your schemas. You can create and edit concepts via the
user interface, bulk import them from for example CSV files or external sources,
or load them via the Data Graphs UI. Once created, you can search , discover,
and maintain the data via the UI.

SaaSification
Data Graphs is fully SaaSified - you can sign up online [https://datagraphs.com]
, frictionless, and get started immediately with a free trial. You can manage
your own users, roles, your own backups, your own API keys and oAuth
credentials. 

The User Experience
Delivering a great data management experience was our primary goal when creating
Data Graphs. In almost all the knowledge graph and information management
projects we have worked on, data management and curation tooling was always an
area that was a challenge. We set out to build something extremely intuitive to
use, with a UI that would be self-aware of the data types, the relationship
semantics, and make it super easy for users to explore, find, and navigate data.

Dog Food
We are great fans of the “Eat your own dog food” approach to product
development. If our APIs are not good enough for ourselves to build an
application upon, how can we expect them to be good enough for others? With this
in mind, all the features in the Data Graphs Application are built upon the same
APIs exposed to the Data Graphs client, and with that in mind, let’s talk about
integration..

Integration
There is no point having a fabulous user experience for ingesting and managing
your data if you cannot integrate it. As with most SaaS products Data Graphs
comes with an API. A set of REST endpoints for searching, filtering, querying,
faceting, retrieving and creating data. It is fast, it is flexible.

With the API you can access data only in specified datasets, or you can federate
a query across all your data.

If used with only your API key, you access only the data in public dataset. If
used with your oAuth credentials, you can access both public or private
datasets.

Web Hooks - Triggered
The API gives us pull integration, web hooks provide us with push. You can
configure web hooks for Concept create, update, delete, and also candidate
(concept) approval. You define optional endpoints (typically in your  own
platform) that are invoked whenever a specified Data Graphs event is triggered.
For example you could trigger some downstream system when a Concept of a
specific type is created or updated. For example if an Organization concept is
updated, trigger some workflow in an existing business system. The web-hook
events supply the entire concept data in its payload, which means you could use
this to push data into another database or application.

Querying data - The Data Graphs Filter Expression Language
As well as features such as sort and facet (by concept property) the API lets
you query your data using a flexible filter expression language (often called
NQL).  In simple terms, this lets us filter data like this 
filter=property1:value1,property2:value2

A filter expression is a string which provides a (Concept)  property, operator
and value in the form where property is a JSON path representing the field to
filter on (using JSON dot notation) and operator is optional , and tells the API
how to compare values. value is the value to match against.

Operators can be -(not), >, >=,<<=, and also a test if a value is in a group of
values [value,value,...]

Boolean expressions are also supported including parentheses for grouping
expressions together, and overriding operator precedence. 

Full text search is supported using the traditional q parameter. We index all
the text fields in concepts, and boost the canonical labels.

Combining all these together lets us build extremely high utility data queries,
and very flexible integrations.

You can query within a specified dataset, or you can federate across all your
data, and using the filter expression language it means you can ask extremely
useful questions of your data.

The API is fast - you can expect all requests to be returned in less than a few
hundred milliseconds.

Geospatial
We also support geospatial data. We have primitive datatypes for latitude, 
longitude and geohash - so both our API and UI can be geospatially aware for
concepts with location information. As well as being able to explore geospatial
data in the UI on maps, you can also query geospatially using point-radius,
bounding-box, and bounding -polygon queries. Any geospatial query can be
combined with other API filters and full text search.

This means you can ask questions of your data such as “Find me all the concepts
within 50km of New York, classified by X, with property P greater than Y, and
the word “hello” in their label”. Nice.

For example, given a geospatial data set of Airports with relationship to
airlines , get all airports within a 50km radius of London, serviced by EasyJet,
we might use an API request like :

/airports?filter=_near((51.51, -0.12), 50km),airlines:urn:demo:airlines:easyjet

Client Library
For developers we have published a Javascript client library for Data Graphs to
make integration even easier. The client library is particularly useful for
consuming the API using OpenID credentials, as the machine-machine OpenID
authorization flow is all taken care of.  The client library documentation is
published in NPM here : 
https://www.npmjs.com/package/@datalanguage/datagraphs-client

Sub-classing, Inheritance, and Transitive Inference
While Data Graphs is not a full blown graph database, it is a fully managed
structured data solution that can be used to build a high-utility highly
performant knowledge graph for your organization without having to manage any
infrastructure.  There are some key features in knowledge graphs that add
significant value. 

Subclassing and class inheritance is one of those features. In our (extensive)
experience of working with graph databases, inference is a feature that is
widely misunderstood. The set of potential inference rules available in the full
RDF and OWL specification is considerable. However in most practical
implementations of knowledge graphs (outside of academia), there are just one or
two core inferences that add significant value in most business applications.
Subclassing and transitivity is top of that list. Data Graphs has full support
for this.

When you create a new Concept schema definition, you have the option of
declaring a parent type (class) and inheriting its properties. 

Create a Concept with a parent typeOnce you have created your concept schemas,
we have a neat class structure management tool like this :

Concept relationships (and the User Experience) then become class structure
aware, with transitive inference.  When a concept property defines a
relationship to another concept type/class the UI allows you to select concept
instances of that type and all its subclasses.

Transitive relationship propertiesAPI queries also obey the usual semantics of
transitive inference across a class hierarchy. 

Linked Data and Graph Databases
Data Graphs has support for JSON-LD. API responses can be requested with an 
application/ld+json mime-type such that the data is returned in JSON-LD with a
context dynamically constructed from your schema.

This lets you transform data directly to RDF for loading into your own graph
database, or another RDF compliant system. 

So while functioning capably as a standlone fully managed knowledge graph, Data
Graphs could thus also be used as a tool-suite for editing data in a downstream
enterprise graph database / triple store. Using the web-hooks, it is fairly
trivial to build a service that updates entities in a graph database when the
web-hook is triggered from a concept create/update operation in Data Graphs.

The Road Ahead
Data Graphs already has a plethora of great features
[https://datagraphs.com/features], but we have some great new ones lined up in
our road map. Not least is data and model visualisation.

Building on the class structure visualisation we plan to render detailed domain
model / ontology diagrams dynamically from your schema. We also plan a dynamic,
navigable graph visualisation of the instance data itself. On the back-end, we
plan to offer regional data storage options.

We think we have produced something really special with Data Graphs, something
that really delivers utility and value for information management in your
organisation - but don’t take my word for it, go ahead try it out for free
[https://datagraphs.com].

Data Graphs - Fully Managed Knowledge Graphs

In this mini-series we’re looking at rendering interactive graphical UI’s using
components built with React and SVG. Throughout the series we’re putting
together an interactive floor-plan while working through SVG concepts, starting
from the basics and working through some more complicated areas.

If you’re new to React and/or SVG, or you just want to get some context on our
ongoing project, you might want to start from the beginning with part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics]
, Declarative Graphics, where we laid out our basic floor-plan blueprint.

Inpart 2
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-2-arcs-angles-and-transformations] 
we added cameras to our plan and showed their field of view, but there’s a
problem: the camera field of view passes through walls and around corners,
giving a false impression of what the camera can see.

In this, part 3, we’ll use clip paths to constrain the field of view to the room
that the camera is in, preventing it from overflowing into other rooms or
outside the floor-plan entirely.

Our cameras can see through walls and around corners!Boundaries
Let’s start with the easy bit: anything in the room should respect the room’s
boundaries. We already have geometry for the room that defines the constraints
that we want to apply to anything in the room, so how can we apply those
constraints to prevent overflow?

The answer, of course, is clip paths. A clip path (defined with theclipPath 
element) is just what it sounds like: a path that clips any elements to which it
is applied, such that only content that falls inside the defined path is
rendered.

In SVG we can compose clip paths from any drawable SVG elements, e.g. circle, 
rect, polygon, path, and even text!

Binocular Vision
For a simple but fun first example, let’s bring back the Pac-Man ghosts from 
part 1
[https://datalanguage.com/blog/graphical-uis-with-svg-and-react-part-1-declarative-graphics] 
and use a clip-path to pretend we’re looking at them through binoculars.

First, here’s the scene without clipping, showing our four ghosts on a blue
background:

<g>  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />
  <Ghost x={400} y={350} scale={2} colour="red" />
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

un-clipped ghostiesNow let's create our first clip path to constrain the drawing
to two adjoining circles in the old movie trope binocular shape.

<clipPath id="binoculars">
  <circle cx={500} cy={400} r={200} />
  <circle cx={800} cy={400} r={200} />
</clipPath>

A couple of things worth highlighting here:

 1. As you can see, we’ve given the clipPath an id, which is necessary in order
    to apply the clip path to another SVG element.
 2. We didn’t stroke or fill the circles — we don’t need to — because clipPath 
    doesn’t pay any attention to stroke and fill, it simply takes the entirety
    of the shapes added to it and allows content to be rendered if it falls
    within those shapes. In that sense, clipPath is all-or-nothing.

Now all we need to do is set the clipPath attribute on the element we want to
clip with the valueurl(#binoculars). In our case we just need to add that to the 
<g> element containing the ghosts.

<g clipPath="url(#binoculars)">  
  <rect x={0} y={0} width="100%" height="100%" fill="blue" />  
  <Ghost x={400} y={350} scale={2} colour="red" />  
  <Ghost x={600} y={500} scale={2} colour="yellow" />  
  <Ghost x={800} y={300} scale={2} colour="cyan" />  
  <Ghost x={900} y={450} scale={2} colour="magenta" />
</g>

The black background of the SVG shows through outside the clip pathWhat goes on
in the room, stays in the room
Back to the floor-plan. Our Room component currently renders as a group<g> 
containing walls, corners, and cameras (and I added the room name as red text in
the corner of each room). Here’s the current code and its output:

The room component from part 2 - walls, corners, cameras, and labelsWe want to
modify this component so that the contents of a Room can no longer render
outside the bounds of the room. To do that we want to do two things:

 1. Change the render order such that the contents of the room (currently really
    just the camera’s) are drawn first, and therefore are rendered beneath the
    walls and corners (in z order).
 2. Render the room contents inside a <g> element that is constrained by a clip
    path along the room’s boundaries, so that contents cannot overflow the room.

Step 1 just means pulling the camera’s up to be the first thing rendered inside
the Room’s <g> element, but why do we even need step 1? Well, our walls and
corners are rendered along the boundary of the room, but they have some
thickness and so half of the thickness of the wall falls inside the boundary of
the room.

The clip path will prevent contents of the room from being rendered outside the
boundary, but would not stop them from rendering over the inner part of the
walls, so this is a simple workaround to prevent that. A more satisfying — but
more complicated — alternative would be to adjust the clip path to account for
the wall thickness, but for now we’ll keep things simple.

Room geometry as clip path
Having pulled our camera’s up to the first-child position in the room, we can
nest it in a new group <g> element, to which we will later apply our clip path.

<g>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} />  
  ))}
</g>

Now let’s create the clip path. We need to give it an id so that we can refer to
it from the clipPath attribute of the camera group element. Our room data
includes an id so we can use that to make up the clip path id, e.g.

<clipPath id={`${id}-content-clip`}></clipPath>

To define the shape of the clip path we need to use the room boundary
coordinates to construct a solid shape. Our room geometries are just a series of
coordinates with no flashy curves or arcs, so while we could use the
all-powerful path element we have a simpler alternative in the polygon element.

The SVG polygon element gives us a convenient way to convert our coordinates
into a rendered shape: polygon takes a points attribute, which is a
space-separated list of comma-separated coordinates, like this:

<polygon points="20,20 40,30 30,30 ..."/>

This is perfect for us, because we can simply map our room coordinates array to
a string for the points attribute and we’re done!

<clipPath id={`${id}-content-clip`}>  
  <polygon points={coords.map(({x,y}) => `${x},${y}`).join(" ")}/></clipPath>

Now all we need to do is add the clipPath attribute to our camera group to apply
the clip-path.

<g clipPath={`url(#${id}-content-clip)`}>  
  {cameras?.map((camera) => (    
    <Camera key={camera.id} {...camera} /> 
  ))}
</g>

Win! The camera fields of view are now clipped to the room boundary!This is
great, with some very simple changes we’ve clipped the content of the room so
that it doesn’t spill outside the boundary!

Lines of Sight
Look closely at the floor-plan image above though, and you’ll see something’s
not quite right. Our cameras can’t see through walls any more, but they can see
around corners!

To solve this we’re going to need the camera itself to clip the field of view
according to its lines of sight. This will involve creating a more intricate
clip-path shape, but most of what follows is just a fun bit of problem solving,
so if you’re not into algorithms feel free to skip to the bottom to see the
results.

The Disjoint Segment Visibility Problem
To clip the camera field of view to what the camera would actually be able to
see accounting for occlusion due to walls blocking lines of sight, we need to
solve the Disjoint Segment Visibility Problem where the segments are our walls.

When dealing with large numbers of segments this is typically done with a
sweep-line algorithm that projects rays out from the view-point (our camera) at
discrete intervals around a full 360 degree sweep, checking for the nearest
intersection with a segment at each interval.

sweep-line algorithm at 10° intervals, red dots show nearest intersection to
cameraOne problem with the sweep-line approach is that the use of discrete
intervals (e.g. 1 degree) means that the precision is only as high as the chosen
interval allows for, and we might produce a field of view that visibly cuts
through corners of walls or misses the corner and leaves a small gap.

corners cut through or cut off due to lack of precision (interval too large)One
way to deal with this would be to use smaller intervals until we get the
precision we want, with the trade-off that the smaller the interval the more
calculations we need to make.

In our case, however, we know that each room is composed of a very small number
of segments (walls) so we can take a different approach: rather than sweep
through 360 degrees, we can check only the angles to known segment end-points,
which reduces the number of calculations we need to make, and simultaneously
allows us to achieve very high precision at little cost.

The Visibility Polygon
Our task, then, is to create a “visibility polygon” describing the critical
lines of sight from the camera position to the nearest walls of the containing
room. We’ll use a modified  sweep-line algorithm that boils down to some pretty
simple steps:

 1. Find the angle to each wall end-point from the camera
 2. Reduce to the set of unique angles
 3. Sort the unique angles into clockwise order
 4. Insert two new angles θ- and θ+ to either side of θ, where θ- is a fraction
    smaller than (CCW of)θ and θ+ is the same fraction larger than (CW of) θ.
 5. For each angle in the resulting list, find the nearest wall intersection and
    add it to our visibility polygon

At step 1, we can find the angle to each wall end-point simply by converting
each coordinate from the Cartesian coordinate system to a polar coordinate
system with the camera as the origin. The angle part of the polar coordinate is
thus the angle to the wall end-point.

Step 2 eliminates any duplicate work, for example where a camera might be
looking directly along the line of a wall such that both wall end-points are at
the same angle, or in more complex room shapes where distant corners might align
with nearer ones along the cameras line of sight.

distant corners that align with nearer ones can be eliminated from further
processingBecause rooms can potentially be composed of complicated arrangements
of walls, Step 3 is important to ensure that the points of our visibility
polygon are arranged clockwise around its boundary to produce a continuous shape
from start to finish.

It doesn’t strictly matter whether we sort now or at the end, but it is more
efficient to do it now because step 4 triples the number of points that would
need to be sorted.

wall coordinates for complex room shapes may not be clockwise ordered from the
camera perspectiveStep 4 gives us the high precision we want — the angles
fractionally before and after the angle to each wall end-point are the critical
line of sight intervals we need to test in order to get very precise results
with no visible corner cutting.

critical line-of-sight angles fractionally before and after the angle to the
wall corner pointsStep 5 converts the collected angles of critical lines of
sight into the Cartesian coordinates of our visibility polygon, by finding the
nearest intersection of the line of sight with a wall.

In javascript the whole thing looks like this:

const getVisibilityPolygon = (from, cartesianCoords) => {
  // wall corners as indexes into the cartesian coords array
  let walls = cartesianCoords.map((_, i) => [    
    i, (i + 1) % cartesianCoords.length  
  ]);  
  
  // get all points in polar coords, giving us angles from camera  
  let polar = cartesianCoords.map((c) => cartesianToPolar(c, from));  
    
  // reduce to the set of unique angles that we need to test  
  const unique = Array.from(    
    polar.reduce((r, { angle }) => r.add(angle), new Set())  
  );  
    
  // sort into clockwise order  
  const clockwise = unique.sort((a, b) => a - b);  
    
  // produce angles slightly CCW and CW of each angle in the list
  const checkPoints = clockwise    
    .map((angle) => [angle - 0.01, angle, angle + 0.01])    
    .flat();  
    
  // find the nearest intersection with a wall  
  return checkPoints.map((angle) => {
    return nearestIntersection(from, angle, walls, cartesianCoords);     
  });
}

For this to work we’ll need to add a few more utilities to our geometry toolkit:
First we’ll need to modify our polarToCartesian function so that we can pass the
camera’s Cartesian coordinates as the origin for the polar coordinates. Second,
we need to add the inverse function cartesianToPolar to convert in the other
direction.

export const REFERENCE_DIRECTION = -90;

export const degreesToRadians = (degrees) => 
  (degrees * Math.PI) / 180.0;

export const radiansToDegrees = (radians) => 
  radians * (180.0 / Math.PI);

export const vector = (from, to) => ({ 
  x: from.x - to.x, 
  y: from.y - to.y
});

export const magnitude = (vector) =>  
  Math.sqrt(Math.pow(vector.x, 2) + Math.pow(vector.y, 2));

export const distance = (from, to) => magnitude(vector(from, to));

export const polarToCartesian = (
  { angle, range }, orig = { x: 0, y: 0 }
) => {  
  let angleInRadians = degreesToRadians(angle + REFERENCE_DIRECTION);
  let cartesian = {    
    x: range * Math.cos(angleInRadians),    
    y: range * Math.sin(angleInRadians)  
  };  
  return { x: orig.x + cartesian.x, y: orig.y + cartesian.y };
};

export const cartesianToPolar = (point, orig = { x: 0, y: 0 }) => {  
  let v = vector(point, orig);  
  let range = magnitude(v);  
  let angle = radiansToDegrees(Math.atan2(v.y, v.x));  
  return { range, angle: angle - REFERENCE_DIRECTION };
};

Finally, we need a functionnearestIntersection that takes a Cartesian coordinate
of a point of interest such as our camera location, an angle, and a list of
segments (walls) to test for intersection at that angle, and returns the nearest
intersection to the point of interest as a Cartesian coordinate.

const nearestIntersection = (from, angle, walls, coords) => { ... }

This is going to come down to finding intersections between line segments, where
one segment is a wall, and the other is the line of sight — an imagined ray
projected out at angle from the camera position — so we will need a function to
test whether and where two segments intersect.

Finding the intersection of two line equations (think lines of infinite length)
is well-known mathematics, where one possible solution in code looks like this:

export const lineIntersectsLine = ([a, b], [c, d]) => {  
  let a1 = b.y - a.y;  
  let b1 = a.x - b.x;  
  let c1 = a1 * a.x + b1 * a.y;  
  let a2 = d.y - c.y;  
  let b2 = c.x - d.x;  
  let c2 = a2 * c.x + b2 * c.y;  
  let determinant = a1 * b2 - a2 * b1;  
  if (determinant === 0) {    
    // parallel lines never intersect    
    return false;  
  } else {    
    let x = (b2 * c1 - b1 * c2) / determinant;    
    let y = (a1 * c2 - a2 * c1) / determinant;    
    return { x, y };  
  }
};

This gives us the intersection point of the two line equations unless the lines
are parallel, and is a step in the right direction, but we need to know that the
point of intersection actually lies somewhere on both finite length sub-segments 
of the line equation (the wall, and the ray from the camera). We can check this
by testing that the intersection point lies within the bounds of both line
segments:

export const MARGIN_OF_ERROR = 1e-9;

const pointIsOnSegment = ({ x, y }, [a, b]) =>  
  x - MARGIN_OF_ERROR <= Math.max(a.x, b.x) &&  
  x + MARGIN_OF_ERROR >= Math.min(a.x, b.x) &&  
  y - MARGIN_OF_ERROR <= Math.max(a.y, b.y) &&  
  y + MARGIN_OF_ERROR >= Math.min(a.y, b.y);

Notice that we’re allowing a tiny margin of error here to account for errors in
floating point math. And now we can write a function that tests whether and
where two line segments intersect:

const segmentIntersectsSegment = (l1, l2) => {  
  let pointOfIntersection = lineIntersectsLine(l1, l2);  
  if (    
    pointOfIntersection &&    
    pointIsOnSegment(pointOfIntersection, l1) &&     
    pointIsOnSegment(pointOfIntersection, l2)  
  ) {    
    return pointOfIntersection;  
  } else {
    return false;  
  }
};

Which gives us everything we need to complete our implementation of 
nearestIntersection as follows:

const LARGE_DISTANCE = 1e9;

const nearestIntersection = (from, angle, walls, coords) => {  
  let to = polarToCartesian({ range: LARGE_DISTANCE, angle }, from);  
  let intersections = walls    
    .map(([a, b]) => 
      segmentIntersectsSegment([from, to], [coords[a], coords[b]])    
    )    
    .filter((intersection) => !!intersection)    
    .map((intersection) => ({      
      intersection,      
      distance: distance(intersection, from)    
    }))    
    .sort((a, b) => a.distance - b.distance);  
  return intersections?.[0]?.intersection;
};

When projecting the ray from the camera i’m using a very large distance instead
of the camera’s range, which will give us a visibility polygon defined as points
along the  walls of the room, regardless whether the camera can see that far.
 This slightly simplifies the algorithm because it means there will always be at
least one intersection for each angle.

Below you can see what happens when we test this with a more complex room shape
and a bunch of different cameras. To illustrate the visibility polygon vs the
camera field of view I have not clipped the field of view — instead I’ve drawn
both the field of view (green) and the visibility polygon (red) with some
transparency so that the intersection (orange) and difference are clearly
visible. Once the clip is applied, only the orange part will be visible.

When clipped to the visibility polygon, only the orange part of the field of
view would remainFeeding this back into our Camera component is straight-forward
and gives us the following:

Overlapping fields of view, clipped along lines of sightWhew, that got a little
heavy, but we now have a clean, simple, and fast mechanism for finding the
visibility polygon from any given point in a room, and we can use that
visibility polygon as a clip-path to constrain the camera field of view in a
realistic manner.

Recap
In this post we learned what clip-paths do, how to compose them from arbitrary
SVG shapes and paths, and how to apply them to elements or groups to constrain
what gets drawn.

We then took a bit of a dive into some fun algorithms that once again take
advantage of alternative coordinate systems to make our lives easier.

In part 4 we’ll look at masks — a more sophisticated method of constraining the
rendered output that gives us more control than clip-paths.

Graphical UI's with SVG and React, Part 3 - Clip Paths

Graphical UI’s with SVG and React, part 2 — Arcs, Angles, and Transformations

BOTT is a data ethics framework for the practical assessment of AI and predictive analytics models within their ecosystems; Bias, Outcome, Target, Trust

BOTT Framework for Practical AI Ethics - Bias, Outcome, Target, Trust

As a data company, data ethics has always been important to us, but in the last
few years, as Machine learning and AI have become front and centre as a
significant part of digital transformation in the enterprise, data ethics has
become a hot topic. It bears such a considerable weight in predictive analytics,
not just in conveying trust to consumers of machine intelligence, but also in
ensuring predictive models deliver genuine value, make unbiased predictions, and
reflect the true state of the environment within which they live. When data
ethics go wrong, it can go badly wrong, potentially bringing down an entire
organisation, with substantial direct and indirect collateral damage,
potentially even on national democracy and social cohesion - the Cambridge
Analytica incident a case in point.

We all need to take it seriously.

In 2018 Data Language was privileged to have been accepted onto the Digital
Catapult “Machine Intelligence Garage
[https://www.digicatapult.org.uk/for-startups/acceleration-programmes/machine-intelligence-garage]
” programme. As well as supporting us with a terrific programme including
compute resource, strategic help and industry engagement, Digital Catapult also
took Data Language through their ethics programme. This was like a cold-shower,
really waking us up to depth and breadth of data ethics. A realisation that data
ethics is not just about data (technically), but about the way you approach your
entire business. 

With all this in mind, we were motivated to develop a data ethics framework that
we could use internally at Data Language to assess all our predictive analytics
projects, and that could guide us practically during implementation. We call
this framework BOTT.

The BOTT Framework
BOTT (all good frameworks need a memorable acronym) stands for Bias, Outcome,
Target and Trust. Through these four pillars, it allows our data science team to
examine, and assess a predictive analytics model within its local and external
ecosystem for quality, bias, and accuracy, and ensure it meets high standards of
data ethics.

Bias
An analysis of potential bias that may be baked into the model during its
life-cycle, either literally through data or code, or intangibly through for
example, some conscious or subconscious human bias. We examine bias from the
following perspectives :

Training data bias - is the training data diverse enough and comprehensive
enough to accurately reflect the environment of the predictive model. What
potential biases might skew predictive results either for good or bad outcomes -
is there any chance these biases might exist in the training data we have used.

Algorithmic bias - is the model algorithm, its mathematical basis, or code
potentially biased?

Real-time input data - if real-time (predict time) input data is involved in
making some prediction, what might these data biases be, and how can we prevent
them from impacting outputs.

Team bias - has or could conscious or unconscious bias of the model development
team contribute to model bias, and have we mitigated against it. Often just an
open and frank team discussion around bias (with respect to the project under
consideration) is enough to flush out these sort of issues

Political, environmental, and social bias - similar to the team bias but looking
at potential external factors through a similar lens - for both ideally the team
needs to be very comfortable talking about potentially politically or socially
sensitive topics in an open, friendly and non-defensive manner.

Output sampling - essentially QA - have you done effective output sampling and
for quality assurance - does the risk of bias in the predictive model call for
independent third party reviews in QA workflows, etc

Outcome
An analysis and survey of the full range of possible outcomes of the model:

Catalogue - construct a catalogue of the potential outcome scenarios

Impact - What does success and failure of the model mean :

 * in the context of the target use-cases
 * best and worst case scenario modelling
 * model failure consequences, e.g. could somebody die if the model fails ?

Target
Analysis of who or what the model is targeted at and the context in which these
users operate.

Users -  Who are target users of the model  - who or what is the direct consumer
of the model ?

Purpose - What problem is the model solving?

Context: Is it part of an ensemble of models, is there a wider predictive
ecosystem the model sits within - what impact does this model have within this
context ?

Life span -  What is the life expectancy of the model, and how does its
environment change over time

Subversion - Can other users or consumers subvert the model for
unintended/undesirable ends?

Catalogue - Construct  a catalogue of potential unintended uses/consequences,
and ascertain if those vulnerabilities can be designed out of the architecture?

Trust
Providing some visibility of the "behind the scenes" implementation can raise
confidence in the models and foster a trust relationship between consumers and
predictive services.

Explainable AI (XAI) is becoming a field in its own right, and an important part
of machine learning and predictive analytics. Codable frameworks are emerging to
aid explainability (for example Google Model Cards
[https://modelcards.withgoogle.com/about]). This codability will be key in being
able to compare and assess models through some normative evaluation mechanism.

Transparency - what should we be exposing of the lineage and decision making
pipeline of the model prediction algorithm :

 * training cycles / training samples
 * outputs of intermediate predictions
 * decision inputs/outputs

Impact - Why should these be exposed ? - perform an impact analysis of the
possible effects of not exposing model lineage, with respect to legal,
environmental, and model improvement consequences.

Data Ethics

The BOTT Framework for Practical Data Ethics

Jim Howard

Get top tips and insights from Jim Howard on valuable use cases for your Sports Data, and avoid the common pitfalls.

Getting the most from your Sports Data - Insights and Best Practice

My name is Jim Howard and I am a freelance Product Manager specialising in
developing products using data and content. After working at the BBC for a
number of years, I have worked on projects for a number of clients including
Government Digital Service, Eurosport and Stats Perform.

I got to know Data Language after working with some of the team on big
publishing and audience-facing Sport products (World Cups, Olympics etc), when I
was at the BBC and The Telegraph.

We have been discussing some of the issues that we have had to deal with over
the years and the opportunities that are still there for companies to make more
of the data that they consume. In this post, I will try and summarise some of
the key areas we have identified and some areas that could still be developed
further.

Sport, data and content in publishing

Now that the restrictions on Sport are gradually being eased, we thought it
would be an opportunity to review some of the areas where Sport-related products
have been developed and business issues have been addressed with a more
innovative approach to the use of data (not analytics) as a core ‘building
block’ to development. 

As always in organisations, there are lots of great ideas around innovation but
the business needs a focus - and sport is an ideal test subject. As we have
found out, you can look at a major sporting event in isolation (Olympics,
Football World Cup etc), review the outcomes, decide on what areas worked and
then integrate into wider product development strategies. 

I will try and break this down into areas that will be relevant to other
products, not just Sport.

Buying in sports data 
First off, you should look at refocusing Sport as a data proposition and not
just a content one.

Most major publishers typically buy in fixtures, results and tables from a
specialist sports data company. Users, and editorial teams expect that.
Depending on the amount and scale of data, this contract is usually paid
annually and is an ongoing cost. So how do you get maximum value from the data
you are buying?

The data usually contains unique IDs for teams, players and competitions, as
well as additional metadata such as dates, times, locations, rounds
(semi-finals, round 13 etc) and venues. These data feeds also contain names of
players, teams and competitions. A whole wealth of information that we could
exploit much better than we were. 

When we started looking in a lot more detail at the data that was being bought,
you could start to see the opportunities to properly join the dots between the
content proposition, the technical challenges, and what we wanted to build as a
Product. 

It wasn’t perfect the first time round we tried this, although we did get to the
stage very quickly of having a process to tag content and create automated pages
for all players, teams and groups at the 2010 World Cup
[https://www.bbc.co.uk/blogs/bbcinternet/2010/07/bbc_world_cup_2010_dynamic_sem.html]
.

We then realised that as Sport is so consistent, once you had a basic domain
model in place, then you could approach development and products in a different
way. 

Working up a Sport domain modelUsing the data to help your tagging 
Workflow is always a challenge, whether you are introducing a whole new system
or just tweaking an existing one. 

From a product perspective, I want consistent and accurately tagged content to
avoid issues with SEO, duplication, URLs and historical inconsistency across
different systems within a business. 

Other business aspects to consider were ensuring we could align the editorial
style guide, improving editorial team efficiency, enabling consistent reporting
of traffic and ad revenue, allow for cleaner syndication of content and at
times, manage tags in multiple languages (Bayern Munich and Bayern München are
the same team), and managing alternate names and synonyms for your tags.

When we first introduced manual tagging of content, we had issues with
consistency and accuracy. Even with guidance, individuals approached relevance
and accuracy in slightly different ways. 

Taking feedback from this, our next iteration was a semi-automated tagging
process using NLP techniques, where we focussed tagging only on headlines and
first 4 or 5 paragraphs of the content. The copy was cross referenced with the
sport vocabulary we had created from the bought in data and tags suggested to
journalists. 

This created a much quicker and accurate workflow for the editorial teams. 

There were unexpected challenges too, such as when a new CMS allowed creation of
multiple tags by content teams, with no editorial oversight on tag quality. The
CMS allowed audience-facing pages to be created automatically once a tag had
been used several times - even if they were incorrect. This caused major issues
for the editorial team, the SEO strategy and User Experience. 

We revisited the key terms around sport that the data supplier had delivered,
used that as a starting point for a controlled vocabulary for the editorial
team, and then used machine intelligence to automatically retag the archive of
existing articles, video and other content in the CMS, to ensure consistency. 

Linking a content model and data 
Another key part of this process is understanding the editorial workflow around
fixtures and events and where a level of automation can help save time and
effort. 

There is a considerable amount of consistency around how events are approached
within newsrooms, with editorial teams creating match previews, live coverage,
reports, photo galleries, post-match analysis etc, all of which fit within a 
content model
[https://www.cmswire.com/content-strategy/content-modeling-what-it-is-and-how-to-get-started/] 
and data strategy approach. This thinking equally applies to video and audio,
live and clips. 

As part of a workflow review we could see that the editorial teams were taking
time to manually create pages in the CMS in the days before a big weekend of
sport. 

We suggested using the bought in fixture list to drive automated creation of
pages in the CMS for journalists. We identified what templates needed creating
around a match - which links nicely with the content model work - and spun-up
pre-tagged templates in the CMS when the fixtures were announced. 

Before a match starts, we already know the teams, competition, venue, historical
records etc.  Depending on the sport, we will also know the athletes taking part
in a match (football tends to announce starting line-ups about 45 mins before
kick off) although other sports make announcements further in advance. 

Silver Oliver’s workshop flipchart diagram of editorial workflowScaling Up
This data-driven approach allows you to scale quickly. New competitions or
events can be approached from a ‘we just need some new data’ perspective and
this should fit with your ongoing data management strategy. 

This is especially relevant around major events where you may want to cover new
teams or events as part of the Olympics, Commonwealth Games or research has
shown you that your audience wants more detailed club level NFL or NRL coverage. 

If you want to create a new type of written content, such as a match report,
that is an extension of your existing content model. 

When you start adding in multiple media types used for coverage such as
photographs and social media, you can begin to see where consistency of tagging
and annotation is key to creating product propositions. 

These methods can also be used for content gathering as well - from external
suppliers to content supplied by correspondents and social media sources. 

Data visualisation
One of the challenges every media business has, is how do you stand out from the
crowd around Sport? Everyone has the same fixture list (that you have no control
over) therefore a focus on the user experience and a unique proposition helps
you stand out. 

Depending on the amount of data that you are buying, there is a lot of mileage
in offering a unique insight, and deeper analysis into the events or
competition. 

I worked on the Roboblogger project
[https://www.niemanlab.org/2016/06/the-telegraph-is-trying-to-streamline-soccer-live-blogging-with-an-automated-graphic-system/]
, part funded by the Google Digital News Innovation fund
[https://newsinitiative.withgoogle.com/dnifund/]. We really dug into the details
of what was in the football data that we bought and pushed the automated
publishing of infographics as far as we could.

It helped that we had already put in place a controlled vocabulary of sport
terms, which made the consistency between systems much easier. 

We ensured that any graphics created fitted with the visual style guide of The
Telegraph and were getting them published in under a few minutes after we
received data about the events. 

It’s not all about the website or app
Once you start digging into how a publishing business works, you can start to
see how progress could be made by thinking about an ‘open data strategy’ within
your organisation and allowing different systems to use the same vocabulary to
make life easier, although that does start changing the business focus to manage
data better - still a very overlooked area at the moment.

There are still multiple opportunities to join the data dots in businesses.
Especially linking back end systems and archiving with audience facing products.

 * Unlocking content archives. Always handy for Olympics, World Cups, and major
   events.
 * Ensuring you can map multiple, external suppliers to a single tag or ID.
 * Advertising and commercial consistency for teams and competitions and even
   individuals athletes.
 * Providing consistency between content and tags after mergers between
   companies.
 * Electronic Programme Guides (EPG).
 * Video tagging - enhancing video with structured data - both your own video
   and inbound social media video
 * Analysis and reporting.

Sport - or even an individual sport such as football - is a very consistent
domain which makes it easier to integrate into your business model, or at the
very least, look at Sport, as your first step towards utilising data and content
in a structured and consistent way.

The M word
I have avoided writing metadata all the way through this so far. 

Data suppliers provide very consistent products and can help you solve some
business and product issues. Linking together data and content to create better
products both for internal processes and audience-facing or syndication
purposes.

For example, you shouldn't have to rely on editorial teams to rebuild a section
or feature. We can do this based on the data we have. If google enables a new
rich snippet type we can add this markup etc. 

Most media organisations buy in content feeds, and although the assets are
tagged, they won’t be quite the same as the vocabulary that you use within your
organisation. Clarifying that vocabulary will save time and money between
systems and your own team.

The key to structured data is a well structured content model, a data management
strategy and metadata.

If you want to extend the model, then you can have a list of your contributors
and journalists as well. This would enable me to create a feed or even a
subscription based-product based on a question such as:

> “Give me all match reports (content type) written by Joe Bloggs (Author) about
Melchester Rovers (team) in the past 6 months (date range)”
Are you making the most of your bought data and content?
Data management is a challenge and unfortunately it does tend to get lumped in
with data science and analytics within a business.

I prefer to look at the reference data as a core building block for the Product.
I want to look at the details of the data contract to see what is being bought.
Does it fit with the current editorial strategy or is it just a historical add
on? Is this what the audience is looking for or expecting? Do they come to your
site or buy your paper for a specific type of coverage?

The cost of the data contract varies from company to company and of course also
depends on the amount of data bought. One thing in my experience that is
consistent though, is that usually only a small amount of the data is being used
by the business.

The coronavirus pandemic has caused a raft of fixture cancellations,
rearrangements, postponements and economic challenges across the board for
industries associated with sport. 

Meeting new business challenges is going to be about agility and speed of
response to new opportunities. 

Companies will have to be more agile in the future and the key to this agility
is having scalable and well marked up content that can be repurposed. 

The future
Hopefully 2021 will be a big year for Sport with a still to be defined weekly
sporting calendar, a rescheduled Olympics in Japan, a rescheduled European
football championships, a Lions Tour to South Africa and lots more. 

Companies will want to make the most of these opportunities for engaging
audiences and hopefully these examples can start some processes we have been
involved in:

 * Advertising and commercial consistency - for teams and competitions and even
   individual athletes
 * Unlocking under utilised content archives
 * Providing consistency across content and data after mergers between
   organisations
 * Electronic programme guides - a key syndication product and internal tool for
   some organisations
 * Video clips and live coverage

Around major events, other departments within your organisation that don’t
usually do sport coverage, may well want to start creating content about the
event such as documentaries etc that you may want to leverage.

Many of these ideas are not just relevant to Sport, the patterns I describe can
be applied to many domains and publishing any form of content, whether that is
in news, sport, entertainment, healthcare, science, or finance. Adding
geospatial metadata into the mix adds another dimension, and more avenues for
adding value.

Every domain has its own challenges but Sport is a very nice place to start. If
you would like to know more, please do get in touch.

Data Language is an industry leader in Information Architecture, ontology design and practical knowledge graph implementation. We use Domain Driven Design practises and user-centred design to focus information and data design on core business needs.

Find Information Architecture news and product innovation updates here, including ontology design, vocabulary management, domain & data modelling and more.

Information Architecture - News, Product Innovation and Technology

Information Architecture

Getting the most from your Sports Data

A few weeks ago Cochrane launched their COVID-19 Study Register and I described
the data and technical architecture in detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
.

I described how PICO graphs [https://data.cochrane.org/ontologies/] are used to
augment the Studies with structured linked data that describe the Population,
Intervention, Comparator and Outcome of the trial. Cochrane have now started in
earnest annotating PICO graphs onto the Studies, and these are now being
surfaced in the Study Register application [https://covid-19.cochrane.org/].

Here, we can see the labels of the PICO concepts rendered in the UI, but in the
data itself, these are arranged as a PICO graph structure conforming to the 
PICO
ontology [https://data.cochrane.org/ontologies/pico/index-en.html].

We have also added faceted search by PICO concept into the sidebar. This lets
users quickly filter down their search results using the Cochrane vocabulary
terms.

Once we have a critical mass of PICOs annotated in the register, each vocabulary
concept will be searchable in the search bar at the top of the app, extending
the previously full-text search into a controlled-vocabulary search. A user can
now find studies using a combination of terms chosen from the Cochrane linked
data vocabulary, to deliver extremely accurate results. Moreover we can combine
a metadata driven search with full-text where needed.

Schema.org
Each Study page rendered in the application is also marked up with schema.org
[https://schema.org]structured data. So while internally we are describing our
content and PICO data with the Cochrane bounded context ontologies, at render
time, we are transforming our model to the schema.org MedicalEntity model to
provide machine readable content for search engines and other content consumers.
You can read more about the schema.org medical schema here.
[https://schema.org/docs/meddocs.html]

As an example, for this Studyhttps://covid-19.cochrane.org/studies/crs-13266121 
when viewing the source of the page, you can see we are marking up the clinical
trial as a MedicalStudy in the schema.org ontology, and identifying the 
healthCondition terms, and the interventions, including the Drug Chloroquine.
Each term has associated medical codes mapped to bioinformatics taxonomies
including SNOMED-CT, MeSH, RxNorm and the ATC codes for Drugs where available.
The resulting schema.org JSON-LD for this study is :

{
  "@context": "http://schema.org",
  "@graph": [
    {
      "@type": "MedicalStudy",
      "id": "http://data.cochrane.org/studies/crs-13266121",
      "name": "NCT04333628",
      "identifier": "NCT04333628",
      "description": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
      "mainEntityOfPage": {
        "@type": "WebPage",
        "url": "https://covid-19.cochrane.org/studies/crs-13266121",
        "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
        "description": "Clinical study : Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial"
      },
      "healthCondition": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        }
      ],
      "studySubject": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        },
        {
          "type": "MedicalProcedure",
          "id": "http://data.cochrane.org/concepts/r4hp5z141f1t",
          "url": "https://data.cochrane.org/concepts/r4hp5z141f1t",
          "name": "Usual Care",
          "alternateName": [
            "Routine Prenatal Care",
            "Routine Care",
            "Standard Care",
            "Providing care according to standard",
            "SOC"
          ],
          "code": [
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "372921003"
            }
          ]
        },
        {
          "type": "Drug",
          "id": "http://data.cochrane.org/concepts/r4hp13k8jxft",
          "url": "https://data.cochrane.org/concepts/r4hp13k8jxft",
          "name": "Chloroquine",
          "alternateName": [
            "Chloroquine",
            "Product containing chloroquine"
          ],
          "code": [
            {
              "codingSystem": "MeSH",
              "codeValue": "D002738"
            },
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "14728000"
            },
            {
              "codingSystem": "RxNorm",
              "codeValue": "2393"
            },
            {
              "codingSystem": "ATC",
              "codeValue": "P01BA01"
            }
          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "name": "Pharmacological Interventions"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "name": "Other"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        }
      ],
      "subjectOf": [
        {
          "@type": "MedicalScholarlyArticle",
          "datePublished": "2020-01-01",
          "url": "https://clinicaltrials.gov/show/NCT04333628",
          "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19",
          "abstract": "19 COVID is a deadly viral disease that has been spreading around the world for several months, and is caused by a CORONA family virus (COVID-19). Following IN-VITRO evidence of the antiviral effect of CHLOROQUINE in CORONA viruses, this drug has been used empirically for COVID-19 patients and is currently recommended in Israel for the treatment of intermediate and severity disease. The mechanism of action of chloroquine is in part by inhibiting the virus distribution, and changing the intracellular acidity, the virus distribution site. The intracellular chloroquine concentration is determined by a pump called PGP that removes the drug from the cell and is activated by the drug. In the treatment of malaria, the benefit of low dosage of the drug has been shown to be effective due to the fact that the intracellular concentration of the drug is probably higher, and therefore the logic to examine this issue in COVID-19 treatment. The purpose of this study is to test whether a low dose of Chloroquine will reduce the duration of the viral shedding and prevent the disease from worsening",
          "author": [
            {
              "type": "Person",
              "name": "HaEmek Medical Center"
            }            
          ]
        }
      ]
    }
  ]
}

PICO and Schema.org
The PICO model and process (https://en.wikipedia.org/wiki/PICO_process) is now
widely adopted for framing research questions in evidence based healthcare with
the BMJ citing this model as best practice
[https://bestpractice.bmj.com/info/toolkit/learn-ebm/how-to-clarify-a-clinical-question/]
.

PICO data is applied by clinicaltrials.gov in a semi-structured tabular format
on most of their published trials and PubMed also has a PICO search tool for
querying MEDLINE, the  database from the US National Library of Medicine.

Supporting PICO would be a welcome addition to the schema.org model, allowing
any publisher of medical literature to mark up their content in a standard way
with machine readable structured PICO data as Cochrane and others are now. I can
imagine a future where Google indexes the schema.org PICO markup and provides
the ultimate PICO based search tool across all the medical literature citations
at their disposal.

At Data Language, as co-authors of the PICO Ontology, we would be more than
happy to help define some simple extensions to the schema.org MedicalEntity
model to make this happen.

PICO, Cochrane and Schema.org

Our global expertise in knowledge graph engineering are helping Cochrane respond to COVID-19. We are using linked data and semantic technologies in this mission.

Knowledge Graph Technology vs. COVID-19 | Cochrane & Data Language

How knowledge graph technology is helping Cochrane respond to COVID-19

Julian Everett

Over the past five years it has been our privilege to be a technology partner of 
Cochrane [https://www.cochrane.org/]. For anyone unfamiliar with their work,
they are globally recognised as the gold standard in evidence-based healthcare.
Historically their primary published output has been systematic reviews, which
evaluate and summarise the current state of human knowledge about specific
clinical questions. These then inform both healthcare policy and clinical
guidelines for point-of-care decision making around the world. For example, over
90% of WHO guidelines are informed by Cochrane reviews. 

The review production process starts with the formalisation of the clinical
question of interest in terms of the Population it relates to (e.g. age, sex,
condition), the Interventions being investigated (e.g. drugs and/or procedures),
the Comparators against which the Interventions are being evaluated (e.g.
placebo), and the Outcomes against which effects are measured. This is then
translated into search protocol and run against trial registries and journal
databases to identify all relevant clinical study information in the public
domain. The matches are then evaluated for risk of bias, and the subset of
results which meet Cochrane’s stringent data quality thresholds are subsequently
aggregated and analysed to determine the cumulative effect estimates. 

In essence, this is about the identification of causal relationships between
interventions and outcomes. To that end, up until now Cochrane has focussed on
double-blind randomised controlled trials. They provide the highest quality
input data: randomisation greatly reduces the likelihood of alternative
explanations for the observed results as the number of participants increases;
blinding of field researchers and participants (i.e. hiding information about
specific interventions) removes the cognitive and behavioural bias risks that
might arise if, for example, placebos are known. 

Finally, review production is not a static process. New information is being
released into the public domain all the time both in the form of additional
journal articles about completed studies as well as information about new
studies. As a result, ongoing evidence surveillance and timely meta-analysis
updates are important additional requirements.

During our time working with Cochrane, we have built a new data platform using
domain-driven design techniques which has an explicit information model encoded
into a knowledge graph at its core. The model is published here
[https://data.cochrane.org/ontologies/]. This is complemented by a loosely
coupled, service-based architecture which ensures each business capability in
the value stream is cohesively encapsulated. In a follow up article, my
colleague Paul Wilton will review how the technical architecture hangs together
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
. The platform design allows services to be very quickly deployed in new
choreographies as the business context changes - something which has proved
invaluable over the last few weeks. 

So, what have been the business function and information flow impacts of the
Covid-19 pandemic on Cochrane? Firstly it has created an unprecedented critical
demand for the most up-to-date, reliable evidence to inform frontline clinical
decision-making around the world in order to save as many lives as possible. To
that end, they have been asked by the WHO to produce a precisely scoped,
authoritative register of Covid-19 studies. This essentially inverts their
business model from one based on a wide coverage of clinical questions using
only the highest quality primary evidence to one based on a single clinical
question which urgently extracts every bit of informational value from every
piece of evidence available. That means using all types of RCTs and
interventional studies, as well as observational studies, modelling studies,
diagnostic/prognostic studies and anything else that might improve outcomes. The
surveillance requirement for updates and new evidence switched from months to
days. The cycle time for systematic review production needed to be cut, and a
new Rapid Review format has been developed as a result to ensure baseline
datasets can be pulled together as quickly as possible and then augmented with
streamlined editorial content to meet clinical need. At the same time, the
feedback loop to frontline workers was closed to ensure that their highest
priority clinical questions are the ones being addressed first.

The Covid-19 Study Register [https://covid-19.cochrane.org/] was launched
internally on Monday 30th March and publicly on Monday 6th April. In all, two
back-end platforms were integrated together and a new front-end web application
was designed, built and shipped in just under 3 weeks. It has necessarily been
our fastest ever delivery by some distance, and was only even remotely possible
because of three core enablers:

 1. A data architecture, technical architecture and continuous integration
    infrastructure which minimised the technical risks and impacts of adapting
    to the changing organisational context.
 2. A small number of individually, amazingly talented developers.
 3. The tacit/emergent team structure and pre-existing relationships between
    those developers, established over a long track record of successful
    delivery on other demanding client projects, which meant that as a delivery
    unit they functioned at a truly optimal level from day one.

Since internal launch, our focus has been on getting as much data as possible
into the information model by any and every means we can think of, as quickly as
possible and at the highest possible quality levels. This is covering expert
crowd annotation, machine learning, and other text analysis and informational
reverse engineering techniques. One of the biggest automation/ML challenges
currently is the lack of training data due to the rapid emergence and rate of
evolution of the Covid-19 domain. We have previous experience of this class of
problem from our work with media newsrooms and automated classification of
breaking news stories. It is likely the crisis will drive extensive further
innovation in this space. 

At the same time, we have started receiving comments from stakeholders and end
users across the organisation. It has been some of the most inspiring and
touching feedback we have ever received both as an organisation and as
individuals, especially in light of the wider situation right now. The high
point so far was one very senior Cochrane staff member who, in wonderfully
memorable terms, said he couldn't overstate how useful the register is to all
aspects of his work and that he's now going to shout about it to everyone who
will listen.

A beautiful moment in the midst of demanding times.