Paul Wilton

Ear of bat, blood of frog, rabbit’s tail. Stir them all up in your graph
database, utter the magic words, and lo, with a sprinkling of sparql dust, gold
speweth forth.

After working with graph databases for 10 years now, it troubles me that this is
still how many implementers expect things to work. Graph databases are not magic
cauldrons. They really are not.

A graph database is no different from any other database. If you put in data
mess, do not expect order to be created from chaos. If you then combine this
with over constrained and over-engineered inference and reasoning you end up
with an even bigger mess. A popcorn machine.

You will not be able to build robust practical business or enterprise grade
applications by treating your graph as some dirty place where ontologists (who?)
 can dip their fingers in, mess around ad-hoc, and make magic happen. “let’s
just add another inference rule, that will sort it..”

Just like any other practical, robust data architecture, data governance is
essential. Just like any other database you would not expect end users to
interface directly with it via a SQL endpoint, or a JDBC connector? No, you
would build robust, well tested APIs and Services to interface with your
database, and let your application builders interface with these services. End
users would then interact with these applications. These API services are the
gatekeepers of good data governance.  No mess should pass this point. In
production, no human hands should *ever* pass this point. 

Any changes to models should only be done to meet some business need or
requirements. Any changes to models should be tested.

Inference and reasoning should be included in your models *only* to meet some
practical value-deriving business need.  If the inference does not deliver any
meaningful or practical value to your end users, it should not be there. 

If you can deliver a business requirement effectively and easily without
inference, do it without inference.

Your models should be as simple as possible to deliver your business
requirements. A SQL data architect would not just add additional random tables
into their SQL model to deliver their requirements, so when you develop a
practical ontology model for your domain, only model what you need to deliver
those requirements. You do not need to model extra stuff. 

While your domain model may extend well beyond the bounds of your business
requirements (in order to better inform requirements and business landscape),
your detailed ontology models should not. By only engineering the very minimum
of the model needed to deliver your current requirements, your platform is
easier to evolve, easier to maintain, and at a much lower total cost of
ownership.

Finally - end users should not have access to your production graph database
sparql or cypher endpoint. Never. 

Stop pretending you are witches. Start thinking like traditional data
architects.

Graph databases

Graph Databases are not Magic Cauldrons

A case study on a recent engagement in the publishing industry using Data Graphs
for centrally managing cross-publication content metadata combined with our Text
AI NLP service for auto-tagging content to provide a high-utility information
backbone.

The Snap
Tagging content has been a publishing staple for two decades, helping publishers
organise content, perform rudimentary content analytics, and also deliver tag
based user journeys and content aggregations. In many cases publishers use the
tag features built into their CMSs. WordPress and its derivatives are so widely
used, yet still have a very weak content tagging capability. This lack of good
information management and structured metadata often leads to a tag mess over
time, as tags cannot be properly disambiguated, duplicates invariably get
created, and journalists apply tags inconsistently. Silver Oliver has written an 
excellent series of articles
[https://datalanguage.com/blog/why-the-web-turned-its-back-on-the-librarians] on
this subject that are worth revisiting.

This is where more contemporary information management technology and AI can be
a game changer. I outline below how we have used our own Text AI and Knowledge
Graph products to solve these issues, and drastically improve how structured
metadata is used at a large global news publisher.

The Back Story
In the context of this project, we were not dealing with a single CMS, but
multiple news publications, each with their own publishing technical stack, CMS,
mechanism for tagging, and entirely separate tag sets. The ambition was to
centralize the metadata used to tag on each publication, and use AI to aid the
journalists, essentially raising the consistency level of tagging across news
desks, and streamlining and optimising the tagging process. Downstream, this
would unlock better user journeys for consumers, better content analytics, and
provide for better and more rapid cross-publication innovation.

To achieve this, we integrated:

 1. Our product Data Graphs to manage a large corpus of the cross-publication
    tags, constructed as a light-weight knowledge graph of news concepts,
    including concept types such as People, Locations, Organisations, Events,
    Brands, Sports teams and athletes, Categories and Topics.
 2. Our AI Text Classification service, Tagmatic - which predicts the
    “aboutness” of content by tagging using the concepts managed in Data Graphs,
    continuously learning on-the-fly as new articles are published.

The Storyline
Our two products Data Graphs and the Text Classification AI work seamlessly
together as an ideal solution for an information backbone in the new generation
of publishing technology ecosystems.

We move from the legacy pattern of tagging with flat tags, to tagging with well
modelled and organised structured data, where each concept we tag with has a
globally unique, persistent and durable identifier (a URI / URN), and has its
own metadata properties that can be used to provide more information about that
concept to help users disambiguate (types, thumbnails, images, descriptions,
categories etc) and also provide better user experiences and journeys with these
concept tags.

Managing this metadata centrally outside of your CMS provides a number of
advantages:

 1. Your tags are not tightly coupled to one CMS (with respect to identity,
    structure, access and availability etc) - giving you an easy upgrade path or
    migration to a different CMS in the future
 2. The ability to easily use that metadata in other parts of your publishing
    stack or share with other CMSs. For example, the same metadata can then be
    used to tag or classify source feeds upstream from content authoring such
    that external content and your own content are tagged using the same
    classifications and concepts.
 3. Decouple the management of the metadata from the CMS, such that a librarian
    or information specialist takes on that task, leaving your journalists to do
    what they do best and author content.

To do this, Data Graphs allows the publisher to:

 1. Create a domain model of their core entities (People, Places, Orgs, Sports,
    Events, Topics, Content Types, and any other metadata they need to drive
    their publishing processes and user experiences)  and how they prefer to
    classify and relate these concepts to each other.
 2. Ensure these concepts are modelled with all the structured data you need for
    concept disambiguation, categorisation, and concept driven user journeys.
 3. Easily populate this model with the instances (taxonomies) of this metadata
    (essentially, publishing reference data) using either the Data Graphs API or
    CSV Uploaders in the Data Graphs web app.
 4. Manage that metadata with a dedicated application, with an intuitive user
    experience.
 5. Integrate the metadata with any of your business systems using its APIs and
    Web Hooks, including your CMSs and other data or content stores.

Using machine intelligence to automatically tag concepts and classifications of
your articles, also provides a number of advantages:

 1. It makes the task of tagging and classifying content easy and faster. Many
    journalists don’t like having to tag their articles, and may not understand
    the value of tagging well. Because the AI learns from all journalists, and
    from golden documents, this means the quality level is raised for all
    authors, and consistency is improved across all news desks.
 2. The AI can be used to tag content outside of the CMS, completely offloading
    this task from the journalists - either using a human-in-the-loop pattern
    (where a totally different person/role validates the quality of the metadata
    the AI is applying), or in a fully automated mode, for say long tail content
    where you may not be so concerned about occasional mistakes being made,
    where the cost of human supervision might exceed the value realised.
 3. The AI can be used to tag content upstream from your CMS, tagging external
    feeds (Reuters, AP, Social feeds etc) with the same metadata you apply to
    your own content. This can deliver much more efficient tooling for the
    processing of this large content funnel on your news desks.

The Data Language Text Classification AI (Tagmatic)  is designed for exactly
this task. It is a high-performance scalable service that learns how you
classify and tag your content, and then applies metadata to new content that is
sent to it for prediction. The service is highly scalable, fully private, and
designed to adapt to each publisher’s unique tagging style. Its REST API allows
for very easy integration with your CMS and publishing stack, such that it
learns continuously on-the-fly, reacting quickly to new concepts that arise in
the ever changing window of news.

Data Graphs and Tagmatic work seamlessly together, through a dedicated
connector, such that when a concept is merged or removed in Data Graphs,
Tagmatic automatically adjusts its models’ weights to reflect the new state.

The Explainer
Technically Data Graphs and our Auto-tagging Text AI work together as shown in
the diagram below. In this case study a custom WordPress metadata plugin was
created that integrated with both the Data Graphs and Tagmatic APIs to provide a
human-in-the-loop tagging UI directly in the publisher’s CMS:

Where:

 1. The article is sent to the Tagmatic Predict service API to suggest concepts
    to tag with, and these are presented to the user in the WordPress plugin.
    The classes of the concepts are used to present these in a tagging template
    suitable for the type of content being authored (News, Sport, Politics etc).
    The user can then keep or remove the tags that seem incorrect.
 2. A user can lookup missing or new concepts in Data Graphs via
    typeahead/autocomplete widgets in the WordPress plugin by using the Data
    Graphs Search API
 3. When the article and metadata is saved / published, WordPress sends the
    content and tags back to Tagmatic to its Train API, so it can use this for
    continuous improvement of its prediction models.  This allows it to learn
    brand new tags from their first appearance in  an article precluding the
    need for scheduled long running training cycles, and instead learns
    on-the-fly.
 4. If the librarian or information specialist removes a tag concept in Data
    Graphs, it automatically fires a webhook to the Tagmatic connector, so
    Tagmatic can disable the classifier model for that concept. If a user merges
    two concepts in Data Graphs, the connector will trigger Tagmatic to re-point
    its classifier from the source concept to the target concept of the merge
    operation - and adjust its weights accordingly.

In the more detailed images below you can see how this WordPress metadata plugin
might look, and how a concept is presented to the user with enough information
from Data Graphs to correctly disambiguate the tag.

Once tags are sent back to the training API with the content, Tagmatic makes
tagging and classification metrics available via another service API that
includes the precision, recall, F1 scores, support for each tag, and also the
date the tag was last trained. These metrics can then be used to provide
insights including usage analytics and patterns to the editorial floor.

The Finished Article
Bringing these technologies together creates a high-utility information backbone
for the modern newsroom, streamlining processes, automating others, and
unlocking publishing innovation.

By centralising and decoupling metadata from your CMS, and introducing machine
intelligence, you unlock a number of key capabilities: 

 * Better information and data governance, the importance of which cannot be
   understated
 * The ability to cross-pollinate your reference data across third party content
   and other internal systems, including both content and user analytics.
 * The ability to automate the classification of upstream content feeds with the
   same metadata, which allows you to scale out the funnel of source material
   coming into the news desks, and provide better search and discovery
   capabilities to those journalists.
 * Making the journalists job easier - letting them focus more on authoring good
   content
 * The ability to automate the aggregation of content, for example for topic
   pages, using structured through metadata driven content query APIs
 * Let the CMS do what it does best: the authoring and structuring of the
   content itself

Ultimately, this technology makes tags and concepts management a first class
citizen in the publishing stack. No longer just an afterthought inherited from
and tightly coupled with the CMS of the day - but something that is built to
stay even when the authoring tooling changes, unlocking the true value of your
content metadata.

Publishing Technology is a core discipline at Data Language. Our AI for Auto-tagging content and text classification combined with your product Data Graphs can provide a rich information backbone for the modern newsroom.

Publishing Tech

Case Study: How Data Graphs and AI Auto-Tagging transforms Publishing

How Cochrane has integrated Data Graphs [https://datagraphs.com] with their own
enterprise graph database of evidence-based healthcare as a complete linked data
management solution.

Population
Data Language has been working with Cochrane as a technical partner for a number
of years now, and we have blogged extensively
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19] 
on the digital transformation at Cochrane, including how linked data and an RDF
based knowledge graph have become a core information backbone for their business
processes.

The Cochrane vocabularies of some 300k concepts provide the linked reference
data that describes the PICO domain of evidence-based healthcare, describing the
Populations (and their conditions, ages, sexes, social context), the
Interventions (drugs, devices, procedures, materials etc), Comparisons, and the
Outcomes. These concepts are annotated onto clinical evidence as PICO
micrograph
data structures [https://data.cochrane.org/ontologies/pico/index-en.html] to
describe the questions being answered in clinical evidence. All of Cochrane’s
vocabularies, content metadata and PICO annotations are stored in an enterprise
graph database, and exposed by a cohesive set of microservice APIs that can be
consumed and recombined to allow rapid innovation.

Data Language has provided the technical architecture, design and implementation
of the knowledge graph, microservices, APIs, and tooling that allows the
Cochrane subject matter experts to curate and QA these PICO data structures as
part of their daily business processes. This linked data platform has allowed
Cochrane to rapidly innovate: enhancing the Cochrane Library with linked data
query capability; and build at lightning speed, the largest Covid-19 database of
clinical evidence in the world, comprising more than 100k studies, described
with well-structured PICO metadata,  serving high volumes of requests and
queries to researchers globally.

Condition
While tooling and processes for Cochrane’s SMEs to create and maintain PICO
graphs describing the Cochrane systematic reviews and studies is state of the
art, the tooling around the management of the reference data vocabularies is
less sophisticated. The Cochane vocabularies consist of a highly curated set of
linked data concepts that form a graph in their own right with class and
property semantics, transitivity, and multi-parent taxonomic structures.
Although Cochrane has an existing visual editor tool to perform operations on
individual concepts in the reference data, it is limited in functionality, and
does not allow for bulk ingest and updates, rich search and discovery operations
for quality assurance, data cleansing, and other operations such as merge,
migrate, and workflow.

To solve the vocabulary management problem, we have integrated our product Data
Graphs [https://datagraphs.com] into the Cochrane linked data ecosystem.

What is Data Graphs ?
Data Graphs is a SaaS product designed to lower the barrier of entry to
Knowledge Graphs. While it can be used purely as a Knowledge Graph in its own
right, it can also be used as a linked data vocabulary management tool, and
integrated with an enterprise graph database, or other business systems using
its API and Web-hook event triggers.  It has an intuitive user experience for
managing linked data, and powerful search, discovery and visualisation
capability, effectively making an ideal vocabulary management tool for Cochrane.

Data Graphs allows you to design or replicate any domain model or ontology, and
populate it with linked data, so it was a straightforward task to replicate the
existing Cochane concept model
[https://data.cochrane.org/ontologies/core/index-en.html] and ingest their
vocabularies. The screenshot below shows the cochrane model recreated in Data
Graphs. 


Cochrane Core Concept Model
Data Graphs provides a number of benefits to Cochrane for managing their linked
data vocabularies :

 * Fully decoupling the workflow and processes around managing their linked
   reference data from their business critical systems.
 * Being able to flexibly and easily modify their core concept model without
   impacting their downstream processes.
 * An intuitive user experience for search, discovery, match, merge and
   migration of data.
 * Graph visualisation for exploring their linked data, and identifying
   anomalies and circular relationships.
 * Easy identification of duplicates
 * A full audit history of all changes made to the vocabularies
 * Injection workflow states into the linked data concepts - for managing
   pre-publication, publication and deprecation of terms.

Cochrane’s Data Architecture
Cochrane’s linked data architecture is centred around an enterprise graph
database (Ontotext’s RDF-based GraphDB
[https://www.ontotext.com/products/graphdb/]), a set of microservices and online
tools for annotating systematic reviews and clinical studies with creating well
formed PICO graphs that describe the evidence. A slice of the logical
architecture  is shown below and is discussed in more detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
:

Cochrane Linked Data ArchitectureIntervention
Integrating Data Graphs
In order to integrate Data Graphs we needed an integration pattern that is
robust and also allows Cochrane SME’s to understand the impact of changes they
may make to the vocabularies, such that they can make an informed decision
whether or not they should proceed with the updates.

For integration with an organisation’s business systems, Data Graphs comes with
an expressive, contemporary REST API, serving JSON-LD payloads (pull), and a
suite of webhooks that can be used for event-driven integrations (push) that can
be configured for different core operations, Create, Update, Delete, Merge etc.
For Cochrane’s purposes the webhooks are ideal. 

Pre-Commit Hooks
Furthermore, for each web-hook, a pre-commit remote endpoint can be configured,
which will be invoked before a write operation is committed in Data Graphs, on
which the target business system can respond with a payload of information that
Data Graphs will render to the user as an impact-assessment if the transaction
is completed. This gives the user an opportunity to back out. These pre-commit
hooks allows the target system to respond for a variety of scenarios :

 1. Silent commit - no impact, just commit the transaction silently, the user is
    not blocked.
 2. Force reject - in which the target system determines the transaction should
    not proceed, and the user is not allowed to commit the change.
 3. An impact assessment, with a risk score, an optional summary of the impact,
    and an optional table of data, that the Data Graphs user will be able to
    make an informed decision upon.

Resilience
When consuming web-hooks (essentially an HTTP endpoint or API in a target
business system, that is configured and invoked on some trigger condition), we
need to ensure to all reasonable endeavours that the event fired will not be
lost due to unforeseen circumstances (eg target system failures, connectivity
issues etc.).  To achieve this, the target system endpoint needs to be as robust
as possible and ideally decoupled from other downstream systems. 

The best way of achieving this in the target system is to build an extremely
lightweight microservice that simply receives the HTTP event payload and places
it on a topic or queue. This queue can then be consumed at the target system's
leisure and can cope with failure downstream (the message will not be lost if
the consumer fails temporarily). The target system can take this further with
dead-letter queues and retries to cope with a variety of challenging
circumstances. This is the pattern adopted at Cochrane and is a pattern we would
generally recommend for consuming webhooks between other products and platforms.
You can see the logical architecture for this below :

Data Graphs Integration Architecture at CochraneThe key integration points
(numbered in red)  within the Cochrane technical stack are :

 1. The Pre-commit web-hook handler. This is a simple webservice that is invoked
    by Data Graphs for a specific transaction event before the transaction is
    committed (for example a Concept Delete). The service makes a query against
    the Cochrane evidence graph to ascertain the impact the change will have.
    For example if a concept is used on thousands of PICO graphs, it is likely
    the user should not be deleting this concept, and the service can respond
    with a straight reject, or return a table of the PICOs affected.  This
    information is presented back to the user to make an informed decision.
 2. The Web-hook handler. This is a lightweight service that is invoked by Data
    Graphs for a specific transaction type (Concept Delete, Merge, Update etc)
     after the change is committed. This service simply places the data event on
    a queue, ready to be processed.
 3. Concept Event Handler. This service does the heavy lifting, taking events
    off the queue, and writing updates to concepts and PICOs in Cochrane’s Graph
    Database using SPARQL. If this fails for whatever reason, the message is
    retained on the queue, and can be tried later. For Cochrane much of this
    code already existed in the legacy LDGraphs API, which was performing a
    similar function for their original linked data editor, so was a case of
    porting this into the event handler.

The webhooks in Data Graphs are easy to configure, as can be seen on the
following screenshots.

First, choose an event type to trigger a web-hook, then configure the endpoint
in the target system, an optional pre-commit hook, and any headers / security
tokens that need to be sent with the request :

Webhook Configuration in Data GraphsMigrating the vocabularies into Data Graphs
The final step was to load the Cochrane linked data vocabularies into Data
Graphs, conforming the domain model we created earlier. This can be done in
different ways

 * Using the Data Graphs REST API
 * Using CSV upload using the CSV to Ontology mapping tool within Data Graphs

In this case, the API was the easiest approach as we could read pages of JSON-LD
concepts from Cochrane’s concepts API, and pipe them into the Data Graphs write
API. As the ontology model’s in both were identical, this was a relatively
trivial migration script.

Outcome
With this in place, Cochrane now has a state of the art linked data management
toolkit. The fully whitelabelled user experience engenders user-ownership.
Cochrane’s information experts can organise their core entities in logical
datasets for rapid navigation:

Cochrane Datasets
Drilling into each dataset, a SME can search for concepts, and easily find
potential duplicates through a complex query or use the label/property facets :

Faceted SearchAnd merge those duplicates with an intuitive UI :

Merging Concepts in Data Graphs
The resulting merge transaction will be handled by the webhook event, not just
to update and merge the concepts in the Cochrane Graph Database, but also update
all PICO graphs in which the source concept was previously used to reference the
target concept instead.  Similarly, deletes will remove the concept reference
from all content and PICO annotations.

Making any changes, such as deleting a concept results in the pre-commit hooks
requesting an impact assessment, so the user can make an informed decision to
continue or cancel:

Pre-commit Impact AssessmentsThe knowledge graph can be explored visually to
help find and correct anomalies in the vocabulary, such as circular
relationships :

Graph Explorer in Data GraphsThe Cochrane bioinformatics SMEs now have many more
capabilities at their disposal taking their ability to deliver the next
generation of evidence-based healthcare innovations to a new level.

.

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

Data Graphs at Cochrane: a Case Study

In my last blog post, I introduced our new SaaS product Data Graphs
[https://datalanguage.com/blog/data-graphs-fully-managed-knowledge-graphs] - a
cloud hosted Knowledge Graphs as a Service platform - effectively lowering the
barriers to entry into knowledge graph adoption. Data Graphs
[https://datagraphs.com] not only makes it trivially easy to spin up a knowledge
graph for a domain or project, but provides a very intuitive user interface for
curating and managing your data within, along with search, discovery and
integration using a richly featured API.

Since then we have been busy rolling out new features, the most striking of
which are the new visualisations. Data Graphs now lets you automatically
visualize your domain model, and lets you navigate your graph data visually too.

The Domain Model
This visualization is a superb way to verify the domain (or ontology) model you
have created in Data Graphs accurately reflects the true nature of your domain
and the semantic relationships between classes of concepts are as you intended.
This visualisation lets you see the structure of your schema, including all
primitive data properties, relationship properties, and any subclasses you have
defined. 

A typical pattern is to draw your domain model first in some external tool, or
on paper or whiteboard, then construct the schema in Data Graphs. Now you can
visually verify if the models align.

The domain model is visualized using a force-directed graph. If the automatic
render is not perfect,  you can use the controls to adjust the layout. Each node
can be pinned and moved simply by dragging it to the position you prefer.  You
can pin all nodes using one of the control bar buttons on the left of the
diagram. Subclass relationships can be toggle on / off, and you can also
remove/prune classes using the sidebar that lists all the concept types in your
project. When happy with the layout, you can save it, so next time you view it,
it will retain the visual changes you have made.

Graph Explorer
You can explore your knowledge graph visually starting from any concept in your
datasets.

There is an option in the concept view screen to “explore graph”. Clicking this
will open the data visualization using the concept selected as the starting
node.

The graph opens and you then have the ability to explore the graph. Clicking on
a node in the graph opens all its inbound and outbound relationships. It also
renders the data about the node selected in the sidebar on the right.

If a concept has a preview image defined, these will be rendered as the
canonical image for each node.

To navigate at a more granular level, right clicking on a node will open a
context menu where you have the option to open or close specific relationships,
or prune the node itself from the graph visualization.

Like in the domain model, holding and dragging a node pins it the place you move
it to.

I hope these are helpful. We have plans to continually improve these
visualization features over the coming months.

Knowledge Graph Visualization in Data Graphs

Data Graphs is the latest Data Language baby. Data Graphs
[https://datagraphs.com] was born out of need, a need for really easy structured
data management. Our thoughts on this evolved to eventually produce a SaaS
product that allows a team to curate a knowledge graph for any domain, from
simple flat concept collections to complex structured domain data. This is all
wrapped up in an extremely intuitive user experience, combined with a fully
comprehensive API and web-hooks to give you extremely efficient integration with
applications and business systems. You can check it out here : 
https://datagraphs.com

What does it do ?
Put simply it provides a dead simple way to create knowledge graph data models,
populate them with data, and interrogate (query) and integrate that data with
your applications using a flexible and easy API.  In essense, it is Knowledege
Graphs as a Service (KGaaS).

Building your own knowledge graph platform can take months, requites highly
specialised expertise and lot of $$, and that is before you start populating it
with data. Even when you build your own KG platform, the information management
tools dont come for free, and are typically sorely lacking in utlity.

We have short-circuited that complex engineering process, and done all the hard
yards for you. Futhermore we have added a stunning information management user
experience. With Data Graphs you can be up and running in minutes, start
creating your graph data models, and managing your data.

You don’t have to worry about scaling, engineering, backups, it’s all handled
for you. But best of all is the super-intuitive user interface for managing,
searching, and maintaining data over time. The API lets you search, query,
filter, and facet on your data - slice and dice it. The API is rocket fast and
easy to use, and also geospatially aware.

We think Data Graphs addresses a lot of use-cases, from straight-forward data
catalogs or vocabulary management, to easy to setup and maintain knowledge
graphs. You could use it for managing structured tagging data for a publishing
system, integrating it with your CMS, or even for maintaining time series
structured data powering charts in WordPress for data journalism. It is ideal as
a central metadata or reference data repository for your enterprise business
systems. You could even use it for maintaining a nicely categorised and faceted
product catalog to integrate with your e-commerce store.

Schemas, Concepts, and Datasets
The Data Graphs model is based around three fundamental ideas - 

Datasets - these are collections of Concepts - organised into groupings of your
choice. Datasets can hold concepts of different types, or concepts all of the
same type. Datasets can also be public or private -

 * a public dataset means the data it holds is accessible via API using only an
   API key, so it could be directly accessible via anyone on the internet, who
   has access to your API key.
 * a private dataset means the data it holds can only be accessed using Open ID
   / oAuth security credentials

Schemas - this is your data model, definitions of the things (or concepts) in
your domain, their properties, and how they are linked together, the
relationships between the things. Data Graphs has an intuitive user interface
that lets you quickly design a data structure that reflects your own domain
model.  Each Concept type in your domain has its own schema definition where you
define the properties of a concept and the data types of those properties
(dates, integers, text etc). For example, a simple schema of a Person might have
a givenName (text), familyName (text), and a dateOfBirth (date), and a headShot 
(image).  Rather than explain it in depth here, if you need to know more, we
have made a short video
[https://datagraphs.com/demos/how-to-setup-your-data-sets] that shows how
quickly you can create a schema for your domain model.

Concepts - the data itself. Concepts are the data instances that you load into
Data Graphs conforming to your schemas. You can create and edit concepts via the
user interface, bulk import them from for example CSV files or external sources,
or load them via the Data Graphs UI. Once created, you can search , discover,
and maintain the data via the UI.

SaaSification
Data Graphs is fully SaaSified - you can sign up online [https://datagraphs.com]
, frictionless, and get started immediately with a free trial. You can manage
your own users, roles, your own backups, your own API keys and oAuth
credentials. 

The User Experience
Delivering a great data management experience was our primary goal when creating
Data Graphs. In almost all the knowledge graph and information management
projects we have worked on, data management and curation tooling was always an
area that was a challenge. We set out to build something extremely intuitive to
use, with a UI that would be self-aware of the data types, the relationship
semantics, and make it super easy for users to explore, find, and navigate data.

Dog Food
We are great fans of the “Eat your own dog food” approach to product
development. If our APIs are not good enough for ourselves to build an
application upon, how can we expect them to be good enough for others? With this
in mind, all the features in the Data Graphs Application are built upon the same
APIs exposed to the Data Graphs client, and with that in mind, let’s talk about
integration..

Integration
There is no point having a fabulous user experience for ingesting and managing
your data if you cannot integrate it. As with most SaaS products Data Graphs
comes with an API. A set of REST endpoints for searching, filtering, querying,
faceting, retrieving and creating data. It is fast, it is flexible.

With the API you can access data only in specified datasets, or you can federate
a query across all your data.

If used with only your API key, you access only the data in public dataset. If
used with your oAuth credentials, you can access both public or private
datasets.

Web Hooks - Triggered
The API gives us pull integration, web hooks provide us with push. You can
configure web hooks for Concept create, update, delete, and also candidate
(concept) approval. You define optional endpoints (typically in your  own
platform) that are invoked whenever a specified Data Graphs event is triggered.
For example you could trigger some downstream system when a Concept of a
specific type is created or updated. For example if an Organization concept is
updated, trigger some workflow in an existing business system. The web-hook
events supply the entire concept data in its payload, which means you could use
this to push data into another database or application.

Querying data - The Data Graphs Filter Expression Language
As well as features such as sort and facet (by concept property) the API lets
you query your data using a flexible filter expression language (often called
NQL).  In simple terms, this lets us filter data like this 
filter=property1:value1,property2:value2

A filter expression is a string which provides a (Concept)  property, operator
and value in the form where property is a JSON path representing the field to
filter on (using JSON dot notation) and operator is optional , and tells the API
how to compare values. value is the value to match against.

Operators can be -(not), >, >=,<<=, and also a test if a value is in a group of
values [value,value,...]

Boolean expressions are also supported including parentheses for grouping
expressions together, and overriding operator precedence. 

Full text search is supported using the traditional q parameter. We index all
the text fields in concepts, and boost the canonical labels.

Combining all these together lets us build extremely high utility data queries,
and very flexible integrations.

You can query within a specified dataset, or you can federate across all your
data, and using the filter expression language it means you can ask extremely
useful questions of your data.

The API is fast - you can expect all requests to be returned in less than a few
hundred milliseconds.

Geospatial
We also support geospatial data. We have primitive datatypes for latitude, 
longitude and geohash - so both our API and UI can be geospatially aware for
concepts with location information. As well as being able to explore geospatial
data in the UI on maps, you can also query geospatially using point-radius,
bounding-box, and bounding -polygon queries. Any geospatial query can be
combined with other API filters and full text search.

This means you can ask questions of your data such as “Find me all the concepts
within 50km of New York, classified by X, with property P greater than Y, and
the word “hello” in their label”. Nice.

For example, given a geospatial data set of Airports with relationship to
airlines , get all airports within a 50km radius of London, serviced by EasyJet,
we might use an API request like :

/airports?filter=_near((51.51, -0.12), 50km),airlines:urn:demo:airlines:easyjet

Client Library
For developers we have published a Javascript client library for Data Graphs to
make integration even easier. The client library is particularly useful for
consuming the API using OpenID credentials, as the machine-machine OpenID
authorization flow is all taken care of.  The client library documentation is
published in NPM here : 
https://www.npmjs.com/package/@datalanguage/datagraphs-client

Sub-classing, Inheritance, and Transitive Inference
While Data Graphs is not a full blown graph database, it is a fully managed
structured data solution that can be used to build a high-utility highly
performant knowledge graph for your organization without having to manage any
infrastructure.  There are some key features in knowledge graphs that add
significant value. 

Subclassing and class inheritance is one of those features. In our (extensive)
experience of working with graph databases, inference is a feature that is
widely misunderstood. The set of potential inference rules available in the full
RDF and OWL specification is considerable. However in most practical
implementations of knowledge graphs (outside of academia), there are just one or
two core inferences that add significant value in most business applications.
Subclassing and transitivity is top of that list. Data Graphs has full support
for this.

When you create a new Concept schema definition, you have the option of
declaring a parent type (class) and inheriting its properties. 

Create a Concept with a parent typeOnce you have created your concept schemas,
we have a neat class structure management tool like this :

Concept relationships (and the User Experience) then become class structure
aware, with transitive inference.  When a concept property defines a
relationship to another concept type/class the UI allows you to select concept
instances of that type and all its subclasses.

Transitive relationship propertiesAPI queries also obey the usual semantics of
transitive inference across a class hierarchy. 

Linked Data and Graph Databases
Data Graphs has support for JSON-LD. API responses can be requested with an 
application/ld+json mime-type such that the data is returned in JSON-LD with a
context dynamically constructed from your schema.

This lets you transform data directly to RDF for loading into your own graph
database, or another RDF compliant system. 

So while functioning capably as a standlone fully managed knowledge graph, Data
Graphs could thus also be used as a tool-suite for editing data in a downstream
enterprise graph database / triple store. Using the web-hooks, it is fairly
trivial to build a service that updates entities in a graph database when the
web-hook is triggered from a concept create/update operation in Data Graphs.

The Road Ahead
Data Graphs already has a plethora of great features
[https://datagraphs.com/features], but we have some great new ones lined up in
our road map. Not least is data and model visualisation.

Building on the class structure visualisation we plan to render detailed domain
model / ontology diagrams dynamically from your schema. We also plan a dynamic,
navigable graph visualisation of the instance data itself. On the back-end, we
plan to offer regional data storage options.

We think we have produced something really special with Data Graphs, something
that really delivers utility and value for information management in your
organisation - but don’t take my word for it, go ahead try it out for free
[https://datagraphs.com].

Data Graphs - Fully Managed Knowledge Graphs

BOTT is a data ethics framework for the practical assessment of AI and predictive analytics models within their ecosystems; Bias, Outcome, Target, Trust

BOTT Framework for Practical AI Ethics - Bias, Outcome, Target, Trust

As a data company, data ethics has always been important to us, but in the last
few years, as Machine learning and AI have become front and centre as a
significant part of digital transformation in the enterprise, data ethics has
become a hot topic. It bears such a considerable weight in predictive analytics,
not just in conveying trust to consumers of machine intelligence, but also in
ensuring predictive models deliver genuine value, make unbiased predictions, and
reflect the true state of the environment within which they live. When data
ethics go wrong, it can go badly wrong, potentially bringing down an entire
organisation, with substantial direct and indirect collateral damage,
potentially even on national democracy and social cohesion - the Cambridge
Analytica incident a case in point.

We all need to take it seriously.

In 2018 Data Language was privileged to have been accepted onto the Digital
Catapult “Machine Intelligence Garage
[https://www.digicatapult.org.uk/for-startups/acceleration-programmes/machine-intelligence-garage]
” programme. As well as supporting us with a terrific programme including
compute resource, strategic help and industry engagement, Digital Catapult also
took Data Language through their ethics programme. This was like a cold-shower,
really waking us up to depth and breadth of data ethics. A realisation that data
ethics is not just about data (technically), but about the way you approach your
entire business. 

With all this in mind, we were motivated to develop a data ethics framework that
we could use internally at Data Language to assess all our predictive analytics
projects, and that could guide us practically during implementation. We call
this framework BOTT.

The BOTT Framework
BOTT (all good frameworks need a memorable acronym) stands for Bias, Outcome,
Target and Trust. Through these four pillars, it allows our data science team to
examine, and assess a predictive analytics model within its local and external
ecosystem for quality, bias, and accuracy, and ensure it meets high standards of
data ethics.

Bias
An analysis of potential bias that may be baked into the model during its
life-cycle, either literally through data or code, or intangibly through for
example, some conscious or subconscious human bias. We examine bias from the
following perspectives :

Training data bias - is the training data diverse enough and comprehensive
enough to accurately reflect the environment of the predictive model. What
potential biases might skew predictive results either for good or bad outcomes -
is there any chance these biases might exist in the training data we have used.

Algorithmic bias - is the model algorithm, its mathematical basis, or code
potentially biased?

Real-time input data - if real-time (predict time) input data is involved in
making some prediction, what might these data biases be, and how can we prevent
them from impacting outputs.

Team bias - has or could conscious or unconscious bias of the model development
team contribute to model bias, and have we mitigated against it. Often just an
open and frank team discussion around bias (with respect to the project under
consideration) is enough to flush out these sort of issues

Political, environmental, and social bias - similar to the team bias but looking
at potential external factors through a similar lens - for both ideally the team
needs to be very comfortable talking about potentially politically or socially
sensitive topics in an open, friendly and non-defensive manner.

Output sampling - essentially QA - have you done effective output sampling and
for quality assurance - does the risk of bias in the predictive model call for
independent third party reviews in QA workflows, etc

Outcome
An analysis and survey of the full range of possible outcomes of the model:

Catalogue - construct a catalogue of the potential outcome scenarios

Impact - What does success and failure of the model mean :

 * in the context of the target use-cases
 * best and worst case scenario modelling
 * model failure consequences, e.g. could somebody die if the model fails ?

Target
Analysis of who or what the model is targeted at and the context in which these
users operate.

Users -  Who are target users of the model  - who or what is the direct consumer
of the model ?

Purpose - What problem is the model solving?

Context: Is it part of an ensemble of models, is there a wider predictive
ecosystem the model sits within - what impact does this model have within this
context ?

Life span -  What is the life expectancy of the model, and how does its
environment change over time

Subversion - Can other users or consumers subvert the model for
unintended/undesirable ends?

Catalogue - Construct  a catalogue of potential unintended uses/consequences,
and ascertain if those vulnerabilities can be designed out of the architecture?

Trust
Providing some visibility of the "behind the scenes" implementation can raise
confidence in the models and foster a trust relationship between consumers and
predictive services.

Explainable AI (XAI) is becoming a field in its own right, and an important part
of machine learning and predictive analytics. Codable frameworks are emerging to
aid explainability (for example Google Model Cards
[https://modelcards.withgoogle.com/about]). This codability will be key in being
able to compare and assess models through some normative evaluation mechanism.

Transparency - what should we be exposing of the lineage and decision making
pipeline of the model prediction algorithm :

 * training cycles / training samples
 * outputs of intermediate predictions
 * decision inputs/outputs

Impact - Why should these be exposed ? - perform an impact analysis of the
possible effects of not exposing model lineage, with respect to legal,
environmental, and model improvement consequences.

Data Ethics

The BOTT Framework for Practical Data Ethics

A few weeks ago Cochrane launched their COVID-19 Study Register and I described
the data and technical architecture in detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
.

I described how PICO graphs [https://data.cochrane.org/ontologies/] are used to
augment the Studies with structured linked data that describe the Population,
Intervention, Comparator and Outcome of the trial. Cochrane have now started in
earnest annotating PICO graphs onto the Studies, and these are now being
surfaced in the Study Register application [https://covid-19.cochrane.org/].

Here, we can see the labels of the PICO concepts rendered in the UI, but in the
data itself, these are arranged as a PICO graph structure conforming to the 
PICO
ontology [https://data.cochrane.org/ontologies/pico/index-en.html].

We have also added faceted search by PICO concept into the sidebar. This lets
users quickly filter down their search results using the Cochrane vocabulary
terms.

Once we have a critical mass of PICOs annotated in the register, each vocabulary
concept will be searchable in the search bar at the top of the app, extending
the previously full-text search into a controlled-vocabulary search. A user can
now find studies using a combination of terms chosen from the Cochrane linked
data vocabulary, to deliver extremely accurate results. Moreover we can combine
a metadata driven search with full-text where needed.

Schema.org
Each Study page rendered in the application is also marked up with schema.org
[https://schema.org]structured data. So while internally we are describing our
content and PICO data with the Cochrane bounded context ontologies, at render
time, we are transforming our model to the schema.org MedicalEntity model to
provide machine readable content for search engines and other content consumers.
You can read more about the schema.org medical schema here.
[https://schema.org/docs/meddocs.html]

As an example, for this Studyhttps://covid-19.cochrane.org/studies/crs-13266121 
when viewing the source of the page, you can see we are marking up the clinical
trial as a MedicalStudy in the schema.org ontology, and identifying the 
healthCondition terms, and the interventions, including the Drug Chloroquine.
Each term has associated medical codes mapped to bioinformatics taxonomies
including SNOMED-CT, MeSH, RxNorm and the ATC codes for Drugs where available.
The resulting schema.org JSON-LD for this study is :

{
  "@context": "http://schema.org",
  "@graph": [
    {
      "@type": "MedicalStudy",
      "id": "http://data.cochrane.org/studies/crs-13266121",
      "name": "NCT04333628",
      "identifier": "NCT04333628",
      "description": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
      "mainEntityOfPage": {
        "@type": "WebPage",
        "url": "https://covid-19.cochrane.org/studies/crs-13266121",
        "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
        "description": "Clinical study : Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial"
      },
      "healthCondition": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        }
      ],
      "studySubject": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        },
        {
          "type": "MedicalProcedure",
          "id": "http://data.cochrane.org/concepts/r4hp5z141f1t",
          "url": "https://data.cochrane.org/concepts/r4hp5z141f1t",
          "name": "Usual Care",
          "alternateName": [
            "Routine Prenatal Care",
            "Routine Care",
            "Standard Care",
            "Providing care according to standard",
            "SOC"
          ],
          "code": [
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "372921003"
            }
          ]
        },
        {
          "type": "Drug",
          "id": "http://data.cochrane.org/concepts/r4hp13k8jxft",
          "url": "https://data.cochrane.org/concepts/r4hp13k8jxft",
          "name": "Chloroquine",
          "alternateName": [
            "Chloroquine",
            "Product containing chloroquine"
          ],
          "code": [
            {
              "codingSystem": "MeSH",
              "codeValue": "D002738"
            },
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "14728000"
            },
            {
              "codingSystem": "RxNorm",
              "codeValue": "2393"
            },
            {
              "codingSystem": "ATC",
              "codeValue": "P01BA01"
            }
          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "name": "Pharmacological Interventions"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "name": "Other"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        }
      ],
      "subjectOf": [
        {
          "@type": "MedicalScholarlyArticle",
          "datePublished": "2020-01-01",
          "url": "https://clinicaltrials.gov/show/NCT04333628",
          "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19",
          "abstract": "19 COVID is a deadly viral disease that has been spreading around the world for several months, and is caused by a CORONA family virus (COVID-19). Following IN-VITRO evidence of the antiviral effect of CHLOROQUINE in CORONA viruses, this drug has been used empirically for COVID-19 patients and is currently recommended in Israel for the treatment of intermediate and severity disease. The mechanism of action of chloroquine is in part by inhibiting the virus distribution, and changing the intracellular acidity, the virus distribution site. The intracellular chloroquine concentration is determined by a pump called PGP that removes the drug from the cell and is activated by the drug. In the treatment of malaria, the benefit of low dosage of the drug has been shown to be effective due to the fact that the intracellular concentration of the drug is probably higher, and therefore the logic to examine this issue in COVID-19 treatment. The purpose of this study is to test whether a low dose of Chloroquine will reduce the duration of the viral shedding and prevent the disease from worsening",
          "author": [
            {
              "type": "Person",
              "name": "HaEmek Medical Center"
            }            
          ]
        }
      ]
    }
  ]
}

PICO and Schema.org
The PICO model and process (https://en.wikipedia.org/wiki/PICO_process) is now
widely adopted for framing research questions in evidence based healthcare with
the BMJ citing this model as best practice
[https://bestpractice.bmj.com/info/toolkit/learn-ebm/how-to-clarify-a-clinical-question/]
.

PICO data is applied by clinicaltrials.gov in a semi-structured tabular format
on most of their published trials and PubMed also has a PICO search tool for
querying MEDLINE, the  database from the US National Library of Medicine.

Supporting PICO would be a welcome addition to the schema.org model, allowing
any publisher of medical literature to mark up their content in a standard way
with machine readable structured PICO data as Cochrane and others are now. I can
imagine a future where Google indexes the schema.org PICO markup and provides
the ultimate PICO based search tool across all the medical literature citations
at their disposal.

At Data Language, as co-authors of the PICO Ontology, we would be more than
happy to help define some simple extensions to the schema.org MedicalEntity
model to make this happen.

Data Language is an industry leader in Information Architecture, ontology design and practical knowledge graph implementation. We use Domain Driven Design practises and user-centred design to focus information and data design on core business needs.

Find Information Architecture news and product innovation updates here, including ontology design, vocabulary management, domain & data modelling and more.

Information Architecture - News, Product Innovation and Technology

Information Architecture

PICO, Cochrane and Schema.org

Our global expertise in knowledge graph engineering are helping Cochrane respond to COVID-19. We are using linked data and semantic technologies in this mission.

Knowledge Graph Technology vs. COVID-19 | Cochrane & Data Language

How knowledge graph technology is helping Cochrane respond to COVID-19

A discussion on the synergies between GraphQL and Graph Databases, data architecture and how ontology models provide the special ingredients.

The synergies between GraphQL and Graph Databases

5 latin characters
Until recently, my understanding was that apart from 5 latin characters GraphQL
has got little or nothing to do with Graph Databases. But things are changing,
Graph Database vendors have started bringing these two technologies together. A
recent project has given me some insights.

For some background, GraphQL is a contemporary API interface standard used as an
alternative to REST. GraphQL was originally developed by Facebook. It provides a
structured query language for querying (and mutating) data, along with a runtime
implementation to invoke those queries upon your data. In pretty much all
implementations it uses JSON as its serialization format.

REST (I suspect most readers know this) is a lightweight low-level API standard
for performing uniform and a predefined set of stateless operations on data
aligned with the HTTP verbs. With REST you have to design and  build your API
interface contract for that set of operations - and while of course there are
common patterns in REST APIs, this is very much arbitrary. REST makes no
assumptions about the high level API contract or the underlying data model. The
developer defines both. With GraphQL you only define the data model (the GraphQL
schema), exposing to the GraphQL runtime and your consumers what the shape of
the data is.

With a GraphQL query you can request only the minimum data required to fulfil
your current need. Of course you can do this with a REST API too - in fact it is
a common pattern with JSON REST APIs to filter response data by JSON fields
using dot notation. But of course you need to build this yourself.

GraphQL does a lot more for you to boot, as well as out of the box query
patterns for mutating data, it also supports subscriptions (pub/sub style) - so
your GraphQL client can subscribe to server side events, and have data pushed to
it.

The developer ecosystem around GraphQL is superb now - The Apollo GraphQL
[https://www.apollographql.com] implementation works beautifully with React -
integrating with React Hooks for query and subscriptions. Apollo also has
in-browser developer tools akin to the redux-devtools for GraphQL debugging, and
you can manage local app state via Apollo using the GraphQL schemas / queries
etc.

Caching and performance
Requests to GraphQL are typically all POST requests - not great for caching you
ask ? Indeed, this is where REST has excelled for years. But the GraphQL server
implementations have come to the rescue. The Apollo GraphQL server has an
internal caching layer that supports full response caching, and fine grained
field-level caching (read more here
[https://www.apollographql.com/docs/apollo-server/performance/caching/] )

Amazon’s AWS AppSync is a fully managed GraphQL service built upon the Apollo
technologies - and can source data out of the box from other services such as
DynamoDB, ElasticSearch, REST APIs etc. With fully managed caching and real-time
data subscriptions, it really does make your developer life easy.

On the surface it still appears GraphQL has little to do with Graph Databases.
Yet increasingly we are hearing about how they should go together like bread and
butter, sausage and mash.

Not all graph databases are equal
I won't go into depth about what a Graph Database is, but am going to explore
some of the options available, and the relationship between Graph Databases and
GraphQL.

Graph databases come in several flavour combinations. First up you have RDF
graphs vs Labelled-Property graphs. The first big choice. Both are very
different [https://en.wikipedia.org/wiki/Graph_database#Graph_models]. Your
primary use case is important here. Essentially if you primarily need to walk,
query and analyse paths through your graph, then Labelled-Property graphs are
probably the way to go. If you are more interested in the nature of the
relationships between things, and creating rich semantic representations of
things, then RDF graphs hit the spot. Both types have their own well-developed
query language. Labelled-Property graphs are typically queried with Cypher or
Gremlin (eg Neo4J), RDF graphs are queried with SPARQL.

While most mainstream RDF based graph databases are fully compliant to the RDF
and SPARQL W3C standards, not all RDF graph databases are equal. Each has its
own implementation nuances and come with their own added extras (decisions,
decisions!):

 * An RDF graph database may or may not support reasoning (to some degree of the
   RDFS and OWL standards).
 * Reasoning may be at query time (backward chaining) or at write-time
   (forward-chaining).
 * The way reasoning is implemented can dramatically impact performance.
 * Add-ons and extensions for GeoSPARQL, Path analysis, Full text search,
   connections to other datastores etc.

To give some examples (DISCLAIMER - apols to those vendors not mentioned, I am
only referencing those I have worked with properly hands-on, and this is not an
exhaustive features list - it’s just a flavour) :

GraphDB [https://www.ontotext.com/products/graphdb/] (Ontotext)

 * Fully compliant with RDF/SPARQL standard;
 * Highly compliant and configurable OWL / RDFS forward-chaining reasoning
   (materialises inferred knowledge at write time);
 * Unparallelled full-text-search capability using lucene, or via bi-drectional
   connectors to external SOLR or ElasticSearch clusters.
 *  Ability to materialise post-inference fully structured JSON or JSON-LD
   objects into ElasticSearch indices or MongoDB
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI
 * UPDATED May 2022 - Path query capability is now avaialble using a SPARQL
   extension plugin

Neptune [https://aws.amazon.com/neptune/] (Amazon AWS)

 * Fully compliant with RDF/SPARQL standards;
 * No reasoning capability
 * Fully managed cloud service
 * Highly, highly horizontally scalable (primarily due to lack of reasoning)
 * Full text search capability via connector to ElasticSearch - materialisation
   into elastic is constrained (you cannot materialise user-modelled JSON
   objects into an elastic index), but delivers good full-text search utility.
 * You use this either as a Labelled-Property graph (using Gremlin) or an RDF
   graph (with SPARQL)
 * No query workbench/admin UI - you need to install your own 

Stardog [https://stardog.com]

 * Fully compliant with RDF/SPARQL standards;
 * Highly compliant and configurable OWL / RDFS backward-chaining reasoning
   (materialises inferred knowledge at query time);
 * Extremely fast online write times (due to backward-chaining model)
 * Adequate full-text-search capability using internal indexing
 * An outstanding SPARQL extension for PATH analysis (Pathfinder) - you can
   query for paths, shortest paths with weighted edges, normally only possible
   with labelled-property graphs
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

So you can see that each of these three are fully W3C compliant RDF / SPARQL
graph databases. But each has one or more killer features, that the others do
not. 

GraphDB - the post-inference connectors to ElasticSearch and MongoDB unlock some
outstanding technical and data architecture patterns out of the box -UPDATED May
2022 - GraphDB now also has comprehensive Path query capability using sparql
extension plugin

Stardog - combines the semantics of RDF with sophisticated Path querying and
analysis.

Neptune - fully managed cloud service with unparalleled horizontal scalability,
but no reasoning.

The upshot - choose wisely based on your requirements now, and what is likely to
matter most in the medium term.

..and then enter stage right, GraphQL

Sausage and Mash or Pineapple on Pizza ?
How do GraphQL and Graph Databases fit together? As we learned earlier, GraphQL
is a REST API alternative with a structured query language combined with a
runtime for query processing and serving data to GraphQL clients (and it is
almost always implemented with JSON in mind).

So why are Graph Database vendors getting involved and what do you get for your
money?

The cynical marketer in me says those 5 latin characters definitely count for
something. Which sales manager wouldn’t love the potential energy pressure in
the ready-to-explode singularity of Graph Databases and GraphQL (while humming
‘Don’t stop me now’).

However, GraphQL does only provide another query mechanism for your graph
database alongside SPARQL or Gremlin.  Moreover it is much less expressive, and
most definitely not designed for exploiting the rich semantics of RDF and OWL. 

In fact if you go all in on a michelin-starred Graph Database, and only use
GraphQL to query it, I would argue you really probably don’t need a Graph
Database - a NOSQL JSON store would probably suffice. Of course, if you have
forward-chaining inference in your graph database (as with GraphDB), then your
GraphQL queries will benefit from the RDFS/OWL semantics you have modelled your
data with, which does add weight to the value-proposition.

Still hungry ?
Ontologies are an important part of the picture. In the RDF world an ontology
semantically describes the model and nature of your domain, and ultimately the
data structures stored in your knowledge graphs, and if inference-aware, they
describe the meaning of the data, and how we reason upon it. Typically these
models are communicated up and down our tech stack (and the responsible
developers), and often all the way out to consumers of your APIs (public or
enterprise-internal).  While we typically move data around using JSON the
linked-data aficionados will know well that if we do this using JSON-LD, we can
use the same ontology models from the knowledge graph (or a subset) for
describing the JSON-LD payloads we ship around outside of our graph database.

So do we need GraphQL to take advantage of these ontological graph models ?.  No
we don’t, we can do it with normal REST APIs (querying our graph with SPARQL)
and using JSON-LD payloads - but of course we have to build more API code to do
this.

With GraphQL though we can (we must) formally describe our data structures as
GraphQL schemas, and so we have a great opportunity here to align our GraphQL
schemas with those same ontology models that describe the data in our knowledge
graph -  this could be either a subset of our ontology landscape, or possibly
the superset, if we are using other GraphQL (served) data sources aside from the
data in the graph database. This is starting to look like very nice decoupled
data architecture with well structured cohesive data models, with possibly the
most efficient mechanism for consuming data in our web/mobile applications (a
GraphQL client).

It is looking more tasty
Adding GraphQL client interfaces to a Graph Database does make life easier - if
you are rapidly prototyping straight from the graph, or federating data into an
external GraphQL server there is a nice pattern for making a common set of well
described data models.

The Graph Database vendors have indeed jumped on-board the GraphQL sushi train
(and it’s not all about those 5 latin characters). Stardog and GraphDB (from
Ontotext) both offer GraphQL clients now.

At this point I was going to show an example of how GraphQL might be used to
query a Graph Database, but Jem Rayfield at Ontotext has already done the job in
his superb Star Wars blog post
[https://www.ontotext.com/blog/the-rise-of-the-knowledge-graph/] (if you have
got this far, then this is a must read).

Ontotext has taken the next logical step and created a ‘Semantic Object Service’
 that automatically generates GraphQL schemas bound to the underlying RDF data
model  in GraphDB, moving us towards the ontology ubiquity I described above,
where your domain ontologies are expressed in the knowledge graph, and across
your wider data-estate, including your GraphQL schemas. Nice work.

Jem concludes by saying :

> GraphQL is a very popular graph query language, well used, known and liked by
developers. The Ontotext Platform uses GraphQL to lower the barrier of entry to
knowledge graph data, whilst still providing the full expressivity and power of
SPARQL.
and this is a key message, which equally applies to Stardog and any other graph
database that may yet offer a GraphQL client. While GraphQL lowers the barrier
to entry, GraphQL gives you only as much as it is able to with respect to
querying and graph traversal;  to really unlock the power of the graph database,
you will need to use SPARQL.

But if you never plan to, and you are not interested in reasoning either, then
possibly you should ask yourself if you really need a graph database ?

What if you want to use GraphQL as the primary API technology in your stack, but
want to use SPARQL too to access the graph database (amongst other tech)?

A mixed data technology case study
In a very recent engagement, the Data Language team have been working on a Smart
Cities IoT project. The nature of the domain meant the type of data we needed to
deal with was very ‘graphy’ in nature, and the ability to describe our data
semantically and traverse a graph with SPARQL was of high utility. Inference was
less of a concern. The proof of concept for the project was brought together in
AWS with :

 * AWS MSK - a high performance Kafka message bus to consume, orchestrate, and
   distribute IoT sensor events
 * AWS Neptune to give us our scalable RDF based smart city knowledge graph
 * A set of JSON-LD REST API endpoints invoking SPARQL queries on the graph
   running in Kubernetes (EKS)
 * A suite of Kafka event processor consumers and producers deployed into
   Kubernetes mutating the graph, and emitting new events based on our business
   logic rules.
 * AWS AppSync - the GraphQL engine sourcing events and the JSON-LD REST APIs
   from Neptune, providing the primary developer interface to our client
   applications
 * A ReactJS built UI using GraphQL react-hooks to subscribe to real time events
   data, and to query data from the sensors and our knowledge graph (ultimately
   via expressive SPARQL queries).

A simplified view of the architecture looks essentially like this :


The kicker was modeling data across the entire stack using a suite of
ontologies. These ontology models describe our knowledge graph data, our IoT
message payloads in JSON-LD and were all federated in the primary application
API expressed as GraphQL schemas in AppSync. Building the ReactJS front end
using the Apollo GraphQL client and react-hooks was a very nice developer
experience.

Dessert
While GraphQL technically has little to do with Graph Databases, they are not a
perfect match like sausages and mash, GraphQL is just another API / query
technology, and not as expressive as the native graph database query languages
like SPARQL, but in itself is a high-utlity, very sweet technology.

There are some clear synergies, and do provide some of that warm comfort food
feeling when used together. GraphQL does give you an easily understood and very
developer friendly way into a graph database. Once you add the secret sauce of
domain ontologies describing both your knowledge graph and possibly even your
entire data-estate, expressed through your GraphQL schemas, then you have a very
nice pattern for a decoupled data architecture with well structured, nicely
communicated data models across your technology stack, and out to your consumers
too - making for efficient application development using the GraphQL ecosystem.
Get stuck in.

GraphQL and Graph Databases

The problem: flaky connections cause upload of large files to S3 to fail
Uploading files to S3 securely direct from a browser is a common pattern. There
is plenty of information available on how to do this. Typically most examples
documented online use standard S3 signed POST requests (either from direct from
a html form or via an XHR request). This often means using some server side API
call (which you need to build) to generate an AWS signed request, which can then
be used in your web app client side. While being fairly easy to implement, for
very large files a flaky connection or a dropped packet often means a failed
upload and retries. What you need is multipart, chunked uploads that are
resilient. However this is not trivial to achieve with S3.

The solution: secure chunked multipart managed uploads
If you use the AWS SDK there are essentially two routes. Use the S3 REST API and
manage file chunks yourself. This is the low-level approach and is complex.
Alternatively you can use the AWS SDK Managed Uploads API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property]
, in which the SDK does all the hard work for you to manage file chunking,
upload, resilience, and the reconstruction of the file parts into a single S3
file at the end. However to use this you have to supply AWS credentials, and
this is not something you typically want to expose client side. However, if your
upload feature is behind some oAuth or OpenID authentication layer (as in my
case, and I suspect is often the case if you are allowing uploads to your
backend) then you are in business. You can setup AWS to work as OpenID Connect
provider. Finally you need to integrate this with some client side file uploader
library, such as Dropzone [http://www.dropzonejs.com/]. I am going to
demonstrate step-by-step how to :

a. Setup an S3 bucket to receive the uploads with a suitable access policy and
CORS settings.
b. Setup AWS as an OpenID Connect provider for some external oAuth service - in
this case Auth0 [https://auth0.com].
c. Integrate AWS S3 Managed Uploads securely with Dropzone to handle multiple
parallel resilient uploads for very large files, with progress bars, and cancel.

Step 1 - Create an S3 bucket
In your AWS tenancy, create an S3 bucket (e.g. my-bucket) and a CORS policy as
follows.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-meta-custom-header</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>Authorization</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>


You can revoke all public access to the bucket, as we will be writing to the
bucket using an OpenID Connect user that has assumed an IAM Role with the
correct policy to access the bucket.

Step 2 - Create an OpenID Connector Provider in AWS for Auth0
This allows us to use AWS as an intermediary for validating JWT OpenID tokens
against an oAuth service, demonstrated here with Auth0.
Your web app users will authenticate (login) with Auth0 - The oAuth response
from Auth0 authorize endpoint delivers an OpenID JWT token to the browser, which
you typically store in localStorage for securing subsequent requests from the
client to your backend.
To setup the AWS OpenID Connect provider you need the oAuth service root
endpoint, and the oAuth application clientID. In Auth0 login to your dashboard
and in your application settings you can get this info.

The oAuth endpoint is : https://{your-auth0-domain}/ (note the trailing slash),
e.g. https://appname.auth0.com/

The application Client ID is a 32 character Base62 encoded string, e.g. 
6sR4LF3UpznU4T26S3xRheGhjEKdni4H

Now login to your AWS tenancy, navigate to IAM account management > Identity
Providers. Click on Create Provider, and enter the Auth0 details from above as
follows, ensuring you add the trailing slash on the provider URL:

Then click Next and Create the provider.

Step 3 - Create an IAM Role for the OpenID Provider
Navigate to Policies in IAM, and create a new Policy as follows using the JSON
editor, setting the correct name for {my-bucket}:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "*"
            ],
            "Resource": [
                "arn:aws:s3:::{my-bucket}",
                "arn:aws:s3:::{my-bucket}/*"
            ]
        }
    ]
}


Note, this policy will grant the authenticated user assuming the role all
actions on the bucket.. you may want to make this more restrictive. You could
also add other AWS grants here too, so your user can access other AWS services
if need be. Name your policy to something sensible, eg OpenIdS3Policy.

Now navigate to IAM Roles, and create a new Role. Choose Web Identity as below
and choose your OpenID Provider you just created, and the Auth0 Client ID as the
audience in the drop down options :

Next, Set the permissions of the role by choosing the OpenIdS3Policy created
previously. It is easy to find your policy by selecting policy type Customer
managed from the drop down. Save the Role with a suitable name eg OpenIdS3Role.

Step 4 - Grant access to the S3 bucket
Now we need to grant access to the S3 bucket for the IAM Role just created.
Copy the ARN for the Role, and navigate to your bucket setttings, and choose 
Bucket Policy. Add a Policy as follows, using the ARN for the role, and the
correct bucket name :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permissions",
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:GetBucketLocation",
          "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}"
    },
    {
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role"
      },
      "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::{my-bucket}/*"
    }
  ]
}


That is all the AWS work needed. An authenticated OpenID Connect client can now
access your S3 bucket by assuming the role of an IAM user. When the AWS
connection is made, AWS authenticates the supplied JWT token against Auth0 using
the AWS Identity Provider. Thus a user has to be logged into an Auth0 session to
be able to access the bucket. AWS matches the aud value in the decoded JWT token
with the trust relationship condition specified in the role we created. This is
where the all important trailing slash I mentioned comes into play. This is an
artefact of using Auth0 as an OpenID identity service. Auth0 adds a trailing
slash to the aud value in the JWT token it generates. If we dont add the same,
AWS cannot match the role with the token when it tries to assume the identity of
the connecting user. If you use a different oAuth provider than Auth0 you may
not need this trailing slash. Think of it as an Auth0 'gotcha'.

Step 5 - Integrate with Dropzone
Dropzone [http://www.dropzonejs.com] is one of the better, and widely used open
source file upload JS libs. It is very customizable and mostly event driven so
is easy to modify. Out of the box Dropzone uses XHR requests to send files to a
remote server. We are going to override Dropzone's file uploading mechanism to
use the AWS SDK Managed Upload API
[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#send-property] 
instead.

First, install (or make available) the AWS SDK in your web app, e.g.

npm install aws-sdk --save


Now define an accept callback function, that Dropzone invokes when a file has
been accepted for upload. This lets us add an S3.ManagedUpload instance to the
file object being queued by Dropzone for upload :

import S3 from 'aws-sdk/clients/s3';

function acceptCallback(file, done) {
  // options for the managed upload for this accepted file
  // define the bucket, and the S3 key the file will be stored as
  var params = {Bucket: 'my-bucket', Key: file.name, Body: file};
  // add an S3 managed upload instance to the file
  file.s3upload = new S3.ManagedUpload({params: params});
  done();
}


Now we need to override the Dropzone uploadFiles function to upload via the AWS
SDK instead of XHR. Note we are using the file.s3upload instance we added to the
file object when the file was accepted by Dropzone. We emit Dropzone events on
success and failure, and we also listen to the AWS httpUploadProgress event, and
emit the equivalent Dropzone event, so Dropzone can render upload progress for
each file :

// override the uploadFiles function to send via AWS SDK instead of xhr
Dropzone.prototype.uploadFiles = function (files) {
  for (var j = 0; j < files.length; j++) {
    var file = files[j];
    sendFile(file);
  }
};

function sendFile(file) {
  file.s3upload.send(function(err, data) {
    if (err) {
      theDropzone.emit("error", file, err.message);
    } else {
      theDropzone.emit("complete", file);
    }
  });

  // listen to the AWS httpUploadProgress event, and emit an equivalent Dropzone event
  file.s3upload.on('httpUploadProgress', function(progress) {
    if (progress.total) {
      var percent = ((progress.loaded * 100) / progress.total);
      theDropzone.emit('uploadprogress', file, percent, progress.loaded);
    }
  });
}


If we want our users to be able to Cancel uploads mid-progress we need to
implement an abort function that can be invoked on the Dropzone canceled event :

function abortUpload(file) {
  if (file.s3upload) file.s3upload.abort();
}


Finally you need to initialize your Dropzone on some DOM element on your page
and
setup AWS Credentials for the S3 managed uploads. To do this we create an 
AWS.WebIdentityCredentials object using the ARN of the IAM role we created in
Step 3, and also inject the JWT ID token from your local storage/session - the
token returned by the Auth0 client side authentication workflow. When we
initialize Dropzone we also set up the accept and canceled event callbacks.

import AWS from 'aws-sdk/global';
import S3 from 'aws-sdk/clients/s3';

function initializeDropzone(dropzoneElement) {

  var idToken =  'your-jwt-openId-token'; // get from local storage/session
  AWS.config.credentials = new AWS.WebIdentityCredentials({
    RoleArn: 'arn:aws:iam::{my-AWS-Account-ID}:role/OpenIdS3Role',
    WebIdentityToken: idToken
  });

  theDropzone = new Dropzone(dropzoneElement, {
    autoQueue: false, // if you want to manually trigger uploads
    accept: acceptCallback,
    canceled : abortUpload,
    init: function () {
      this.on('removedfile', function (file) {
        abortUpload(file)
      })
    }
  })
};


All done. In these examples, I have shown the minimum you need to do to get S3
Managed Uploads working with, for example, the Boostrap Dropzone demo here
[http://www.dropzonejs.com/bootstrap.html]. There are of course lots more
features of Dropzone you can implement, but this should be sufficient to for you
to build fully resilient secure S3 uploads using Dropzone and OpenID
authentication for very large files.

Blogs by Paul Wilton