Alexi Buckingham

Taken from our newsletter which you can subscribe to here
[https://datalanguage.com/newsletter].

Last week we presented our knowledge graph product, Data Graphs
[https://datagraphs.com/?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=intro] 
at The Knowledge Graph Conference [https://www.knowledgegraph.tech] at Cornell
Tech on Roosevelt Island, New York.

Based on the presentation topics and workshops at the conference, and backed up
by the "2nd Annual Knowledge Graph Survey", here are our 3 takeaway messages for
you:

Takeaway 1: Organisations are struggling to get started
We heard a lot about the challenges of skills gaps, organisational inertia, and
governance gaps, and that when trying to adopt knowledge graphs, many
organisations just don’t know where to start. This is reflected in the 2nd
Knowledge Graph Annual Survey with 86% of responses saying that 'Organisational
Inertia' was an important inhibitor to their adoption of knowledge graphs.

Organisations can break through the inertia by using a reference implementation
Our preferred approach to breaking through this inertia is to build a reference
implementation rapidly. Using Data Graphs, you can have a working reference
implementation ready within a day - we did it in 13 minutes here
[https://datalanguage.com/news/13-mins-to-set-up-a-knowledge-graph-with-netflix-disney-amazonprimevideo-and-hulu]
.

A working reference implementation has 3 main benefits:

 1. It helps stakeholders grasp the value of data meaning and interconnection,
    hands-on.
 2. Seeing the knowledge graph visually, and being able to explore it by simply
    clicking from node to node sparks new ideas for all disciplines.
 3. A working version gives teams a real basis on which to iterate and answer
    key questions, including “What KG features do we need?”, “What model works
    well?”, “What kind of inference and path analysis do we need?”, and more.

Have a look at the Knowledge Graph Explorer
[https://media-graph.datagraphs.io/media-demo/people/concepts/urn:media-demo:Person:3Vn1Bxu4W6yzLNL54eB4b5/explore?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=reference-implementation] 
feature of our product Data Graphs below. A reference implementation like this
helps organisations to break through the inertia, and save time spent wallowing
in indecision.

Knowledge Graph Explorer
[https://media-graph.datagraphs.io/media-demo/people/concepts/urn:media-demo:Person:3Vn1Bxu4W6yzLNL54eB4b5/explore?utm_source=email&utm_medium=newsletter&utm_campaign=internal-inertia]
Click here to try out the Knowledge Graph Explorer
[https://media-graph.datagraphs.io/media-demo/people/concepts/urn:media-demo:Person:3Vn1Bxu4W6yzLNL54eB4b5/explore?utm_source=email&utm_medium=newsletter&utm_campaign=internal-inertia]
Takeaway 2: Organisations are looking for key features to enable their plans
In this slide from the KG Annual Survey, you can see the areas here that reflect
what knowledge graph users are interested in, for example 'Search', 'Better
Ontology Management', 'Integration' and 'No-Code/Low-Code'.

Features of interest to KG Survey Respondents

Search
Data Graphs' Advanced Search feature allows you to build complex queries by
chaining and grouping a series of boolean and numeric expressions.

Try Data Graphs' Advanced Search here
[https://media-graph.datagraphs.io/media-demo/_all/concepts?aq=c[director:Person:2mdyLPMIMopbPB0kVUvghw:Quentin%20Tarantino].(c[cast:Person:3ENPqEkRhLUCxj3W3RG1Hp:Samuel%20L.%20Jackson])&pn=1&ps=15&sf=_label&sd=asc&layout=list&viewport=s000000000,1&utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=search]

Better Ontology Management
This area has always been complex and somewhat expert-dependent. "No code"
schema development is now in demand. In our demo, you can see the domain model
(the output of our no-code custom schema tools) for shows on Netflix, Disney+,
Amazon Prime Video, and Hulu.

Try Data Graphs' Domain Model here
[https://media-graph.datagraphs.io/media-demo/_domain-model?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=ontology-management]

Integration
Using APIs and webhooks, how can we help developers integrate with minimum
friction? This step-by-step guide shows how to set up your knowledge graph
quickly and effectively.
[https://support.datagraphs.io/knowledge-base/get-started/building-a-knowledge-graph?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=integration]

Building a Knowledge Graph: Step by Step Guide
[https://support.datagraphs.io/knowledge-base/get-started/building-a-knowledge-graph?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=integration]

No-Code/Low-Code
How might businesses get their knowledge graph up and running without having to
dedicate engineers to coding? The tools in this space are improving, and you can
see how we set up a knowledge graph with Data Graphs using data from Netflix,
with APIs and fully featured management UI production-ready, in 12 minutes here.

Creating a Knowledge Graph in 12 Minutes
[https://datalanguage.com/blog/creating-a-knowledge-graph-in-minutes&utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=integration]

Takeaway 3: Organisations want to reduce 'Time to Value'
In order to get to market rapidly, and be able to iterate based on real customer
needs, organisations are aiming to get their knowledge graph platforms launched
faster. From inception to commercial operation, how do we do this? Whether it is
for a reference implementation, or for commercial use, tools to enable rapid
model building, data ingest, and ready-to-go APIs are front of mind.

This video shows how a knowledge graph with Netflix, Disney+, Amazon Prime Video
and Hulu data can be built, with APIs and powerful UI, in just 13 minutes.

Watch: 13 minutes to setup the media graph
[https://datalanguage.com/news/13-mins-to-set-up-a-knowledge-graph-with-netflix-disney-amazonprimevideo-and-hulu&utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=time-to-value]

But Remember... Preparation is Key
One thing we consistently emphasise is that good preparation is essential. The
speed and relative simplicity of the setup of Data Graphs is only possible with
solid information preparation - data that was already clearly structured and
well organised. Designing your domain model is an essential step in establishing
the shape of your data before embarking on your first implementation. It will
change, no doubt, but do avoid rushing in without a plan. Spend time thinking
about the specifics of your data, the core entities in your domain, the key
relationships between them, and this will enable you to make an effective start.

More on why good preparation is key
[https://datalanguage.com/blog/creating-a-knowledge-graph-in-minutes?utm_source=website&utm_medium=kgc-newsletter-blog&utm_campaign=preparation]

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

Reflections and Insights from Knowledge Graph Conference 2022

Paul Wilton

Ear of bat, blood of frog, rabbit’s tail. Stir them all up in your graph
database, utter the magic words, and lo, with a sprinkling of sparql dust, gold
speweth forth.

After working with graph databases for 10 years now, it troubles me that this is
still how many implementers expect things to work. Graph databases are not magic
cauldrons. They really are not.

A graph database is no different from any other database. If you put in data
mess, do not expect order to be created from chaos. If you then combine this
with over constrained and over-engineered inference and reasoning you end up
with an even bigger mess. A popcorn machine.

You will not be able to build robust practical business or enterprise grade
applications by treating your graph as some dirty place where ontologists (who?)
 can dip their fingers in, mess around ad-hoc, and make magic happen. “let’s
just add another inference rule, that will sort it..”

Just like any other practical, robust data architecture, data governance is
essential. Just like any other database you would not expect end users to
interface directly with it via a SQL endpoint, or a JDBC connector? No, you
would build robust, well tested APIs and Services to interface with your
database, and let your application builders interface with these services. End
users would then interact with these applications. These API services are the
gatekeepers of good data governance.  No mess should pass this point. In
production, no human hands should *ever* pass this point. 

Any changes to models should only be done to meet some business need or
requirements. Any changes to models should be tested.

Inference and reasoning should be included in your models *only* to meet some
practical value-deriving business need.  If the inference does not deliver any
meaningful or practical value to your end users, it should not be there. 

If you can deliver a business requirement effectively and easily without
inference, do it without inference.

Your models should be as simple as possible to deliver your business
requirements. A SQL data architect would not just add additional random tables
into their SQL model to deliver their requirements, so when you develop a
practical ontology model for your domain, only model what you need to deliver
those requirements. You do not need to model extra stuff. 

While your domain model may extend well beyond the bounds of your business
requirements (in order to better inform requirements and business landscape),
your detailed ontology models should not. By only engineering the very minimum
of the model needed to deliver your current requirements, your platform is
easier to evolve, easier to maintain, and at a much lower total cost of
ownership.

Finally - end users should not have access to your production graph database
sparql or cypher endpoint. Never. 

Stop pretending you are witches. Start thinking like traditional data
architects.

Graph databases

Graph Databases are not Magic Cauldrons

A case study on a recent engagement in the publishing industry using Data Graphs
for centrally managing cross-publication content metadata combined with our Text
AI NLP service for auto-tagging content to provide a high-utility information
backbone.

The Snap
Tagging content has been a publishing staple for two decades, helping publishers
organise content, perform rudimentary content analytics, and also deliver tag
based user journeys and content aggregations. In many cases publishers use the
tag features built into their CMSs. WordPress and its derivatives are so widely
used, yet still have a very weak content tagging capability. This lack of good
information management and structured metadata often leads to a tag mess over
time, as tags cannot be properly disambiguated, duplicates invariably get
created, and journalists apply tags inconsistently. Silver Oliver has written an 
excellent series of articles
[https://datalanguage.com/blog/why-the-web-turned-its-back-on-the-librarians] on
this subject that are worth revisiting.

This is where more contemporary information management technology and AI can be
a game changer. I outline below how we have used our own Text AI and Knowledge
Graph products to solve these issues, and drastically improve how structured
metadata is used at a large global news publisher.

The Back Story
In the context of this project, we were not dealing with a single CMS, but
multiple news publications, each with their own publishing technical stack, CMS,
mechanism for tagging, and entirely separate tag sets. The ambition was to
centralize the metadata used to tag on each publication, and use AI to aid the
journalists, essentially raising the consistency level of tagging across news
desks, and streamlining and optimising the tagging process. Downstream, this
would unlock better user journeys for consumers, better content analytics, and
provide for better and more rapid cross-publication innovation.

To achieve this, we integrated:

 1. Our product Data Graphs to manage a large corpus of the cross-publication
    tags, constructed as a light-weight knowledge graph of news concepts,
    including concept types such as People, Locations, Organisations, Events,
    Brands, Sports teams and athletes, Categories and Topics.
 2. Our AI Text Classification service, Tagmatic - which predicts the
    “aboutness” of content by tagging using the concepts managed in Data Graphs,
    continuously learning on-the-fly as new articles are published.

The Storyline
Our two products Data Graphs and the Text Classification AI work seamlessly
together as an ideal solution for an information backbone in the new generation
of publishing technology ecosystems.

We move from the legacy pattern of tagging with flat tags, to tagging with well
modelled and organised structured data, where each concept we tag with has a
globally unique, persistent and durable identifier (a URI / URN), and has its
own metadata properties that can be used to provide more information about that
concept to help users disambiguate (types, thumbnails, images, descriptions,
categories etc) and also provide better user experiences and journeys with these
concept tags.

Managing this metadata centrally outside of your CMS provides a number of
advantages:

 1. Your tags are not tightly coupled to one CMS (with respect to identity,
    structure, access and availability etc) - giving you an easy upgrade path or
    migration to a different CMS in the future
 2. The ability to easily use that metadata in other parts of your publishing
    stack or share with other CMSs. For example, the same metadata can then be
    used to tag or classify source feeds upstream from content authoring such
    that external content and your own content are tagged using the same
    classifications and concepts.
 3. Decouple the management of the metadata from the CMS, such that a librarian
    or information specialist takes on that task, leaving your journalists to do
    what they do best and author content.

To do this, Data Graphs allows the publisher to:

 1. Create a domain model of their core entities (People, Places, Orgs, Sports,
    Events, Topics, Content Types, and any other metadata they need to drive
    their publishing processes and user experiences)  and how they prefer to
    classify and relate these concepts to each other.
 2. Ensure these concepts are modelled with all the structured data you need for
    concept disambiguation, categorisation, and concept driven user journeys.
 3. Easily populate this model with the instances (taxonomies) of this metadata
    (essentially, publishing reference data) using either the Data Graphs API or
    CSV Uploaders in the Data Graphs web app.
 4. Manage that metadata with a dedicated application, with an intuitive user
    experience.
 5. Integrate the metadata with any of your business systems using its APIs and
    Web Hooks, including your CMSs and other data or content stores.

Using machine intelligence to automatically tag concepts and classifications of
your articles, also provides a number of advantages:

 1. It makes the task of tagging and classifying content easy and faster. Many
    journalists don’t like having to tag their articles, and may not understand
    the value of tagging well. Because the AI learns from all journalists, and
    from golden documents, this means the quality level is raised for all
    authors, and consistency is improved across all news desks.
 2. The AI can be used to tag content outside of the CMS, completely offloading
    this task from the journalists - either using a human-in-the-loop pattern
    (where a totally different person/role validates the quality of the metadata
    the AI is applying), or in a fully automated mode, for say long tail content
    where you may not be so concerned about occasional mistakes being made,
    where the cost of human supervision might exceed the value realised.
 3. The AI can be used to tag content upstream from your CMS, tagging external
    feeds (Reuters, AP, Social feeds etc) with the same metadata you apply to
    your own content. This can deliver much more efficient tooling for the
    processing of this large content funnel on your news desks.

The Data Language Text Classification AI (Tagmatic)  is designed for exactly
this task. It is a high-performance scalable service that learns how you
classify and tag your content, and then applies metadata to new content that is
sent to it for prediction. The service is highly scalable, fully private, and
designed to adapt to each publisher’s unique tagging style. Its REST API allows
for very easy integration with your CMS and publishing stack, such that it
learns continuously on-the-fly, reacting quickly to new concepts that arise in
the ever changing window of news.

Data Graphs and Tagmatic work seamlessly together, through a dedicated
connector, such that when a concept is merged or removed in Data Graphs,
Tagmatic automatically adjusts its models’ weights to reflect the new state.

The Explainer
Technically Data Graphs and our Auto-tagging Text AI work together as shown in
the diagram below. In this case study a custom WordPress metadata plugin was
created that integrated with both the Data Graphs and Tagmatic APIs to provide a
human-in-the-loop tagging UI directly in the publisher’s CMS:

Where:

 1. The article is sent to the Tagmatic Predict service API to suggest concepts
    to tag with, and these are presented to the user in the WordPress plugin.
    The classes of the concepts are used to present these in a tagging template
    suitable for the type of content being authored (News, Sport, Politics etc).
    The user can then keep or remove the tags that seem incorrect.
 2. A user can lookup missing or new concepts in Data Graphs via
    typeahead/autocomplete widgets in the WordPress plugin by using the Data
    Graphs Search API
 3. When the article and metadata is saved / published, WordPress sends the
    content and tags back to Tagmatic to its Train API, so it can use this for
    continuous improvement of its prediction models.  This allows it to learn
    brand new tags from their first appearance in  an article precluding the
    need for scheduled long running training cycles, and instead learns
    on-the-fly.
 4. If the librarian or information specialist removes a tag concept in Data
    Graphs, it automatically fires a webhook to the Tagmatic connector, so
    Tagmatic can disable the classifier model for that concept. If a user merges
    two concepts in Data Graphs, the connector will trigger Tagmatic to re-point
    its classifier from the source concept to the target concept of the merge
    operation - and adjust its weights accordingly.

In the more detailed images below you can see how this WordPress metadata plugin
might look, and how a concept is presented to the user with enough information
from Data Graphs to correctly disambiguate the tag.

Once tags are sent back to the training API with the content, Tagmatic makes
tagging and classification metrics available via another service API that
includes the precision, recall, F1 scores, support for each tag, and also the
date the tag was last trained. These metrics can then be used to provide
insights including usage analytics and patterns to the editorial floor.

The Finished Article
Bringing these technologies together creates a high-utility information backbone
for the modern newsroom, streamlining processes, automating others, and
unlocking publishing innovation.

By centralising and decoupling metadata from your CMS, and introducing machine
intelligence, you unlock a number of key capabilities: 

 * Better information and data governance, the importance of which cannot be
   understated
 * The ability to cross-pollinate your reference data across third party content
   and other internal systems, including both content and user analytics.
 * The ability to automate the classification of upstream content feeds with the
   same metadata, which allows you to scale out the funnel of source material
   coming into the news desks, and provide better search and discovery
   capabilities to those journalists.
 * Making the journalists job easier - letting them focus more on authoring good
   content
 * The ability to automate the aggregation of content, for example for topic
   pages, using structured through metadata driven content query APIs
 * Let the CMS do what it does best: the authoring and structuring of the
   content itself

Ultimately, this technology makes tags and concepts management a first class
citizen in the publishing stack. No longer just an afterthought inherited from
and tightly coupled with the CMS of the day - but something that is built to
stay even when the authoring tooling changes, unlocking the true value of your
content metadata.

Publishing Technology is a core discipline at Data Language. Our AI for Auto-tagging content and text classification combined with your product Data Graphs can provide a rich information backbone for the modern newsroom.

Publishing Tech

Case Study: How Data Graphs and AI Auto-Tagging transforms Publishing

Charlie Furniss

We have been hard at work improving our Knowledge Graph as a Service product 
Data Graphs [https://datagraphs.com] - in fact it is in a state of continuous
improvement. We have rolled out a number of new features in the first quarter of
2022, and here is a brief roundup on what they are:

Advanced search
Data Graphs Advanced Search feature allows you to build complex queries by
chaining and grouping a series of boolean and numeric expressions. For example
for finding a set of concepts in the graph where they are of Type A and Type B
or Type C, with a date property after a specified date, and a property equal to
some specified value. These searches are available in both the Data Graphs API
and now in the app too.

Saved searches
Saved search enables users to label, store and recall search criteria – great
when you are regularly using complex queries, particularly with Advanced Search
where building up items and groups can be time consuming and you want to return
to the same search results later.

Custom validation rules
When creating a property, custom validation allows you to choose from a set of
configurable rules that can be applied to certain data types and used to control
the property’s data, for example an age limit, an expiry date or a control on
input text. These rules include min/max characters (for text, keyword types),
regular expressions, before/after date (for date, dateTime) and min/max value
(for integer, decimal, latitude, longitude). You can also define cardinalities
on arrays and relationships in the graph.

Enum data types
The new enum data type allows you to create a property with a predetermined list
of options, from which users choose when setting its value. This will be useful
for any field whose value you want to limit to a small controlled vocabulary
that you may not want in your graph as entities in their own right. In the Data
Graphs Concept Editor the enum options are presented as a drop-down list on the
property.

Pre-commit web hooks
Data Graphs pre-commit web hooks allow you to perform a remote impact assessment
of a given event, for example, merging two Concepts, or deleting a Concept when
integrated with some target business system. When one of these actions occurs,
Data Graphs makes an HTTP request with the event’s details in the payload to an
endpoint in your environment, allowing you to analyse the downstream impact of
this change and respond by either accepting or declining the event or by
providing a warning (as illustrated). A full integration of these at Cochrane
has been discussed in this case study here
[https://datalanguage.com/blog/data-graphs-at-cochrane-a-case-study].

Improved geospatial visualisations
As well as the default map view of geospatial data, you can now view concepts
geospatially on dark, light, outdoors, and satellite views with automated entity
clustering based on the zoom-level of the map:

News, features and updates about our product Data Graphs. Find out how this intuitive knowledge graph product is easy to setup and connect. Brought to you from the team that was foundational in the earliest large scale knowledge graph projects in the UK.

Data Graphs - News, features and product updates from Data Language

Data Graphs

New Features in Data Graphs - Q1 2022

How Cochrane has integrated Data Graphs [https://datagraphs.com] with their own
enterprise graph database of evidence-based healthcare as a complete linked data
management solution.

Population
Data Language has been working with Cochrane as a technical partner for a number
of years now, and we have blogged extensively
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19] 
on the digital transformation at Cochrane, including how linked data and an RDF
based knowledge graph have become a core information backbone for their business
processes.

The Cochrane vocabularies of some 300k concepts provide the linked reference
data that describes the PICO domain of evidence-based healthcare, describing the
Populations (and their conditions, ages, sexes, social context), the
Interventions (drugs, devices, procedures, materials etc), Comparisons, and the
Outcomes. These concepts are annotated onto clinical evidence as PICO
micrograph
data structures [https://data.cochrane.org/ontologies/pico/index-en.html] to
describe the questions being answered in clinical evidence. All of Cochrane’s
vocabularies, content metadata and PICO annotations are stored in an enterprise
graph database, and exposed by a cohesive set of microservice APIs that can be
consumed and recombined to allow rapid innovation.

Data Language has provided the technical architecture, design and implementation
of the knowledge graph, microservices, APIs, and tooling that allows the
Cochrane subject matter experts to curate and QA these PICO data structures as
part of their daily business processes. This linked data platform has allowed
Cochrane to rapidly innovate: enhancing the Cochrane Library with linked data
query capability; and build at lightning speed, the largest Covid-19 database of
clinical evidence in the world, comprising more than 100k studies, described
with well-structured PICO metadata,  serving high volumes of requests and
queries to researchers globally.

Condition
While tooling and processes for Cochrane’s SMEs to create and maintain PICO
graphs describing the Cochrane systematic reviews and studies is state of the
art, the tooling around the management of the reference data vocabularies is
less sophisticated. The Cochane vocabularies consist of a highly curated set of
linked data concepts that form a graph in their own right with class and
property semantics, transitivity, and multi-parent taxonomic structures.
Although Cochrane has an existing visual editor tool to perform operations on
individual concepts in the reference data, it is limited in functionality, and
does not allow for bulk ingest and updates, rich search and discovery operations
for quality assurance, data cleansing, and other operations such as merge,
migrate, and workflow.

To solve the vocabulary management problem, we have integrated our product Data
Graphs [https://datagraphs.com] into the Cochrane linked data ecosystem.

What is Data Graphs ?
Data Graphs is a SaaS product designed to lower the barrier of entry to
Knowledge Graphs. While it can be used purely as a Knowledge Graph in its own
right, it can also be used as a linked data vocabulary management tool, and
integrated with an enterprise graph database, or other business systems using
its API and Web-hook event triggers.  It has an intuitive user experience for
managing linked data, and powerful search, discovery and visualisation
capability, effectively making an ideal vocabulary management tool for Cochrane.

Data Graphs allows you to design or replicate any domain model or ontology, and
populate it with linked data, so it was a straightforward task to replicate the
existing Cochane concept model
[https://data.cochrane.org/ontologies/core/index-en.html] and ingest their
vocabularies. The screenshot below shows the cochrane model recreated in Data
Graphs. 


Cochrane Core Concept Model
Data Graphs provides a number of benefits to Cochrane for managing their linked
data vocabularies :

 * Fully decoupling the workflow and processes around managing their linked
   reference data from their business critical systems.
 * Being able to flexibly and easily modify their core concept model without
   impacting their downstream processes.
 * An intuitive user experience for search, discovery, match, merge and
   migration of data.
 * Graph visualisation for exploring their linked data, and identifying
   anomalies and circular relationships.
 * Easy identification of duplicates
 * A full audit history of all changes made to the vocabularies
 * Injection workflow states into the linked data concepts - for managing
   pre-publication, publication and deprecation of terms.

Cochrane’s Data Architecture
Cochrane’s linked data architecture is centred around an enterprise graph
database (Ontotext’s RDF-based GraphDB
[https://www.ontotext.com/products/graphdb/]), a set of microservices and online
tools for annotating systematic reviews and clinical studies with creating well
formed PICO graphs that describe the evidence. A slice of the logical
architecture  is shown below and is discussed in more detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
:

Cochrane Linked Data ArchitectureIntervention
Integrating Data Graphs
In order to integrate Data Graphs we needed an integration pattern that is
robust and also allows Cochrane SME’s to understand the impact of changes they
may make to the vocabularies, such that they can make an informed decision
whether or not they should proceed with the updates.

For integration with an organisation’s business systems, Data Graphs comes with
an expressive, contemporary REST API, serving JSON-LD payloads (pull), and a
suite of webhooks that can be used for event-driven integrations (push) that can
be configured for different core operations, Create, Update, Delete, Merge etc.
For Cochrane’s purposes the webhooks are ideal. 

Pre-Commit Hooks
Furthermore, for each web-hook, a pre-commit remote endpoint can be configured,
which will be invoked before a write operation is committed in Data Graphs, on
which the target business system can respond with a payload of information that
Data Graphs will render to the user as an impact-assessment if the transaction
is completed. This gives the user an opportunity to back out. These pre-commit
hooks allows the target system to respond for a variety of scenarios :

 1. Silent commit - no impact, just commit the transaction silently, the user is
    not blocked.
 2. Force reject - in which the target system determines the transaction should
    not proceed, and the user is not allowed to commit the change.
 3. An impact assessment, with a risk score, an optional summary of the impact,
    and an optional table of data, that the Data Graphs user will be able to
    make an informed decision upon.

Resilience
When consuming web-hooks (essentially an HTTP endpoint or API in a target
business system, that is configured and invoked on some trigger condition), we
need to ensure to all reasonable endeavours that the event fired will not be
lost due to unforeseen circumstances (eg target system failures, connectivity
issues etc.).  To achieve this, the target system endpoint needs to be as robust
as possible and ideally decoupled from other downstream systems. 

The best way of achieving this in the target system is to build an extremely
lightweight microservice that simply receives the HTTP event payload and places
it on a topic or queue. This queue can then be consumed at the target system's
leisure and can cope with failure downstream (the message will not be lost if
the consumer fails temporarily). The target system can take this further with
dead-letter queues and retries to cope with a variety of challenging
circumstances. This is the pattern adopted at Cochrane and is a pattern we would
generally recommend for consuming webhooks between other products and platforms.
You can see the logical architecture for this below :

Data Graphs Integration Architecture at CochraneThe key integration points
(numbered in red)  within the Cochrane technical stack are :

 1. The Pre-commit web-hook handler. This is a simple webservice that is invoked
    by Data Graphs for a specific transaction event before the transaction is
    committed (for example a Concept Delete). The service makes a query against
    the Cochrane evidence graph to ascertain the impact the change will have.
    For example if a concept is used on thousands of PICO graphs, it is likely
    the user should not be deleting this concept, and the service can respond
    with a straight reject, or return a table of the PICOs affected.  This
    information is presented back to the user to make an informed decision.
 2. The Web-hook handler. This is a lightweight service that is invoked by Data
    Graphs for a specific transaction type (Concept Delete, Merge, Update etc)
     after the change is committed. This service simply places the data event on
    a queue, ready to be processed.
 3. Concept Event Handler. This service does the heavy lifting, taking events
    off the queue, and writing updates to concepts and PICOs in Cochrane’s Graph
    Database using SPARQL. If this fails for whatever reason, the message is
    retained on the queue, and can be tried later. For Cochrane much of this
    code already existed in the legacy LDGraphs API, which was performing a
    similar function for their original linked data editor, so was a case of
    porting this into the event handler.

The webhooks in Data Graphs are easy to configure, as can be seen on the
following screenshots.

First, choose an event type to trigger a web-hook, then configure the endpoint
in the target system, an optional pre-commit hook, and any headers / security
tokens that need to be sent with the request :

Webhook Configuration in Data GraphsMigrating the vocabularies into Data Graphs
The final step was to load the Cochrane linked data vocabularies into Data
Graphs, conforming the domain model we created earlier. This can be done in
different ways

 * Using the Data Graphs REST API
 * Using CSV upload using the CSV to Ontology mapping tool within Data Graphs

In this case, the API was the easiest approach as we could read pages of JSON-LD
concepts from Cochrane’s concepts API, and pipe them into the Data Graphs write
API. As the ontology model’s in both were identical, this was a relatively
trivial migration script.

Outcome
With this in place, Cochrane now has a state of the art linked data management
toolkit. The fully whitelabelled user experience engenders user-ownership.
Cochrane’s information experts can organise their core entities in logical
datasets for rapid navigation:

Cochrane Datasets
Drilling into each dataset, a SME can search for concepts, and easily find
potential duplicates through a complex query or use the label/property facets :

Faceted SearchAnd merge those duplicates with an intuitive UI :

Merging Concepts in Data Graphs
The resulting merge transaction will be handled by the webhook event, not just
to update and merge the concepts in the Cochrane Graph Database, but also update
all PICO graphs in which the source concept was previously used to reference the
target concept instead.  Similarly, deletes will remove the concept reference
from all content and PICO annotations.

Making any changes, such as deleting a concept results in the pre-commit hooks
requesting an impact assessment, so the user can make an informed decision to
continue or cancel:

Pre-commit Impact AssessmentsThe knowledge graph can be explored visually to
help find and correct anomalies in the vocabulary, such as circular
relationships :

Graph Explorer in Data GraphsThe Cochrane bioinformatics SMEs now have many more
capabilities at their disposal taking their ability to deliver the next
generation of evidence-based healthcare innovations to a new level.

.

Data Graphs at Cochrane: a Case Study

Silver Oliver

Data Graphs [https://datagraphsa.com] is a product we have designed to lower the
barrier of entry for organizations to launch a knowledge graph platform. We have
shortened the time to launch from months down to hours (in a recent event, we
built a Netflix knowledge graph from scratch with Data Graphs in just 12
minutes).

In Gartner's recent report
[https://www.gartner.com/en/articles/the-4-trends-that-prevail-on-the-gartner-hype-cycle-for-ai-2021] 
on the significance of knowledge graphs, they listed some obstacles that many
companies encounter when trying to implement and maintain them successfully. In
this piece, we'll be explaining how our new knowledge graph product, Data Graphs
handles each of these obstacles.

Gartner's Obstacle 1:

Data Graphs Solution:

Lowering the barrier of entry to knowledge graphs will make it easier for
businesses to see value quickly, and therefore pushing them higher up the
priority list. Specifically, this means speeding up the ability to do proof of
concepts, and do simple but informative demos that allow organizations to
realise early value at low cost and low risk. 

Gartner's Obstacle 2:

Data Graphs Solution:

Data Graphs is an out-of-the-box product. All the engineering and system
integration expertise, that Gartner points out as an obstacle to most companies,
is already taken care of. Data Language has already addressed the main
engineering challenge and initial setup within the product, and engineered the
knowledge graph capability. Integrating with Data Graphs is light, easy, and
based on common web standards and well constructed, robust APIs which saves you
having to wrestle with complicated graph data standards.

Gartner's Obstacle 3:

Data Graphs Solution:

This is a slightly more nuanced obstacle that in summary deals with the fact
that there are many different types of knowledge graph. Any one product,
including Data Graphs, is not a one-size-fits-all solution so companies must be
careful to choose the right knowledge graph for their use-case. 

Data Graphs, for example, is not trying to be an RDF Triple Store, which will
provide the infrastructure for a great knowledge graph for some use-cases, but
less suitable for others. 

In Data Graphs we’ve distilled-down what we’ve seen as the most useful
attributes of a knowledge graph and vastly simplified their use for businesses
that don’t want/don’t have the resources to engineer a full RDF triple store or
a property-label  graph solution.

Over our 10-15 years experience in implementing knowledge graph solutions in the
enterprise, we find that a large percentage of solutions don’t need RDF based
platforms. Instead, we have tried to create the perfect cocktail of enough
semantics to deal with the use cases that most businesses have, but at the same
time keeping it simple enough so as not to get bogged down in complexities that
the majority will never need. Data Graphs does provide RDF interoperability
through its JSON-LD APIs, and Data Graphs does not preclude you from using an
enterprise-scale graph database too. Data Graphs can also be used as a graph
entity editing tool, that can be integrated with an RDF or property graph
database, and this is a soluiton pattern used by some of our customers. 

Gartner's Obstacle 4:

Data Graphs Solution:

To tackle this obstacle we have to go back to the main lesson we always try to
impress on our knowledge graph clients: preparation is key. You have to
understand your data and design it well, solid domain modelling is always the
imperative first step. We often see projects get distracted by the technology
when in reality some of the most important work when creating a knowledge graph
is the preparation, making sure your data is described, structured and
understood clearly. Desgining a quality domain model/ontology for your own
use-cases is critical to unlock the value in sharing linked-data. Data Graphs
allows you to construct your domain model and visulize it within the tool, and
also allows you to export the domain model as an RDF/OWL ontology. Data Graphs
uses linked-data at its heart, to enable interoperability through JSON-LD and
JSON-LD APIs.

Find out more about Data Graphs and our knowledge graph experience below.

Data Graphs [https://datagraphs.com/] | Knowledge Graph Engineering
[https://datalanguage.com/knowledge-graph-engineering]

How Data Graphs addresses Gartner’s obstacles to Knowledge Graphs

Last week we presented a talk at the online Knowledge Graph Forum
[https://www.ontotext.com/company/event/knowledge-graph-forum-2021/] hosted by 
Ontotext [https://www.ontotext.com/]. Here’s an overview of our objectives for
this talk, and a summary of some of the things we spoke about. 

Our Aim
We assumed that our audience were people who were either interested or actively
working on knowledge graphs and feeling their way through the problem space.

Our main aim therefore with this presentation was to show the amazing
capabilities of our own knowledge graph product Data Graphs
[https://datagraphs.com/], and how quickly it helps you create a knowledge graph
for a domain.

Common Knowledge Graph Challenges
Before we started with the process of building the knowledge graph, we wanted to
highlight three main challenges that previous clients had raised time and time
again when faced with this task:

1. Semantics in your Knowledge Graph

If you start using an enterprise graph database and RDF, the semantics can get
complicated quickly and it can be easy to get bogged down in a lot more
complexity than you actually need. The core question then is: ‘How much
semantics is enough?’. One of our main objectives for this talk was to show that
we believe Data Graphs has just the right amount of semantics to demonstrate how
we can lower the barrier of entry for knowledge graphs.

2. Technologies

There's a whole suite of technologies which are generally new to most
organisations, they don't have this expertise internally around RDF and
semantics.

3. Insufficient Information Management

Many of the knowledge graph initiatives that we see are technology-driven. Many
companies buy the knowledge graph technology, overlooking the essential
information management side of things. Your knowledge graph is only as good as
your domain model and this must be made up of well curated data. Often companies
that have a knowledge graph will have neglected their information management and
the data quality is poor. We'll return to this later.

The Main Act
We took attendees through a step by step process of creating a knowledge graph
in just 12 minutes with Data Graphs, using the open source data that Netflix has
made available on Kaggle. We started by showing the domain model that we wanted
to replicate and then began the process of building the knowledge graph. 

The first step was to create datasets and concept classes, which is effectively
our project canvas. Then using Data Graphs’ schema builder we can replicate our
domain model. We built out the classes and their relationships and then finally
we have a visual view of our domain model, replicated within Data Graphs. The
next step was to load in the data. Taking individual CSV files prepared from the
Kaggle Netflix data we populated our domain model one concept class at a time,
and let Data Graphs stitch them together with identifiers that we had
predefined. Once all the data was loaded we had a ‘big reveal’, through the Data
Graphs explorer which allows you to navigate and visualise your knowledge graph. 

The audience was able to see all Netflix’ data on movies, shows, actors,
directors, countries etc. presented in an interconnected, structured, and
manageable way.

The Rewards
The rewards can be reaped immediately. Directly within the UI for Data Graphs
you can start giving people access to it to explore, search, query and facet the
data conforming to your domain model. You can also use immediately use the Data
Graphs APIs to build your own applications on top of the data. 

Key Message? Preparation Before Creation
The key message we were trying to impress on our audience was that for knowledge
graphs, good preparation is essential. The speed and relative simplicity of the
setup that we had demonstrated was only possible with solid information
preparation - data that was already clearly structured and well organised.
Pre-defining your domain model is an essential step in establishing the shape of
your data before embarking on a knowledge graph project. Many people jump into
ontologies well too quickly without spending enough time thinking about the
specifics of their data, the core entities in your domain, the key relationships
between them that will enable to you to answer the right questions of your data.

For example, in our case, before this talk we had gone through several questions
that we knew we would need to know the answers to when creating the knowledge
graph: 

 * What will the shape of the data be?
 * What ubiquitous language should we use to describe our model?
 * What identifiers will each of the data points have?
 * What sort of questions will be want our knowledge graph to answer?
 * What will the pros and cons of different approaches be?

And so on...

Making those decisions upfront and doing the work outside of the technology was
essential in allowing that knowledge graph creation to run smoothly. This
preparation is agnostic to the technology solution so once that is aligned, if
you have the right tool, which in this case was Data Graphs, it becomes simple
to put it all together. When that thinking doesn’t happen before the process of
creating a knowledge graph starts, you end up having to work out many imperative
and foundational design decisions within your already-chosen technology and thus
implementation becomes more complex.

For more information of knowledge graphs and how to implement them correctly,
visit the links below.

Data Graphs [https://datagraphs.com/] | Knowledge Graph Engineering
[https://datalanguage.com/knowledge-graph-engineering]

Creating a Knowledge Graph in 12 minutes at Ontotext’s Knowledge Graph Forum

Steve Liles

Maps in web applications are a pretty common feature these days, and with
amazing libraries like Leaflet [https://leafletjs.com/] its now pretty easy to
build a great mapping experience.

Typically these maps use the Web Mercator
[https://en.wikipedia.org/wiki/Web_Mercator_projection] projection, a variant of
the Mercator projection most people are familiar with from paper maps and which
has been in common usage for hundreds of years.

The Mercator projection takes our spherical world and maps (projects) it to a
rectangle, which is very convenient for viewing on a rectangular sheet of paper
or screen, but produces a distorted view of the world that inflates the size of
objects near the poles.

Greenland is NOT actually the same size as the entire African continent!
Likewise, things near the pole that are close together in reality can appear to
be far apart, and it is very difficult to visualise the proximity of objects in
the Arctic Circle when viewed on Mercator projected maps.

Luckily, it is possible to render maps on the web using alternative projections,
although it can be tricky to get up and running. In this post we’ll have a look
at setting up an Arctic projection with React, Leaflet, and Proj4js
[http://proj4js.org/] (via Proj4Leaflet
[https://kartena.github.io/Proj4Leaflet/]).

First lets get a map up and running with the default Web Mercator projection,
then we’ll walk through the modifications necessary to display an Arctic
projection.

Mapping with React + Leaflet
Getting started with the default projection is straight-forward — we just need
to set up a container for our map and render a TileLayer with a suitable tile
provider url. As we’re just testing we’ll use tiles from openstreetmap using the
following url pattern.

const TILES = https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png


The code to render the map with React-Leaflet is short and sweet:

import { MapContainer, TileLayer } from "react-leaflet";

const App = () => (  
  <MapContainer
    style={{
      width: 1000,
      height: 1000
    }}
    center={[0, 0]}
    zoom={2}
    minZoom={1}
  >      
    <TileLayer url={TILES} />
  </MapContainer>
)

export default App;

That’s all we need to render a world map in Mercator projection. Here it is in 
codesandbox [https://codesandbox.io/s/mercator-projection-s1v59]:

Note: it is important to give your map a size (width & height) otherwise you
won’t see anything!Now that we have a map rendering in Mercator projection, lets
convert it to a polar projection, specifically the Arctic Lambert Azimuthal
Equal Area projection (Arctic LAEA for short). To do that we need to provide
Leaflet with two things:

 1. a coordinate reference system (CRS) to enable it to project and unproject
    coordinates, and
 2. a set of tile images are appropriate for that CRS

The CRS implementation comes courtesy of Proj4Leaflet, and must be parameterised
with a proj4 configuration string and details of the map tile bounds and
resolutions.

Proj4 configuration
We can find the proj4 configuration string at epsg.io [https://epsg.io] simply
by searching for the projection we want to use. There are several Arctic LAEA
projections we could use, I’m going to go ahead with epsg:3575
[https://epsg.io/3575]. The configuration string we need can be found under the
section titled “Export” for the proj4js link, which gives the following:

proj4.defs(
  "EPSG:3575",
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"
);

Map Tile Configuration
Next, we’ll need a compatible set of tiles for the EPSG:3575 projection. I’m
going to use tiles hosted by the Global Biodiversity Information Facility
[https://tile.gbif.org/ui/]. The map tile url pattern is given on that page as:

https://tile.gbif.org/{srs}/{tileset}/{z}/{x}/{y}{format}{params}

Here, {srs} is the spatial reference system (in our case 3575), {tileset} can
only be omt, {format} allows for a choice of vector or various resolutions of
png (i’ll use @4x.png), and {params} allows us to choose a tile style - only for
png tiles - where i’ll use ?style=gbif-classic, which altogether gives us a tile
url like this:

https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en

We need to provide the CRS with a little configuration related to the map tiles,
which we can derive from the extents provided by Global Biodiversity Information
Facility (see the table in the usage section). The tiles are 512x512 and the
provided extents for the northern hemisphere are:

–6371007.2 √2, 6371007.2 √2, 6371007.2 √2, –6371007.2 √2

So we can derive the extent and resolutions like this:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1)  
  .fill()  
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));

Complete Proj4Leaflet Configuration
Putting it all together we get the following:

const MAX_ZOOM = 16;
const TILE_SIZE = 512;

const extent = Math.sqrt(2) * 6371007.2;
const resolutions = Array(MAX_ZOOM + 1) 
  .fill() 
  .map((_, i) => extent / TILE_SIZE / Math.pow(2, i - 1));
  
export const ARCTIC_LAEA = new Proj.CRS(
  "EPSG:3575", 
  "+proj=laea +lat_0=90 +lon_0=10 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs", 
  { 
    origin: [-extent, extent], 
    bounds: L.bounds(
      L.point(-extent, extent), 
      L.point(extent, -extent)
    ), 
    resolutions: resolutions 
  });
  
const ARCTIC_TILES_URL = 
  "https://tile.gbif.org/3575/omt/{z}/{x}/{y}@4x.png?style=osm-bright-en";

All that remains is to update MapContainer to refer to the new CRS and center on
the pole.

crs={ARCTIC_LAEA}
center={[90, 135]}

... modify TileLayer to use the arctic tiles ...

url={ARCTIC_TILES_URL}

And we’re done! Check out the complete working code on codesandbox
[https://codesandbox.io/s/react-leaflet-arctic-bxw7d].

Boom! Arctic Map!Welcome to the Arctic!

Software Development

Arctic Projections with Leaflet and React

In my last blog post, I introduced our new SaaS product Data Graphs
[https://datalanguage.com/blog/data-graphs-fully-managed-knowledge-graphs] - a
cloud hosted Knowledge Graphs as a Service platform - effectively lowering the
barriers to entry into knowledge graph adoption. Data Graphs
[https://datagraphs.com] not only makes it trivially easy to spin up a knowledge
graph for a domain or project, but provides a very intuitive user interface for
curating and managing your data within, along with search, discovery and
integration using a richly featured API.

Since then we have been busy rolling out new features, the most striking of
which are the new visualisations. Data Graphs now lets you automatically
visualize your domain model, and lets you navigate your graph data visually too.

The Domain Model
This visualization is a superb way to verify the domain (or ontology) model you
have created in Data Graphs accurately reflects the true nature of your domain
and the semantic relationships between classes of concepts are as you intended.
This visualisation lets you see the structure of your schema, including all
primitive data properties, relationship properties, and any subclasses you have
defined. 

A typical pattern is to draw your domain model first in some external tool, or
on paper or whiteboard, then construct the schema in Data Graphs. Now you can
visually verify if the models align.

The domain model is visualized using a force-directed graph. If the automatic
render is not perfect,  you can use the controls to adjust the layout. Each node
can be pinned and moved simply by dragging it to the position you prefer.  You
can pin all nodes using one of the control bar buttons on the left of the
diagram. Subclass relationships can be toggle on / off, and you can also
remove/prune classes using the sidebar that lists all the concept types in your
project. When happy with the layout, you can save it, so next time you view it,
it will retain the visual changes you have made.

Graph Explorer
You can explore your knowledge graph visually starting from any concept in your
datasets.

There is an option in the concept view screen to “explore graph”. Clicking this
will open the data visualization using the concept selected as the starting
node.

The graph opens and you then have the ability to explore the graph. Clicking on
a node in the graph opens all its inbound and outbound relationships. It also
renders the data about the node selected in the sidebar on the right.

If a concept has a preview image defined, these will be rendered as the
canonical image for each node.

To navigate at a more granular level, right clicking on a node will open a
context menu where you have the option to open or close specific relationships,
or prune the node itself from the graph visualization.

Like in the domain model, holding and dragging a node pins it the place you move
it to.

I hope these are helpful. We have plans to continually improve these
visualization features over the coming months.

Knowledge Graph Visualization in Data Graphs

Data Graphs is the latest Data Language baby. Data Graphs
[https://datagraphs.com] was born out of need, a need for really easy structured
data management. Our thoughts on this evolved to eventually produce a SaaS
product that allows a team to curate a knowledge graph for any domain, from
simple flat concept collections to complex structured domain data. This is all
wrapped up in an extremely intuitive user experience, combined with a fully
comprehensive API and web-hooks to give you extremely efficient integration with
applications and business systems. You can check it out here : 
https://datagraphs.com

What does it do ?
Put simply it provides a dead simple way to create knowledge graph data models,
populate them with data, and interrogate (query) and integrate that data with
your applications using a flexible and easy API.  In essense, it is Knowledege
Graphs as a Service (KGaaS).

Building your own knowledge graph platform can take months, requites highly
specialised expertise and lot of $$, and that is before you start populating it
with data. Even when you build your own KG platform, the information management
tools dont come for free, and are typically sorely lacking in utlity.

We have short-circuited that complex engineering process, and done all the hard
yards for you. Futhermore we have added a stunning information management user
experience. With Data Graphs you can be up and running in minutes, start
creating your graph data models, and managing your data.

You don’t have to worry about scaling, engineering, backups, it’s all handled
for you. But best of all is the super-intuitive user interface for managing,
searching, and maintaining data over time. The API lets you search, query,
filter, and facet on your data - slice and dice it. The API is rocket fast and
easy to use, and also geospatially aware.

We think Data Graphs addresses a lot of use-cases, from straight-forward data
catalogs or vocabulary management, to easy to setup and maintain knowledge
graphs. You could use it for managing structured tagging data for a publishing
system, integrating it with your CMS, or even for maintaining time series
structured data powering charts in WordPress for data journalism. It is ideal as
a central metadata or reference data repository for your enterprise business
systems. You could even use it for maintaining a nicely categorised and faceted
product catalog to integrate with your e-commerce store.

Schemas, Concepts, and Datasets
The Data Graphs model is based around three fundamental ideas - 

Datasets - these are collections of Concepts - organised into groupings of your
choice. Datasets can hold concepts of different types, or concepts all of the
same type. Datasets can also be public or private -

 * a public dataset means the data it holds is accessible via API using only an
   API key, so it could be directly accessible via anyone on the internet, who
   has access to your API key.
 * a private dataset means the data it holds can only be accessed using Open ID
   / oAuth security credentials

Schemas - this is your data model, definitions of the things (or concepts) in
your domain, their properties, and how they are linked together, the
relationships between the things. Data Graphs has an intuitive user interface
that lets you quickly design a data structure that reflects your own domain
model.  Each Concept type in your domain has its own schema definition where you
define the properties of a concept and the data types of those properties
(dates, integers, text etc). For example, a simple schema of a Person might have
a givenName (text), familyName (text), and a dateOfBirth (date), and a headShot 
(image).  Rather than explain it in depth here, if you need to know more, we
have made a short video
[https://datagraphs.com/demos/how-to-setup-your-data-sets] that shows how
quickly you can create a schema for your domain model.

Concepts - the data itself. Concepts are the data instances that you load into
Data Graphs conforming to your schemas. You can create and edit concepts via the
user interface, bulk import them from for example CSV files or external sources,
or load them via the Data Graphs UI. Once created, you can search , discover,
and maintain the data via the UI.

SaaSification
Data Graphs is fully SaaSified - you can sign up online [https://datagraphs.com]
, frictionless, and get started immediately with a free trial. You can manage
your own users, roles, your own backups, your own API keys and oAuth
credentials. 

The User Experience
Delivering a great data management experience was our primary goal when creating
Data Graphs. In almost all the knowledge graph and information management
projects we have worked on, data management and curation tooling was always an
area that was a challenge. We set out to build something extremely intuitive to
use, with a UI that would be self-aware of the data types, the relationship
semantics, and make it super easy for users to explore, find, and navigate data.

Dog Food
We are great fans of the “Eat your own dog food” approach to product
development. If our APIs are not good enough for ourselves to build an
application upon, how can we expect them to be good enough for others? With this
in mind, all the features in the Data Graphs Application are built upon the same
APIs exposed to the Data Graphs client, and with that in mind, let’s talk about
integration..

Integration
There is no point having a fabulous user experience for ingesting and managing
your data if you cannot integrate it. As with most SaaS products Data Graphs
comes with an API. A set of REST endpoints for searching, filtering, querying,
faceting, retrieving and creating data. It is fast, it is flexible.

With the API you can access data only in specified datasets, or you can federate
a query across all your data.

If used with only your API key, you access only the data in public dataset. If
used with your oAuth credentials, you can access both public or private
datasets.

Web Hooks - Triggered
The API gives us pull integration, web hooks provide us with push. You can
configure web hooks for Concept create, update, delete, and also candidate
(concept) approval. You define optional endpoints (typically in your  own
platform) that are invoked whenever a specified Data Graphs event is triggered.
For example you could trigger some downstream system when a Concept of a
specific type is created or updated. For example if an Organization concept is
updated, trigger some workflow in an existing business system. The web-hook
events supply the entire concept data in its payload, which means you could use
this to push data into another database or application.

Querying data - The Data Graphs Filter Expression Language
As well as features such as sort and facet (by concept property) the API lets
you query your data using a flexible filter expression language (often called
NQL).  In simple terms, this lets us filter data like this 
filter=property1:value1,property2:value2

A filter expression is a string which provides a (Concept)  property, operator
and value in the form where property is a JSON path representing the field to
filter on (using JSON dot notation) and operator is optional , and tells the API
how to compare values. value is the value to match against.

Operators can be -(not), >, >=,<<=, and also a test if a value is in a group of
values [value,value,...]

Boolean expressions are also supported including parentheses for grouping
expressions together, and overriding operator precedence. 

Full text search is supported using the traditional q parameter. We index all
the text fields in concepts, and boost the canonical labels.

Combining all these together lets us build extremely high utility data queries,
and very flexible integrations.

You can query within a specified dataset, or you can federate across all your
data, and using the filter expression language it means you can ask extremely
useful questions of your data.

The API is fast - you can expect all requests to be returned in less than a few
hundred milliseconds.

Geospatial
We also support geospatial data. We have primitive datatypes for latitude, 
longitude and geohash - so both our API and UI can be geospatially aware for
concepts with location information. As well as being able to explore geospatial
data in the UI on maps, you can also query geospatially using point-radius,
bounding-box, and bounding -polygon queries. Any geospatial query can be
combined with other API filters and full text search.

This means you can ask questions of your data such as “Find me all the concepts
within 50km of New York, classified by X, with property P greater than Y, and
the word “hello” in their label”. Nice.

For example, given a geospatial data set of Airports with relationship to
airlines , get all airports within a 50km radius of London, serviced by EasyJet,
we might use an API request like :

/airports?filter=_near((51.51, -0.12), 50km),airlines:urn:demo:airlines:easyjet

Client Library
For developers we have published a Javascript client library for Data Graphs to
make integration even easier. The client library is particularly useful for
consuming the API using OpenID credentials, as the machine-machine OpenID
authorization flow is all taken care of.  The client library documentation is
published in NPM here : 
https://www.npmjs.com/package/@datalanguage/datagraphs-client

Sub-classing, Inheritance, and Transitive Inference
While Data Graphs is not a full blown graph database, it is a fully managed
structured data solution that can be used to build a high-utility highly
performant knowledge graph for your organization without having to manage any
infrastructure.  There are some key features in knowledge graphs that add
significant value. 

Subclassing and class inheritance is one of those features. In our (extensive)
experience of working with graph databases, inference is a feature that is
widely misunderstood. The set of potential inference rules available in the full
RDF and OWL specification is considerable. However in most practical
implementations of knowledge graphs (outside of academia), there are just one or
two core inferences that add significant value in most business applications.
Subclassing and transitivity is top of that list. Data Graphs has full support
for this.

When you create a new Concept schema definition, you have the option of
declaring a parent type (class) and inheriting its properties. 

Create a Concept with a parent typeOnce you have created your concept schemas,
we have a neat class structure management tool like this :

Concept relationships (and the User Experience) then become class structure
aware, with transitive inference.  When a concept property defines a
relationship to another concept type/class the UI allows you to select concept
instances of that type and all its subclasses.

Transitive relationship propertiesAPI queries also obey the usual semantics of
transitive inference across a class hierarchy. 

Linked Data and Graph Databases
Data Graphs has support for JSON-LD. API responses can be requested with an 
application/ld+json mime-type such that the data is returned in JSON-LD with a
context dynamically constructed from your schema.

This lets you transform data directly to RDF for loading into your own graph
database, or another RDF compliant system. 

So while functioning capably as a standlone fully managed knowledge graph, Data
Graphs could thus also be used as a tool-suite for editing data in a downstream
enterprise graph database / triple store. Using the web-hooks, it is fairly
trivial to build a service that updates entities in a graph database when the
web-hook is triggered from a concept create/update operation in Data Graphs.

The Road Ahead
Data Graphs already has a plethora of great features
[https://datagraphs.com/features], but we have some great new ones lined up in
our road map. Not least is data and model visualisation.

Building on the class structure visualisation we plan to render detailed domain
model / ontology diagrams dynamically from your schema. We also plan a dynamic,
navigable graph visualisation of the instance data itself. On the back-end, we
plan to offer regional data storage options.

We think we have produced something really special with Data Graphs, something
that really delivers utility and value for information management in your
organisation - but don’t take my word for it, go ahead try it out for free
[https://datagraphs.com].

Blog