Paul Wilton

Data Language are trusted experts delivering knowledge graph products and services to global brands since 2010.

Knowledge Graph Engineering experts - Data Language

Data Language has exceptional experience in the design, implementation and
deployment of knowledge graphs, linked data, and semantic web technologies. We
have been working with graph databases since our inception, when our founders
worked on possibly the UK’s first enterprise scale knowledge graph for BBC News
and Sport. Since then we have helped global organisations such as Cochrane,
Wellcome, and Euromoney implement them too. Our typical approach to implementing
a knowledge graph solution is:

Domain Driven Design
Through collaborative domain modelling exercises, we work with stakeholders to
gain a broad understanding of the target domain, iteratively refining and
validating ontology models to achieve the ideal level of detail to deliver the
best outcomes, business value and utility.

Technical and Data Architecture
Not all graph databases are equal. We are wholly vendor-agnostic, and our deep
experience in designing scalable, maintainable, and evolvable knowledge graph
solutions, means we will always select the right graph technology to deliver
your use-cases. We understand the difference between label-property graphs and
RDF based graph databases. We have a total grasp on the practical and pragmatic
use of semantics and inference to ensure your solution is not over-engineered
but delivers just the right level of complexity to optimise your total cost of
ownership, and ensure your solution can evolve as your business evolves.

Integration - Data and Software Engineering
We can help you integrate the knowledge graph with your business systems and
workflows, either by helping your own software engineering team with querying
the graph using query languages such as SPARQL or Gremlin, through to developing
microservices and web interfaces that encapsulate your target use-cases. We are
skilled at data engineering, and can help you populate your knowledge graph,
transforming and loading your own business data, and linking to or ingesting
open-data as required.

Automation and Deployment
We can work with and guide your own infrastructure team, or take on the task of
engineering, integration and deployment entirely ourselves. We are skilled at
contemporary best practices for deployment of infrastructure in the cloud or
on-prem. Our devops engineers are skilled in the use of infrastructure-as-code
for fully automating the deployment and maintenance of the entire knowledge
graph solution.

Graph databases

Knowledge Graph Engineering

Matt Shearer

Domain-driven design, or domain modelling is about optimising a team's knowledge and understanding of the domain it works in. Find out how DDD can benefit your project here.

What is Domain Driven Design and why do you need it?

What is Domain-Driven Design?
Domain-Driven Design (DDD) is a method for developing a team's understanding of
a problem space. It emphasises placing the primary focus of a project on the
core area of the business (the core domain). This often takes the form of
facilitated workshops with domain experts and the development of a shared set of
conceptual models. DDD has the additional benefit of creating a shared
understanding (ubiquitous language) between teams when designing technical
solutions.

Skilled facilitation of domain modelling sessions helps our clients get to grips
with their domain. Through collaborative domain modelling exercises, we work
with stakeholders to gain a broad understanding of the target domain. We
typically focus on the high value elements of the domain, iteratively refining
and validating models to achieve the right level of detail to deliver the most
business value.

Would my project benefit from Domain-Driven design?
Common reasons that you would look to Domain Driven development would be
projects that need to distill the knowledge of domain experts into a product.
These types of projects can often suffer from a disconnect in domain
understanding. Discovering this disconnect in code or in data models is
expensive to fix. DDD can address this by tackling differences in domain
understanding before implementation begins. It can create a perfect bridge of
understanding between the business stakeholders, the problem-space, and the
technical teams.

Another application of DDD is in projects that require strong information
architecture. Where the intention is to organise a product around the entities
and relationships of the domain. This is often seen in approaches that build on
taxonomies or knowledge graphs.

We see Domain-Driven Design as a critical starting point for all our knowledge
graph implementations, and on the majority of our project engagements, including
development of our own products.

News, features and updates about our product Data Graphs. Find out how this intuitive knowledge graph product is easy to setup and connect. Brought to you from the team that was foundational in the earliest large scale knowledge graph projects in the UK.

Data Graphs - News, features and product updates from Data Language

Data Graphs

Domain-Driven Design

Find out about our feature at KGC2022, and explore this Netflix, Hulu, and Amazon Prime knowledge graph, using our product "Data Graphs".

Data Graphs at the Knowledge Graph Conference 2022

We've launched a read-only version of Data Graphs for participants at the
Knowledge Graph Conference 2022 to have a go, hands-on.

Data Graphs is our Knowledge Graphs as a Service product. It is used by global
brands to provide a knowledge graph platform to accelerate product
time-to-value.

Lowering the barrier to entry for Knowledge Graph Platforms
Often knowledge graphs are seen as a high-cost and high-risk strategy that only
the largest of companies can implement, over a very long time. 

With Data Graphs, we've ensured that businesses can see value rapidly.
Specifically, this means speeding up the ability to do proof of concepts, and
simple but informative demos that allow organizations to realise early value at
low cost and low risk. Here's an example of one of those demos right now.

13 mins to set up:
Netflix, Disney+, AmazonPrime, and Hulu
When developing Data Graphs, we wanted to make the setup and creation of a
knowledge graph, simple and quick. 

This video, prepared for KGC2022, shows how we created a knowledge graph,
complete with secure API and powerful UI tools, ready for production, in 13
minutes:

Try it yourself! - play with the Data Graphs demo here
[https://media-graph.datagraphs.io/media-demo/_all/concepts/?q=&pn=1&ps=15&sf=_label&sd=asc&layout=list&viewport=s000000000,1&utm_source=website&utm_medium=kgc-post&utm_campaign=faceted-search]
This demo version was created using Netflix, Disney+, AmazonPrime, and Hulu
data, and gives you a good view of just some of the key Data Graphs features.

1. Create your Domain Model for your Knowledge Graph
After you have designed and created your domain model with the business entities
and the relationships that link them together, Data Graphs generates a visual
representation of the model so that you can see clearly how the knowledge of
your business interconnects. 

Check out the Domain Model for Shows, Cast, Genres, Ratings here
[https://media-graph.datagraphs.io/media-demo/_domain-model/?utm_source=website&utm_medium=kgc-post&utm_campaign=domain-model]
.

2. Use our powerful Faceted Search over your knowledge graph
The search capabilities of Data Graphs are stellar and allow you to find and
explore your data precisely and most importantly, quickly. With the Advanced
Search feature, you can build complex queries by chaining and grouping a series
of boolean and numeric expressions, to pinpoint the targets of your search. 

Test the search capabilities of Data Graphs here, on 115k concepts including
shows, cast, directors, genres, countries, and more
[https://media-graph.datagraphs.io/media-demo/_all/concepts/?utm_source=website&utm_medium=kgc-post&utm_campaign=faceted_search]
.

As well as an intuitive visual search tool, all the Data Graphs search features
are available via API so you can integrate them with your business systems and
apps.

Explore the Data Graphs API here [https://support.datagraphs.io/].

3. Use the Graph Explorer to visualise your knowledge graph
The graph explorer is one of the star features of Data Graphs. You can now
visualise your knowledge graph in an interactive representation of your data,
that you can then explore and dive into, using the nodes. By visualising your
entire knowledge graph, from any concept in your dataset, we open up a wealth of
possibilities for your structured data. 

Interact with the knowledge graph explorer here, starting from the "Stranger
Things" node
[https://media-graph.datagraphs.io/media-demo/shows/concepts/urn:media-demo:TvShow:netflix-s3686/explore/?utm_source=website&utm_medium=kgc-post&utm_campaign=knowledge-graph-explorer]
.

4. Use the Graph Path Analysis to gain insights
Data Graphs lets you visually explore paths between any entities in the graph,
in order to find the shortest paths, and to specify constraints on both nodes
and properties that make up these paths. A key feature of knowledge graphs, this
type of analysis is typically used in compliance, fraud detection, and supply
chain analytics. 

You can try out the graph path analysis here
[https://media-graph.datagraphs.io/media-demo/_all/_graph-search/?utm_source=website&utm_medium=kgc-post&utm_campaign=graph-path-analysis]
.

About the Data Graphs demo
The Data Graphs demo is a read-only view of the full version of Data Graphs
[https://datagraphs.com]. The complete product provides a full suite of tools
for curating, ingesting, editing, and maintaining data in a knowledge graph,
along with APIs for integration of your graph data with apps and business
systems.

About the Media Graph data
The media graph data has been sourced from the open data provided by Netflix,
Hulu, Amazon Prime, and Disney+ via kaggle.com here
[https://www.kaggle.com/datasets/shivamb/netflix-shows]. The data sets from
these 4 publishers have been cleaned, loaded, and linked together in our graph
to join 72k actors and directors across 21k  TV shows and movies, all faceted by
genre, rating, country, and year of production.

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

Data Graphs at The Knowledge Graph Conference 2022

Alexi Buckingham

Get hands-on and try the Graph Explorer yourself. Explore the data and interconnections, and help everyone understand your knowledge graph.

Graph Explorer - To demonstrate your Knowledge Graph

The Graph Explorer feature on Data Graphs [https://datagraphs.com/] allows you
to navigate through your knowledge graph visually and interact with your data in
a new light. Being able to present your information visually in the Graph
Explorer helps provide the context, meaning, and interconnection of your data,
an essential asset to your digital transformation.

Demonstrate your Knowledge Graph
A clickable graph user interface enables a visual way of explaining how your
knowledge graph can work for your stakeholders and users. This approach enables
a clickable journey around your business data so that it brings the key
information to life. In our experience this approach generates that "lightbulb
moment" that you have been looking for!

Try it yourself on our Media Demo [https://media-graph.datagraphs.io/media-demo]
Click here [https://media-graph.datagraphs.io/media-demo] to try out our Media
Demo, a read-only version of our product made solely for exploration purposes so
that you can try out the amazing capabilities of the product with no strings
attached. This knowledge graph is populated by thousands of data points from
Netflix, Disney, Hulu, and Amazon and has a plethora of features available to
try out including Search
[https://media-graph.datagraphs.io/media-demo/_all/concepts], Advanced Search
[https://media-graph.datagraphs.io/media-demo/_all/concepts?aq=], Path Analysis
[https://media-graph.datagraphs.io/media-demo/_all/_graph-search], and Domain
Model [https://media-graph.datagraphs.io/media-demo/_domain-model].

Kevin Bacon on Graph Explorer
Click here
[https://media-graph.datagraphs.io/media-demo/people/concepts/urn:media-demo:Person:3Vn1Bxu4W6yzLNL54eB4b5/explore] 
to try out the exact same action being completed in the gif below - exploring
the all media data that links to Kevin Bacon... With Graph Explorer you can
navigate visually through his films, the countries of those films, other actors
he's worked with and much more.

[https://media-graph.datagraphs.io/media-demo/people/concepts/urn:media-demo:Person:3Vn1Bxu4W6yzLNL54eB4b5/explore]
What's the shortest path between Sandra Oh and Ricky Gervais?
Click here
[https://media-graph.datagraphs.io/media-demo/_all/_graph-search/?from=Sandra+Oh%28urn%3Amedia-demo%3APerson%3A6XCMzX4fttIARGWlEPynkN%29&to=Ricky+Gervais%28urn%3Amedia-demo%3APerson%3A2AafOAewdEpIBHxVTKAOHg%29&constraints=0%7Bi%5Byear%3Ar%3A1990%3A2020%5D-%3Ecast-%3E*%7D&view=graph?utm_source=socials&utm_medium=twitter&utm_campaign=shortest-path] 
to complete the action below - finding the shortest path between Sandra Oh and
Ricky Gervais, with path constraints on other actors and shows between 1990 and
2020, and then clicking the Graph Explorer button to visualise the path.

[https://media-graph.datagraphs.io/media-demo/_all/_graph-search/?from=Sandra+Oh%28urn%3Amedia-demo%3APerson%3A6XCMzX4fttIARGWlEPynkN%29&to=Ricky+Gervais%28urn%3Amedia-demo%3APerson%3A2AafOAewdEpIBHxVTKAOHg%29&constraints=0%7Bi%5Byear%3Ar%3A1990%3A2020%5D-%3Ecast-%3E*%7D&view=graph?utm_source=socials&utm_medium=twitter&utm_campaign=shortest-path]
And this entire knowledge graph can be created in just 13 minutes in Data
Graphs... Watch this video to find out how.

Find out more about Data Graphs here [https://datagraphs.com/].

Graph Explorer - Demonstrate your Knowledge Graph

Our knowledge graph product Data Graphs [https://datagraphs.com/] has a brand
new feature. You can now view your entire knowledge graph in the 'Domain Model'
view. This allows you to see your data presented as a spider diagram and explore
it interactively by moving each data point to wherever you like. 

Being able to present your data visually is an essential facet in your company's
digitisation journey and this new Data Graphs feature makes this simple.

Find out more about Data Graphs and Data Visualisation here
[https://datagraphs.com/].

Domain Modelling

New Data Graphs Feature: 'Domain Model' View

Who is Cochrane?
Cochrane [https://www.cochrane.org/] is the global not-for-profit organisation
whose core mission is to put evidence at the heart of health decision-making all
over the world. Over 90% of WHO guidelines are informed by Cochrane’s systematic
reviews.

What is the Covid-19 Study Register?
The Register is a precisely scoped, authoritative source of the latest clinical
evidence from around the world concerning the Covid-19 pandemic, that is being
used to inform point-of-care clinical decision making and evolving healthcare
policy internationally.

Healthcare professionals can use it to locate all relevant, reliable clinical
evidence about specific healthcare questions from the latest Randomised
Controlled Trial (RCT), observational, modelling and other studies to support
the fight against Covid-19.

This crucial, trusted resource is essential at a time when the evidence base is
developing rapidly, as healthcare professionals require structured and reliable
up-to-date information.

Partners with Cochrane since 2014
We first worked with Cochrane in 2014 on a digital transformation project,
designing and delivering a radical new platform architecture centred around
linked-data and knowledge graph technologies. Since then we have worked to
refine, iterate and scale their platform to create a suite of applications to
publish evidence in healthcare using structured data.

Solid data architecture makes rapid innovation possible
This Covid-19 Studies Register is the latest example of how Cochrane is able to
innovate rapidly on the scalable knowledge graph platform architecture delivered
by Data Language. The structure and flexibility means that new use cases can be
delivered rapidly and efficiently.

See the Covid-19 Study Register here [https://covid-19.cochrane.org/].

Find out more
 * Cochrane's Covid-19 Resources and News
   [https://www.cochrane.org/coronavirus-covid-19-cochrane-resources-and-news] 
   (page updated regularly).
 * More about Cochrane and the rapid delivery of the Covid-19 Study Register
   project
   [https://datalanguage.com/blog/covid-19-rapid-response-technical-delivery-at-cochrane]
   .
 * Our work with Cochrane since 2014 - Information Architecture, PICO Ontology
   and more
   [https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
   .
 * More about our Knowledge Graph Engineering
   [https://datalanguage.com/knowledge-graph-engineering].

Launching the Cochrane Covid-19 Study Register

In my last blog post, I introduced our new SaaS product Data Graphs
[https://datalanguage.com/blog/data-graphs-fully-managed-knowledge-graphs] - a
cloud hosted Knowledge Graphs as a Service platform - effectively lowering the
barriers to entry into knowledge graph adoption. Data Graphs
[https://datagraphs.com] not only makes it trivially easy to spin up a knowledge
graph for a domain or project, but provides a very intuitive user interface for
curating and managing your data within, along with search, discovery and
integration using a richly featured API.

Since then we have been busy rolling out new features, the most striking of
which are the new visualisations. Data Graphs now lets you automatically
visualize your domain model, and lets you navigate your graph data visually too.

The Domain Model
This visualization is a superb way to verify the domain (or ontology) model you
have created in Data Graphs accurately reflects the true nature of your domain
and the semantic relationships between classes of concepts are as you intended.
This visualisation lets you see the structure of your schema, including all
primitive data properties, relationship properties, and any subclasses you have
defined. 

A typical pattern is to draw your domain model first in some external tool, or
on paper or whiteboard, then construct the schema in Data Graphs. Now you can
visually verify if the models align.

The domain model is visualized using a force-directed graph. If the automatic
render is not perfect,  you can use the controls to adjust the layout. Each node
can be pinned and moved simply by dragging it to the position you prefer.  You
can pin all nodes using one of the control bar buttons on the left of the
diagram. Subclass relationships can be toggle on / off, and you can also
remove/prune classes using the sidebar that lists all the concept types in your
project. When happy with the layout, you can save it, so next time you view it,
it will retain the visual changes you have made.

Graph Explorer
You can explore your knowledge graph visually starting from any concept in your
datasets.

There is an option in the concept view screen to “explore graph”. Clicking this
will open the data visualization using the concept selected as the starting
node.

The graph opens and you then have the ability to explore the graph. Clicking on
a node in the graph opens all its inbound and outbound relationships. It also
renders the data about the node selected in the sidebar on the right.

If a concept has a preview image defined, these will be rendered as the
canonical image for each node.

To navigate at a more granular level, right clicking on a node will open a
context menu where you have the option to open or close specific relationships,
or prune the node itself from the graph visualization.

Like in the domain model, holding and dragging a node pins it the place you move
it to.

I hope these are helpful. We have plans to continually improve these
visualization features over the coming months.

Knowledge Graph Visualization in Data Graphs

Julian Everett

Over the past five years it has been our privilege to be a technology partner of 
Cochrane [https://www.cochrane.org/]. For anyone unfamiliar with their work,
they are globally recognised as the gold standard in evidence-based healthcare.
Historically their primary published output has been systematic reviews, which
evaluate and summarise the current state of human knowledge about specific
clinical questions. These then inform both healthcare policy and clinical
guidelines for point-of-care decision making around the world. For example, over
90% of WHO guidelines are informed by Cochrane reviews. 

The review production process starts with the formalisation of the clinical
question of interest in terms of the Population it relates to (e.g. age, sex,
condition), the Interventions being investigated (e.g. drugs and/or procedures),
the Comparators against which the Interventions are being evaluated (e.g.
placebo), and the Outcomes against which effects are measured. This is then
translated into search protocol and run against trial registries and journal
databases to identify all relevant clinical study information in the public
domain. The matches are then evaluated for risk of bias, and the subset of
results which meet Cochrane’s stringent data quality thresholds are subsequently
aggregated and analysed to determine the cumulative effect estimates. 

In essence, this is about the identification of causal relationships between
interventions and outcomes. To that end, up until now Cochrane has focussed on
double-blind randomised controlled trials. They provide the highest quality
input data: randomisation greatly reduces the likelihood of alternative
explanations for the observed results as the number of participants increases;
blinding of field researchers and participants (i.e. hiding information about
specific interventions) removes the cognitive and behavioural bias risks that
might arise if, for example, placebos are known. 

Finally, review production is not a static process. New information is being
released into the public domain all the time both in the form of additional
journal articles about completed studies as well as information about new
studies. As a result, ongoing evidence surveillance and timely meta-analysis
updates are important additional requirements.

During our time working with Cochrane, we have built a new data platform using
domain-driven design techniques which has an explicit information model encoded
into a knowledge graph at its core. The model is published here
[https://data.cochrane.org/ontologies/]. This is complemented by a loosely
coupled, service-based architecture which ensures each business capability in
the value stream is cohesively encapsulated. In a follow up article, my
colleague Paul Wilton will review how the technical architecture hangs together
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
. The platform design allows services to be very quickly deployed in new
choreographies as the business context changes - something which has proved
invaluable over the last few weeks. 

So, what have been the business function and information flow impacts of the
Covid-19 pandemic on Cochrane? Firstly it has created an unprecedented critical
demand for the most up-to-date, reliable evidence to inform frontline clinical
decision-making around the world in order to save as many lives as possible. To
that end, they have been asked by the WHO to produce a precisely scoped,
authoritative register of Covid-19 studies. This essentially inverts their
business model from one based on a wide coverage of clinical questions using
only the highest quality primary evidence to one based on a single clinical
question which urgently extracts every bit of informational value from every
piece of evidence available. That means using all types of RCTs and
interventional studies, as well as observational studies, modelling studies,
diagnostic/prognostic studies and anything else that might improve outcomes. The
surveillance requirement for updates and new evidence switched from months to
days. The cycle time for systematic review production needed to be cut, and a
new Rapid Review format has been developed as a result to ensure baseline
datasets can be pulled together as quickly as possible and then augmented with
streamlined editorial content to meet clinical need. At the same time, the
feedback loop to frontline workers was closed to ensure that their highest
priority clinical questions are the ones being addressed first.

The Covid-19 Study Register [https://covid-19.cochrane.org/] was launched
internally on Monday 30th March and publicly on Monday 6th April. In all, two
back-end platforms were integrated together and a new front-end web application
was designed, built and shipped in just under 3 weeks. It has necessarily been
our fastest ever delivery by some distance, and was only even remotely possible
because of three core enablers:

 1. A data architecture, technical architecture and continuous integration
    infrastructure which minimised the technical risks and impacts of adapting
    to the changing organisational context.
 2. A small number of individually, amazingly talented developers.
 3. The tacit/emergent team structure and pre-existing relationships between
    those developers, established over a long track record of successful
    delivery on other demanding client projects, which meant that as a delivery
    unit they functioned at a truly optimal level from day one.

Since internal launch, our focus has been on getting as much data as possible
into the information model by any and every means we can think of, as quickly as
possible and at the highest possible quality levels. This is covering expert
crowd annotation, machine learning, and other text analysis and informational
reverse engineering techniques. One of the biggest automation/ML challenges
currently is the lack of training data due to the rapid emergence and rate of
evolution of the Covid-19 domain. We have previous experience of this class of
problem from our work with media newsrooms and automated classification of
breaking news stories. It is likely the crisis will drive extensive further
innovation in this space. 

At the same time, we have started receiving comments from stakeholders and end
users across the organisation. It has been some of the most inspiring and
touching feedback we have ever received both as an organisation and as
individuals, especially in light of the wider situation right now. The high
point so far was one very senior Cochrane staff member who, in wonderfully
memorable terms, said he couldn't overstate how useful the register is to all
aspects of his work and that he's now going to shout about it to everyone who
will listen.

A beautiful moment in the midst of demanding times.

Technical Strategy

Covid-19: Rapid Response Technical Delivery at Cochrane

In this post we are going to look at some of the ways in which AI and data
science is impacting the practical delivery of technology solutions today. To be
clear from the outset, this will not include a discussion of future-looking
topics that have been getting coverage recently such as fully automated code
generation or algorithmic product development, etc. Rather, we will be looking
the impacts which are already happening right now, the consequences of which are
often just as significant albeit in different ways.

Strategic Alignment Model
To get a clearer idea of what is going on, it will be helpful to start by
reviewing the dramatic shift that has occurred recently within the discipline of
data engineering. In order to do that, we are going to use a version of Niel
Nickolaisen’s Strategic Alignment Model
[http://www.accelinnova.com/strategy.html], which is one of our most frequently
utilised tools in the inception phases of data programs.

It splits an organisation’s capabilities in two core dimensions:

1.     Whether or not they are market differentiating

2.     Whether or not they are mission critical core competencies

This then informs technology delivery strategy as follows:

·      Systems which deliver business capabilities which are mission-critical
market-differentiators you are going to need to build - as by definition there
will be no commoditised solution available yet otherwise it wouldn’t be a
differentiator - and you are going to need to build them very well whilst
effectively managing risk and uncertainty, as this is where your future revenue
streams are coming from (as a side note, this quadrant represents almost all of
our consulting work).

·      Systems which deliver business capabilities which are mission-critical
but not market-differentiators should be bought in. That’s because they are
generally commoditised solutions to common problems (ERP, payroll, invoicing,
etc), hence re-inventing those solutions is a pointless waste of money. In some
cases, organisations might have non-standard workflows which don’t match
commodity solutions, but in the long run it is almost always better to change
those workflows rather than build bespoke software.

·      Systems which deliver business capabilities which are not
mission-critical but are market-differentiating are generally good candidates
for partnerships, i.e. with businesses for whom the newly developed capabilities
will become core competencies.

·      Systems which deliver business capabilities which are neither
mission-critical nor market-differentiating should generally be avoided, as
their value is as best highly questionable.

Such analysis frequently provides very useful insight to organisations. However,
a critical point to remember is that markets are dynamic, so such
categorisations are never fixed. Indeed, it is interesting to see how
technologies traverse this diagram over time. The most common transition is from
market-differentiator to non-market differentiator as technologies mature: new
features and capabilities that were once enough to attract new customers and
generate additional revenue become widely adopted, turning into commodity
features that are required by any product attempting to enter the market or
protect its existing revenue streams. There are also examples, such as online
video within media organisations, which start out as non-mission critical
differentiators before becoming mission-critical differentiators and then
non-differentiating core competencies (as a side note, the other important shift
as functionality matures from differentiating to non-differentiating is the
associated switch in customer priorities from functional to non-functional
requirements – what matters for non-differentiating systems is
standards-compliance, portability and the avoidance of vendor lock-in, etc).

Data Engineering As Market Differentiator
So what does this have to do with data engineering? Well, the really interesting
thing about the recent history of data engineering is that is represents the
much less common example of a technology moving in the opposite direction –
upwards, from non-differentiating to differentiating:

Data engineering used to be the domain of the back office: of ETLs, PL/SQL
programming and cron jobs. It was one of the areas of software development most
insulated from the market pressures which shaped agile software delivery
techniques and lean product development. It was also one of the least impacted
as a result: rare indeed was it to find an ETL developer writing their code
test-first.

However, the revolutions in firstly data analytics and statistical software
development, and then in machine learning techniques, have radically transformed
the discipline and moved it front and centre in the new era of data-driven
product innovation. This has created a sudden critical demand firstly for access
to these new technologies, a need which has driven the recent adoption and
growth of data science as a commercial capability; and secondly for the software
development skills that allow innovative product ideas to be tested and iterated
rapidly under conditions of uncertainty without incurring crippling amounts of
technical debt.

One of the most prevalent organisational responses to these demands is for
data-driven product delivery programs to employ three discrete technology
functions:

1.     Data science teams: typically, recent PhDs with good theoretical
knowledge but little experience of commercial software development. They often
work in what is effectively an R&D function developing prototype implementations
in their preferred language.

2.     Software delivery teams: where these function best, they are most
typically staffed by experienced/senior developers well versed in extreme
programming and continuous delivery techniques and ideally who have worked
together previously (the importance of this last point is so often overlooked –
no matter how good the individual members are, if the team itself is new then
you are going to burn significant time and cash whilst relationships, trust and
delivery flow are negotiated and established
[https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development]).

3.     Machine learning engineering teams: who are responsible for taking the
outputs of data science and adapting them into something which can be integrated
by software delivery. I use the term “adapting” rather than “refactoring” here
intentionally, as due to the “specialised” nature of the data science algorithms
the aim is normally to add runtime tooling into a software execution wrapper
that allows the prototype implementation to be invoked concurrently or to handle
larger datasets. Most typically this will be suites of map-reduce and derivative
tooling (not least because these teams are also the most natural home for data
engineers who invested in Hadoop stack upskilling during the first data
analytics revolution).

Implications for Software Architecture
This differentiation is having profound, and most highly undesirable, results.
My colleague Fabio Colasanti has previously written persuasively
[https://datalanguage.com/features/data-science-is-about-building-software] 
about the consequences at the code level. In the rest of this article, we will
explore the implications for software architecture, and the resulting impacts on
the business value actually being realised by these programs of work.

Arbitrary Service Boundaries
Conway’s Law [http://wiki.c2.com/?ConwaysLaw] states that the architecture of
systems tends to reflect the architecture of the organisations that build them.
The most reliable way to mitigate this tendency is to create a single
cross-functional delivery team that includes domain experts and works as close
to customers as possible. That’s because the domain model and business
capabilities encoded into software create the channels along which information
flows and business value is then generated and delivered.

In the absence of such an integrated, cross-functional approach, the likely
outcome will instead be an architecture where service boundaries reflect the
arbitrary divisions of labour in the construction of the codebase. Is that
really so bad? Well, yes actually - it is. Once the code sequences required to
implement and integrate the business capabilities are defined, it becomes
immediately apparent that the natural execution flow is being critically
disrupted by meaningless service interfaces that require mapping layers,
additional infrastructure complexity, kill performance, undermine cacheability,
expose security vulnerabilities, and introduce a host of other avoidable
problems.

In doing so, they also undermine the chances of a successful delivery.
Successful technology delivery is ultimately a question of risk management: of
identifying risks and then managing or mitigating them so that their cumulative
impact is as low as possible. That means ruthlessly removing all avoidable
sources of risk and noise - so that the inherent risks of the market (changing
customer demand, competitor behaviour, etc) can then be observed, responded to,
and their associated opportunities exploited.

Self-generating Complexity
In response to the implementation barriers created by arbitrary service
boundaries, delivery teams then typically attempt to work around them via the
lowest friction route available. Rather than conducting a politically
unpalatable root cause analysis and proposing a simplified approach that
conflicts with the structure of the overall delivery program, it is easier to
try using additional software and infrastructure complexity to address the
problem. This results in an escalating feedback loop of bad design: not only is
the architecture unnecessarily complex already, but additional complexity is
then introduced to manage that complexity.

Unfortunately, the only way to break out of this loop is for some brave soul to
put their hand up and conduct a “survival of the fittest” design cull, where
every component of the architecture is ripped out and tested for critical system
viability until what remains is the minimal skeleton actually required to
deliver the baseline functionality.

Machine Learning as Universal Panacea
A more subtle variant of self-generating complexity relates to the application
of machine learning itself. Given the relative immaturity and vast potential of
the field of data science, there is now the additional lurking temptation to try
using ML to work around any problem rather than analysing and solving its root
cause. However, as Robert Frost said “the best way out is always through” – a
phrase which captures the essence of how to avoid incurring crippling technical
debt. Applied to data-driven product innovation, that means conducting patient
and precise analysis to identify underlying causes of problems and solving them
at source rather than building an ever-expanding shanty town of workarounds that
is liable to catastrophic and terminal failure.

Effective Mitigation Strategies
What techniques can be employed to mitigate against these problems and create a
reliable foundation on which to base your delivery? In our experience the two
most valuable tools are a domain model and sets of BDD scenarios (or terse use
cases) which describe the state and behaviour on which the anticipated business
value depends:

Domain Model: this is a collaboratively defined conceptual model of the problem
space shared between the subject matter experts and technical staff. Its key
importance is that it explicitly names the entities of the business domain and
their relationships to each other. In doing so, it creates a worldview and
language that is shared by all stakeholders/members of the delivery team (a
purposefully limited and relativistic scope, which is enough to ensure use cases
can be fulfilled whilst avoiding the many pitfalls and dead-ends of enterprise
data modelling). It is not a knowledge graph, ontology or database schema.
Rather it is something which SMEs might express as a frequently iterated
whiteboard diagram of labelled entities and connectors, and developers might
describe using an ERD or high-level class diagram. This foundation is what
allows code to be written in a way that precisely and unambiguously matches the
collective intent of the team. Without it, you are building on quicksand.

BDD Scenarios: once a shared language has been established which describe the
entities in the domain and how they relate to each other, the state transitions
involved in the information flows and generated business value can then be
captured (in practice this tends to be highly iterative, with new scenarios
being sought that break the model and thereby drive its improvement). The
emphasis here is not on automated testing capability – although that can be a
very helpful by-product - but the communication of state transitions and
workflows [https://lizkeogh.com/2013/10/24/bdd-before-the-tools/]. 

Once the workflows and information flows have been defined via which business
value will be delivered, questions concerning the optimal deployment of human
and machine agents can then be correctly and meaningfully addressed. At this
point, it is then really important to return to the Strategic Alignment Model
for one more pass. Newly equipped with a powerful model that clearly describes
the state/data and behaviour of your domain, you can review each entity and
function in turn to create an unambiguous and precise definition of what your
market differentiators really are. Is it really the bespoke algorithm that your
data scientists are working on, or is it in fact the data that the algorithm is
outputting? If the latter case, is it really justifiable to keep investing in
developing something in-house which ought to be (and will almost certainly end
up eventually) a commodity purchase? These considerations will then be primary
factors informing product roadmap planning – a subject beyond the scope of this
post.

The result should be a system architecture that is the simplest thing possible
[https://www.jamesshore.com/Agile-Book/simple_design.html] to deliver the
identified business value, and as a result is the cheapest to build, contains
the minimum number of things that can go wrong, and is fastest to adapt to
future market change.

AI is a core discipline at Data Language. We have an expert team skilled in machine learning and deep learning, and importantly we also have the engineering and deployment skills to deliver AI at scale for your production systems.

AI is a core discipline at Data Language: Our expert team is skilled in machine learning and the engineering skills needed to deliver AI at scale.

Artificial Intelligence delivered at scale - our core discipline.

Artificial Intelligence

Current impacts of AI on Technology Delivery

Silver Oliver

An introductory guide to tagging: part 4

In the previous post I talked about the challenges of tagging
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites].
Here I will start to look at some approaches that can help.

One technique that I will always come back to is domain-driven design (DDD). The
principles behind domain-driven design go back to Eric Evans and his book 
Domain-Driven Design: Tackling Complexity in the Heart of Software
[https://www.oreilly.com/library/view/domain-driven-design-tackling/0321125215/]
.  The starting point for domain-driven design has similarities to domain
analysis in Library Science. 

In domain analysis Albrechtsen and Hjørland make the case for domain knowledge
being given greater importance in the role it plays in Information Science.

> Many approaches to information science and Knowledge Organisation may be
understood as attempts to pass over subject knowledge (or at least not make
subject knowledge explicit in their methodologies). Domain analysis, on the
other hand, makes subject knowledge an explicit and important part of the
methodologies of information science and knowledge organization.
Towards a new horizon in Information Science
[https://www.researchgate.net/publication/220434297_Toward_a_New_Horizon_in_Information_Science_Domain-Analysis]

DDD, though aimed at software developers, comes to a similar conclusion.  This
is that many software projects fail due to passing over subject knowledge and
not making understanding the domain a central part of the process. The problem
being that everyone on a team goes away and builds according to their private
view of the domain. Then when it comes to putting the design pieces together
they don't fit.

DDD ensures a team spend time together at the outset of a project to align their
understanding.  This is before they design or build anything.  The success of
this approach is that it supplements existing expertise with a strong and shared
knowledge of the domain.

The aligning of language during these sessions is crucial.  Participants talk to
each other, draw for each other and do everything they can to understand where
differences of understanding or terminology are lurking. 

Diagrams and word documents have the habit of hiding our differences as opposed
to surfacing them. True shared understanding is something that can only really
be achieved by talking about the domain from as many angles as possible. 

It is as simple as that. 

When starting on a project to design and build anything, I would suggest the
following principles:

 * Recognise that caring about and understanding the subject domain is
   important.
 * Domain experts are a critical part of the team.
 * Everyone will have different views, preconceptions, simplifications they
   bring to a domain and this manifests itself in the language they use.
 * Learning where these differences are will be expensive if learnt through
   writing code,  building a taxonomy or designing a user interface. Better to
   learn this cheaply in front of a whiteboard.
 * The best way to surface these differences and align language is in talking, a
   lot, together as a team.
 * Simple domain model diagrams are a great way to share and challenge
   understanding of the domain.
   

The main outcome of all this is the shared language during conversations and in
design decisions.  There will likely also be some domain model diagrams.  An
example of this is the classic Polar Bear presentation
[https://www.slideshare.net/reduxd/beyond-the-polar-bear/22-stadiumbr_competitionbr_teambr_matchbr_roundbr] 
by Mike Atherton of a sports website. 


Credit Mike Atherton [https://twitter.com/mikeatherton] and Paul Rissen
[https://twitter.com/r4isstatic] (model design)This represents the project
team's shared understanding of the domain and consensus regarding how to talk
about it. In this model are all the types of tags we might need. As we
understand more about the website's goals and the plan for which pages are to be
built first we can ensure tagging will support them.

For example we might decide we want to capture tags for teams and matches. Why
not stadiums? If we tag with a match we already have the stadium from the model.
 So we can reduce the burden on taggers to that which is absolutely necessary.
We can also see we get competition round from the model as well. Each tag is
part of a larger model and we adjust our tagging strategy based on our
understanding of the model.

Why not tag with player?  Analysis may show that there is little value in
including players. For example we have seen little use of player pages in other
parts of the site. Then there is no justification in tagging them. Some might
argue that we need to preempt any possible use of the system in the future, so
we should tag players even though this doesn't add value now.  In my experience
this is where we get into trouble.  Keeping a tight focus on the value and the
tagging needed to create that value is crucial. The domain model will help us
map out possibilities but an understanding of our goals helps up pick the right
path through that map.

To continue this example, if we identify that tagging with teams and matches
will  support the objectives of the site then we have an idea of what good looks
like. This informs how we develop the tools, vocabularies and guidelines to
support this goal. An understanding of the domain, the websites goals and its
evolving scope have defined how we design our tagging.

That is a simple example but there are a whole range of more complex case
studies to support this from Wellcome
[https://stacks.wellcomecollection.org/building-a-free-and-unrestricted-digital-museum-and-library-6cdf28784068] 
, BBC [https://www.bbc.co.uk/blogs/radiolabs/2009/01/how_we_make_websites.shtml] 
and Parliament
[https://pds.blog.parliament.uk/2016/09/09/modelling-parliaments/].

I often get asked if this goes against user-centred design philosophies? Yes and
no. 

When working at its best DDD is complementary and works hand in hand with user
centred approaches.  At its core is the idea that the domain as a key organising
principle.  This can still sit within a user-first approach.  However, on the
whole I feel a team aligning its understanding and language within a domain
before engaging with users can only be a good thing.

In summary, one part of the solution to better tagging is to design with a
collective understanding of your systems domain. 

Part 1: Why the web turned its back on the librarians. And why we need them
back.
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]

Part 2: The importance of tagging when publishing on the web
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]

Part 3: Why good subject tagging is hard
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites]

We use Domain-Driven Design (DDD) as our fundamental foundation for most projects. The skilled facilitation of domain modelling workshops is used to reach a domain model. This helps teams understand core business entities, relationships, and to target areas of crucial strategic value and corresponding pain points.

Domain-Driven Design is a collaborative process that leads to a Domain Model – a point of reference to inform all strategic data and technology decisions.

Domain-driven Design (DDD) - Creating the map of your business

Domain-driven Design

How to use domain modeling to improve subject tagging

Tolga Ergin

Data Graphs provides effortless knowledge graphs. Set up, manage and build apps on your knowledge graph in one easy-to-use product. Ideal for MVP launches.

Data Graphs - Effortless Knowledge Graphs

Data Graphs provides you with ‘effortless knowledge graphs’. Whether you are in
publishing, agriculture, finance, law, government or any other sector, this is a
foundational knowledge graph tool for your digital transformation.

You can find out more, and sign up for a free trial of Data Graphs at 
https://datagraphs.com

Setup is Easy
Setup takes minutes, and this easy-to-use SaaS product can be shared with your
team to easily integrate with whichever systems, apps and services you use. 

Intuitive User Interface
As a web application, you can import, create, organize, edit, or consolidate
your data effortlessly with its intuitive user experience.

Powerful API
Connect your data via its powerful and fast API, define your business critical
data and enable your team to explore new opportunities, all managed through the
elegant user interface.

The best way to deploy a Knowledge Graph rapidly
From our years of experience building knowledge graphs for global brands, Data
Graphs enables you to get your knowledge graphs launched, build apps, and
connect services in minutes and hours, rather than days and months!

Domain-driven Design

Our Products involved in Domain-driven Design

Our Domain-driven Design work

Domain-driven Design Blog Posts

News about Domain-driven Design