Jim Howard

Get top tips and insights from Jim Howard on valuable use cases for your Sports Data, and avoid the common pitfalls.

Getting the most from your Sports Data - Insights and Best Practice

My name is Jim Howard and I am a freelance Product Manager specialising in
developing products using data and content. After working at the BBC for a
number of years, I have worked on projects for a number of clients including
Government Digital Service, Eurosport and Stats Perform.

I got to know Data Language after working with some of the team on big
publishing and audience-facing Sport products (World Cups, Olympics etc), when I
was at the BBC and The Telegraph.

We have been discussing some of the issues that we have had to deal with over
the years and the opportunities that are still there for companies to make more
of the data that they consume. In this post, I will try and summarise some of
the key areas we have identified and some areas that could still be developed
further.

Sport, data and content in publishing

Now that the restrictions on Sport are gradually being eased, we thought it
would be an opportunity to review some of the areas where Sport-related products
have been developed and business issues have been addressed with a more
innovative approach to the use of data (not analytics) as a core ‘building
block’ to development. 

As always in organisations, there are lots of great ideas around innovation but
the business needs a focus - and sport is an ideal test subject. As we have
found out, you can look at a major sporting event in isolation (Olympics,
Football World Cup etc), review the outcomes, decide on what areas worked and
then integrate into wider product development strategies. 

I will try and break this down into areas that will be relevant to other
products, not just Sport.

Buying in sports data 
First off, you should look at refocusing Sport as a data proposition and not
just a content one.

Most major publishers typically buy in fixtures, results and tables from a
specialist sports data company. Users, and editorial teams expect that.
Depending on the amount and scale of data, this contract is usually paid
annually and is an ongoing cost. So how do you get maximum value from the data
you are buying?

The data usually contains unique IDs for teams, players and competitions, as
well as additional metadata such as dates, times, locations, rounds
(semi-finals, round 13 etc) and venues. These data feeds also contain names of
players, teams and competitions. A whole wealth of information that we could
exploit much better than we were. 

When we started looking in a lot more detail at the data that was being bought,
you could start to see the opportunities to properly join the dots between the
content proposition, the technical challenges, and what we wanted to build as a
Product. 

It wasn’t perfect the first time round we tried this, although we did get to the
stage very quickly of having a process to tag content and create automated pages
for all players, teams and groups at the 2010 World Cup
[https://www.bbc.co.uk/blogs/bbcinternet/2010/07/bbc_world_cup_2010_dynamic_sem.html]
.

We then realised that as Sport is so consistent, once you had a basic domain
model in place, then you could approach development and products in a different
way. 

Working up a Sport domain modelUsing the data to help your tagging 
Workflow is always a challenge, whether you are introducing a whole new system
or just tweaking an existing one. 

From a product perspective, I want consistent and accurately tagged content to
avoid issues with SEO, duplication, URLs and historical inconsistency across
different systems within a business. 

Other business aspects to consider were ensuring we could align the editorial
style guide, improving editorial team efficiency, enabling consistent reporting
of traffic and ad revenue, allow for cleaner syndication of content and at
times, manage tags in multiple languages (Bayern Munich and Bayern München are
the same team), and managing alternate names and synonyms for your tags.

When we first introduced manual tagging of content, we had issues with
consistency and accuracy. Even with guidance, individuals approached relevance
and accuracy in slightly different ways. 

Taking feedback from this, our next iteration was a semi-automated tagging
process using NLP techniques, where we focussed tagging only on headlines and
first 4 or 5 paragraphs of the content. The copy was cross referenced with the
sport vocabulary we had created from the bought in data and tags suggested to
journalists. 

This created a much quicker and accurate workflow for the editorial teams. 

There were unexpected challenges too, such as when a new CMS allowed creation of
multiple tags by content teams, with no editorial oversight on tag quality. The
CMS allowed audience-facing pages to be created automatically once a tag had
been used several times - even if they were incorrect. This caused major issues
for the editorial team, the SEO strategy and User Experience. 

We revisited the key terms around sport that the data supplier had delivered,
used that as a starting point for a controlled vocabulary for the editorial
team, and then used machine intelligence to automatically retag the archive of
existing articles, video and other content in the CMS, to ensure consistency. 

Linking a content model and data 
Another key part of this process is understanding the editorial workflow around
fixtures and events and where a level of automation can help save time and
effort. 

There is a considerable amount of consistency around how events are approached
within newsrooms, with editorial teams creating match previews, live coverage,
reports, photo galleries, post-match analysis etc, all of which fit within a 
content model
[https://www.cmswire.com/content-strategy/content-modeling-what-it-is-and-how-to-get-started/] 
and data strategy approach. This thinking equally applies to video and audio,
live and clips. 

As part of a workflow review we could see that the editorial teams were taking
time to manually create pages in the CMS in the days before a big weekend of
sport. 

We suggested using the bought in fixture list to drive automated creation of
pages in the CMS for journalists. We identified what templates needed creating
around a match - which links nicely with the content model work - and spun-up
pre-tagged templates in the CMS when the fixtures were announced. 

Before a match starts, we already know the teams, competition, venue, historical
records etc.  Depending on the sport, we will also know the athletes taking part
in a match (football tends to announce starting line-ups about 45 mins before
kick off) although other sports make announcements further in advance. 

Silver Oliver’s workshop flipchart diagram of editorial workflowScaling Up
This data-driven approach allows you to scale quickly. New competitions or
events can be approached from a ‘we just need some new data’ perspective and
this should fit with your ongoing data management strategy. 

This is especially relevant around major events where you may want to cover new
teams or events as part of the Olympics, Commonwealth Games or research has
shown you that your audience wants more detailed club level NFL or NRL coverage. 

If you want to create a new type of written content, such as a match report,
that is an extension of your existing content model. 

When you start adding in multiple media types used for coverage such as
photographs and social media, you can begin to see where consistency of tagging
and annotation is key to creating product propositions. 

These methods can also be used for content gathering as well - from external
suppliers to content supplied by correspondents and social media sources. 

Data visualisation
One of the challenges every media business has, is how do you stand out from the
crowd around Sport? Everyone has the same fixture list (that you have no control
over) therefore a focus on the user experience and a unique proposition helps
you stand out. 

Depending on the amount of data that you are buying, there is a lot of mileage
in offering a unique insight, and deeper analysis into the events or
competition. 

I worked on the Roboblogger project
[https://www.niemanlab.org/2016/06/the-telegraph-is-trying-to-streamline-soccer-live-blogging-with-an-automated-graphic-system/]
, part funded by the Google Digital News Innovation fund
[https://newsinitiative.withgoogle.com/dnifund/]. We really dug into the details
of what was in the football data that we bought and pushed the automated
publishing of infographics as far as we could.

It helped that we had already put in place a controlled vocabulary of sport
terms, which made the consistency between systems much easier. 

We ensured that any graphics created fitted with the visual style guide of The
Telegraph and were getting them published in under a few minutes after we
received data about the events. 

It’s not all about the website or app
Once you start digging into how a publishing business works, you can start to
see how progress could be made by thinking about an ‘open data strategy’ within
your organisation and allowing different systems to use the same vocabulary to
make life easier, although that does start changing the business focus to manage
data better - still a very overlooked area at the moment.

There are still multiple opportunities to join the data dots in businesses.
Especially linking back end systems and archiving with audience facing products.

 * Unlocking content archives. Always handy for Olympics, World Cups, and major
   events.
 * Ensuring you can map multiple, external suppliers to a single tag or ID.
 * Advertising and commercial consistency for teams and competitions and even
   individuals athletes.
 * Providing consistency between content and tags after mergers between
   companies.
 * Electronic Programme Guides (EPG).
 * Video tagging - enhancing video with structured data - both your own video
   and inbound social media video
 * Analysis and reporting.

Sport - or even an individual sport such as football - is a very consistent
domain which makes it easier to integrate into your business model, or at the
very least, look at Sport, as your first step towards utilising data and content
in a structured and consistent way.

The M word
I have avoided writing metadata all the way through this so far. 

Data suppliers provide very consistent products and can help you solve some
business and product issues. Linking together data and content to create better
products both for internal processes and audience-facing or syndication
purposes.

For example, you shouldn't have to rely on editorial teams to rebuild a section
or feature. We can do this based on the data we have. If google enables a new
rich snippet type we can add this markup etc. 

Most media organisations buy in content feeds, and although the assets are
tagged, they won’t be quite the same as the vocabulary that you use within your
organisation. Clarifying that vocabulary will save time and money between
systems and your own team.

The key to structured data is a well structured content model, a data management
strategy and metadata.

If you want to extend the model, then you can have a list of your contributors
and journalists as well. This would enable me to create a feed or even a
subscription based-product based on a question such as:

> “Give me all match reports (content type) written by Joe Bloggs (Author) about
Melchester Rovers (team) in the past 6 months (date range)”
Are you making the most of your bought data and content?
Data management is a challenge and unfortunately it does tend to get lumped in
with data science and analytics within a business.

I prefer to look at the reference data as a core building block for the Product.
I want to look at the details of the data contract to see what is being bought.
Does it fit with the current editorial strategy or is it just a historical add
on? Is this what the audience is looking for or expecting? Do they come to your
site or buy your paper for a specific type of coverage?

The cost of the data contract varies from company to company and of course also
depends on the amount of data bought. One thing in my experience that is
consistent though, is that usually only a small amount of the data is being used
by the business.

The coronavirus pandemic has caused a raft of fixture cancellations,
rearrangements, postponements and economic challenges across the board for
industries associated with sport. 

Meeting new business challenges is going to be about agility and speed of
response to new opportunities. 

Companies will have to be more agile in the future and the key to this agility
is having scalable and well marked up content that can be repurposed. 

The future
Hopefully 2021 will be a big year for Sport with a still to be defined weekly
sporting calendar, a rescheduled Olympics in Japan, a rescheduled European
football championships, a Lions Tour to South Africa and lots more. 

Companies will want to make the most of these opportunities for engaging
audiences and hopefully these examples can start some processes we have been
involved in:

 * Advertising and commercial consistency - for teams and competitions and even
   individual athletes
 * Unlocking under utilised content archives
 * Providing consistency across content and data after mergers between
   organisations
 * Electronic programme guides - a key syndication product and internal tool for
   some organisations
 * Video clips and live coverage

Around major events, other departments within your organisation that don’t
usually do sport coverage, may well want to start creating content about the
event such as documentaries etc that you may want to leverage.

Many of these ideas are not just relevant to Sport, the patterns I describe can
be applied to many domains and publishing any form of content, whether that is
in news, sport, entertainment, healthcare, science, or finance. Adding
geospatial metadata into the mix adds another dimension, and more avenues for
adding value.

Every domain has its own challenges but Sport is a very nice place to start. If
you would like to know more, please do get in touch.

Data Language is an industry leader in Information Architecture, ontology design and practical knowledge graph implementation. We use Domain Driven Design practises and user-centred design to focus information and data design on core business needs.

Find Information Architecture news and product innovation updates here, including ontology design, vocabulary management, domain & data modelling and more.

Information Architecture - News, Product Innovation and Technology

Information Architecture

Getting the most from your Sports Data

Paul Wilton

A few weeks ago Cochrane launched their COVID-19 Study Register and I described
the data and technical architecture in detail here
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
.

I described how PICO graphs [https://data.cochrane.org/ontologies/] are used to
augment the Studies with structured linked data that describe the Population,
Intervention, Comparator and Outcome of the trial. Cochrane have now started in
earnest annotating PICO graphs onto the Studies, and these are now being
surfaced in the Study Register application [https://covid-19.cochrane.org/].

Here, we can see the labels of the PICO concepts rendered in the UI, but in the
data itself, these are arranged as a PICO graph structure conforming to the 
PICO
ontology [https://data.cochrane.org/ontologies/pico/index-en.html].

We have also added faceted search by PICO concept into the sidebar. This lets
users quickly filter down their search results using the Cochrane vocabulary
terms.

Once we have a critical mass of PICOs annotated in the register, each vocabulary
concept will be searchable in the search bar at the top of the app, extending
the previously full-text search into a controlled-vocabulary search. A user can
now find studies using a combination of terms chosen from the Cochrane linked
data vocabulary, to deliver extremely accurate results. Moreover we can combine
a metadata driven search with full-text where needed.

Schema.org
Each Study page rendered in the application is also marked up with schema.org
[https://schema.org]structured data. So while internally we are describing our
content and PICO data with the Cochrane bounded context ontologies, at render
time, we are transforming our model to the schema.org MedicalEntity model to
provide machine readable content for search engines and other content consumers.
You can read more about the schema.org medical schema here.
[https://schema.org/docs/meddocs.html]

As an example, for this Studyhttps://covid-19.cochrane.org/studies/crs-13266121 
when viewing the source of the page, you can see we are marking up the clinical
trial as a MedicalStudy in the schema.org ontology, and identifying the 
healthCondition terms, and the interventions, including the Drug Chloroquine.
Each term has associated medical codes mapped to bioinformatics taxonomies
including SNOMED-CT, MeSH, RxNorm and the ATC codes for Drugs where available.
The resulting schema.org JSON-LD for this study is :

{
  "@context": "http://schema.org",
  "@graph": [
    {
      "@type": "MedicalStudy",
      "id": "http://data.cochrane.org/studies/crs-13266121",
      "name": "NCT04333628",
      "identifier": "NCT04333628",
      "description": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
      "mainEntityOfPage": {
        "@type": "WebPage",
        "url": "https://covid-19.cochrane.org/studies/crs-13266121",
        "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial",
        "description": "Clinical study : Chloroquine for Mild Symptomatic and Asymptomatic COVID-19 in A Two Staged, Multicenter, Open Label and Randomized Trial"
      },
      "healthCondition": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        }
      ],
      "studySubject": [
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "url": "https://data.cochrane.org/concepts/NrO30O5ZnYIkjE",
          "name": "COVID-19",
          "alternateName": [
            "COVID-19",
            "Coronavirus disease 2019",
            "2019-nCoV acute respiratory disease",
            "Wuhan flu",
            "HCoV-19",
            "SARS-CoV2",
            "SARS CoV2 infection"
          ]
        },
        {
          "type": "MedicalCondition",
          "id": "http://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "url": "https://data.cochrane.org/concepts/B5JrjqYxGmF8me",
          "name": "Viral load"
        },
        {
          "type": "MedicalProcedure",
          "id": "http://data.cochrane.org/concepts/r4hp5z141f1t",
          "url": "https://data.cochrane.org/concepts/r4hp5z141f1t",
          "name": "Usual Care",
          "alternateName": [
            "Routine Prenatal Care",
            "Routine Care",
            "Standard Care",
            "Providing care according to standard",
            "SOC"
          ],
          "code": [
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "372921003"
            }
          ]
        },
        {
          "type": "Drug",
          "id": "http://data.cochrane.org/concepts/r4hp13k8jxft",
          "url": "https://data.cochrane.org/concepts/r4hp13k8jxft",
          "name": "Chloroquine",
          "alternateName": [
            "Chloroquine",
            "Product containing chloroquine"
          ],
          "code": [
            {
              "codingSystem": "MeSH",
              "codeValue": "D002738"
            },
            {
              "codingSystem": "SNOMED-CT",
              "codeValue": "14728000"
            },
            {
              "codingSystem": "RxNorm",
              "codeValue": "2393"
            },
            {
              "codingSystem": "ATC",
              "codeValue": "P01BA01"
            }
          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6lz",
          "name": "Pharmacological Interventions"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "url": "https://data.cochrane.org/concepts/kn3ptfq7c6m7",
          "name": "Other"
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        },
        {
          "type": "MedicalEntity",
          "id": "http://data.cochrane.org/concepts/q25g9q497cwj",
          "url": "https://data.cochrane.org/concepts/q25g9q497cwj",
          "name": "Physiological or clinical",
          "code": [

          ]
        }
      ],
      "subjectOf": [
        {
          "@type": "MedicalScholarlyArticle",
          "datePublished": "2020-01-01",
          "url": "https://clinicaltrials.gov/show/NCT04333628",
          "headline": "Chloroquine for Mild Symptomatic and Asymptomatic COVID-19",
          "abstract": "19 COVID is a deadly viral disease that has been spreading around the world for several months, and is caused by a CORONA family virus (COVID-19). Following IN-VITRO evidence of the antiviral effect of CHLOROQUINE in CORONA viruses, this drug has been used empirically for COVID-19 patients and is currently recommended in Israel for the treatment of intermediate and severity disease. The mechanism of action of chloroquine is in part by inhibiting the virus distribution, and changing the intracellular acidity, the virus distribution site. The intracellular chloroquine concentration is determined by a pump called PGP that removes the drug from the cell and is activated by the drug. In the treatment of malaria, the benefit of low dosage of the drug has been shown to be effective due to the fact that the intracellular concentration of the drug is probably higher, and therefore the logic to examine this issue in COVID-19 treatment. The purpose of this study is to test whether a low dose of Chloroquine will reduce the duration of the viral shedding and prevent the disease from worsening",
          "author": [
            {
              "type": "Person",
              "name": "HaEmek Medical Center"
            }            
          ]
        }
      ]
    }
  ]
}

PICO and Schema.org
The PICO model and process (https://en.wikipedia.org/wiki/PICO_process) is now
widely adopted for framing research questions in evidence based healthcare with
the BMJ citing this model as best practice
[https://bestpractice.bmj.com/info/toolkit/learn-ebm/how-to-clarify-a-clinical-question/]
.

PICO data is applied by clinicaltrials.gov in a semi-structured tabular format
on most of their published trials and PubMed also has a PICO search tool for
querying MEDLINE, the  database from the US National Library of Medicine.

Supporting PICO would be a welcome addition to the schema.org model, allowing
any publisher of medical literature to mark up their content in a standard way
with machine readable structured PICO data as Cochrane and others are now. I can
imagine a future where Google indexes the schema.org PICO markup and provides
the ultimate PICO based search tool across all the medical literature citations
at their disposal.

At Data Language, as co-authors of the PICO Ontology, we would be more than
happy to help define some simple extensions to the schema.org MedicalEntity
model to make this happen.

PICO, Cochrane and Schema.org

A knowledge graph is a collation of information and data typically within the domain of your business context. The information is stored in a database, often a graph database such that it can be queried, maintained, and interacted with by humans or machines. We have exceptional experience in the design, implementation and deployment of knowledge graphs. We have been working with graph databases since our inception and have delivered knowledge graphs solutions for global organisations.

We have exceptional experience in the design and implementation of knowledge graphs and have delivered knowledge graphs solutions for global organisations.

Knowledge Graphs - our Work, Features, News and Updates

Knowledge Graphs

How knowledge graph technology is helping Cochrane respond to COVID-19

5 latin characters
Until recently, my understanding was that apart from 5 latin characters GraphQL
has got little or nothing to do with Graph Databases. But things are changing,
Graph Database vendors have started bringing these two technologies together. A
recent project has given me some insights.

For some background, GraphQL is a contemporary API interface standard used as an
alternative to REST. GraphQL was originally developed by Facebook. It provides a
structured query language for querying (and mutating) data, along with a runtime
implementation to invoke those queries upon your data. In pretty much all
implementations it uses JSON as its serialization format.

REST (I suspect most readers know this) is a lightweight low-level API standard
for performing uniform and a predefined set of stateless operations on data
aligned with the HTTP verbs. With REST you have to design and  build your API
interface contract for that set of operations - and while of course there are
common patterns in REST APIs, this is very much arbitrary. REST makes no
assumptions about the high level API contract or the underlying data model. The
developer defines both. With GraphQL you only define the data model (the GraphQL
schema), exposing to the GraphQL runtime and your consumers what the shape of
the data is.

With a GraphQL query you can request only the minimum data required to fulfil
your current need. Of course you can do this with a REST API too - in fact it is
a common pattern with JSON REST APIs to filter response data by JSON fields
using dot notation. But of course you need to build this yourself.

GraphQL does a lot more for you to boot, as well as out of the box query
patterns for mutating data, it also supports subscriptions (pub/sub style) - so
your GraphQL client can subscribe to server side events, and have data pushed to
it.

The developer ecosystem around GraphQL is superb now - The Apollo GraphQL
[https://www.apollographql.com] implementation works beautifully with React -
integrating with React Hooks for query and subscriptions. Apollo also has
in-browser developer tools akin to the redux-devtools for GraphQL debugging, and
you can manage local app state via Apollo using the GraphQL schemas / queries
etc.

Caching and performance
Requests to GraphQL are typically all POST requests - not great for caching you
ask ? Indeed, this is where REST has excelled for years. But the GraphQL server
implementations have come to the rescue. The Apollo GraphQL server has an
internal caching layer that supports full response caching, and fine grained
field-level caching (read more here
[https://www.apollographql.com/docs/apollo-server/performance/caching/] )

Amazon’s AWS AppSync is a fully managed GraphQL service built upon the Apollo
technologies - and can source data out of the box from other services such as
DynamoDB, ElasticSearch, REST APIs etc. With fully managed caching and real-time
data subscriptions, it really does make your developer life easy.

On the surface it still appears GraphQL has little to do with Graph Databases.
Yet increasingly we are hearing about how they should go together like bread and
butter, sausage and mash.

Not all graph databases are equal
I won't go into depth about what a Graph Database is, but am going to explore
some of the options available, and the relationship between Graph Databases and
GraphQL.

Graph databases come in several flavour combinations. First up you have RDF
graphs vs Labelled-Property graphs. The first big choice. Both are very
different [https://en.wikipedia.org/wiki/Graph_database#Graph_models]. Your
primary use case is important here. Essentially if you primarily need to walk,
query and analyse paths through your graph, then Labelled-Property graphs are
probably the way to go. If you are more interested in the nature of the
relationships between things, and creating rich semantic representations of
things, then RDF graphs hit the spot. Both types have their own well-developed
query language. Labelled-Property graphs are typically queried with Cypher or
Gremlin (eg Neo4J), RDF graphs are queried with SPARQL.

While most mainstream RDF based graph databases are fully compliant to the RDF
and SPARQL W3C standards, not all RDF graph databases are equal. Each has its
own implementation nuances and come with their own added extras (decisions,
decisions!):

 * An RDF graph database may or may not support reasoning (to some degree of the
   RDFS and OWL standards).
 * Reasoning may be at query time (backward chaining) or at write-time
   (forward-chaining).
 * The way reasoning is implemented can dramatically impact performance.
 * Add-ons and extensions for GeoSPARQL, Path analysis, Full text search,
   connections to other datastores etc.

To give some examples (DISCLAIMER - apols to those vendors not mentioned, I am
only referencing those I have worked with properly hands-on, and this is not an
exhaustive features list - it’s just a flavour) :

GraphDB [https://www.ontotext.com/products/graphdb/] (Ontotext)

 * Fully compliant with RDF/SPARQL standard;
 * Highly compliant and configurable OWL / RDFS forward-chaining reasoning
   (materialises inferred knowledge at write time);
 * Unparallelled full-text-search capability using lucene, or via bi-drectional
   connectors to external SOLR or ElasticSearch clusters.
 *  Ability to materialise post-inference fully structured JSON or JSON-LD
   objects into ElasticSearch indices or MongoDB
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

Neptune [https://aws.amazon.com/neptune/] (Amazon AWS)

 * Fully compliant with RDF/SPARQL standards;
 * No reasoning capability
 * Fully managed cloud service
 * Highly, highly horizontally scalable (primarily due to lack of reasoning)
 * Full text search capability via connector to ElasticSearch - materialisation
   into elastic is constrained (you cannot materialise user-modelled JSON
   objects into an elastic index), but delivers good full-text search utility.
 * You use this either as a Labelled-Property graph (using Gremlin) or an RDF
   graph (with SPARQL)
 * No query workbench/admin UI - you need to install your own 

Stardog [https://stardog.com]

 * Fully compliant with RDF/SPARQL standards;
 * Highly compliant and configurable OWL / RDFS backward-chaining reasoning
   (materialises inferred knowledge at query time);
 * Extremely fast online write times (due to backward-chaining model)
 * Adequate full-text-search capability using internal indexing
 * An outstanding SPARQL extension for PATH analysis (Pathfinder) - you can
   query for paths, shortest paths with weighted edges, normally only possible
   with labelled-property graphs
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

So you can see that each of these three are fully W3C compliant RDF / SPARQL
graph databases. But each has one or more killer features, that the others do
not. 

GraphDB - the post-inference connectors to ElasticSearch and MongoDB unlock some
outstanding technical and data architecture patterns out of the box.

Stardog - combines the semantics of RDF with sophisticated Path querying and
analysis.

Neptune - fully managed cloud service with unparalleled horizontal scalability,
but no reasoning.

The upshot - choose wisely based on your requirements now, and what is likely to
matter most in the medium term.

..and then enter stage right, GraphQL

Sausage and Mash or Pineapple on Pizza ?
How do GraphQL and Graph Databases fit together? As we learned earlier, GraphQL
is a REST API alternative with a structured query language combined with a
runtime for query processing and serving data to GraphQL clients (and it is
almost always implemented with JSON in mind).

So why are Graph Database vendors getting involved and what do you get for your
money?

The cynical marketer in me says those 5 latin characters definitely count for
something. Which sales manager wouldn’t love the potential energy pressure in
the ready-to-explode singularity of Graph Databases and GraphQL (while humming
‘Don’t stop me now’).

However, GraphQL does only provide another query mechanism for your graph
database alongside SPARQL or Gremlin.  Moreover it is much less expressive, and
most definitely not designed for exploiting the rich semantics of RDF and OWL. 

In fact if you go all in on a michelin-starred Graph Database, and only use
GraphQL to query it, I would argue you really probably don’t need a Graph
Database - a NOSQL JSON store would probably suffice. Of course, if you have
forward-chaining inference in your graph database (as with GraphDB), then your
GraphQL queries will benefit from the RDFS/OWL semantics you have modelled your
data with, which does add weight to the value-proposition.

Still hungry ?
Ontologies are an important part of the picture. In the RDF world an ontology
semantically describes the model and nature of your domain, and ultimately the
data structures stored in your knowledge graphs, and if inference-aware, they
describe the meaning of the data, and how we reason upon it. Typically these
models are communicated up and down our tech stack (and the responsible
developers), and often all the way out to consumers of your APIs (public or
enterprise-internal).  While we typically move data around using JSON the
linked-data aficionados will know well that if we do this using JSON-LD, we can
use the same ontology models from the knowledge graph (or a subset) for
describing the JSON-LD payloads we ship around outside of our graph database.

So do we need GraphQL to take advantage of these ontological graph models ?.  No
we don’t, we can do it with normal REST APIs (querying our graph with SPARQL)
and using JSON-LD payloads - but of course we have to build more API code to do
this.

With GraphQL though we can (we must) formally describe our data structures as
GraphQL schemas, and so we have a great opportunity here to align our GraphQL
schemas with those same ontology models that describe the data in our knowledge
graph -  this could be either a subset of our ontology landscape, or possibly
the superset, if we are using other GraphQL (served) data sources aside from the
data in the graph database. This is starting to look like very nice decoupled
data architecture with well structured cohesive data models, with possibly the
most efficient mechanism for consuming data in our web/mobile applications (a
GraphQL client).

It is looking more tasty
Adding GraphQL client interfaces to a Graph Database does make life easier - if
you are rapidly prototyping straight from the graph, or federating data into an
external GraphQL server there is a nice pattern for making a common set of well
described data models.

The Graph Database vendors have indeed jumped on-board the GraphQL sushi train
(and it’s not all about those 5 latin characters). Stardog and GraphDB (from
Ontotext) both offer GraphQL clients now.

At this point I was going to show an example of how GraphQL might be used to
query a Graph Database, but Jem Rayfield at Ontotext has already done the job in
his superb Star Wars blog post
[https://www.ontotext.com/blog/the-rise-of-the-knowledge-graph/] (if you have
got this far, then this is a must read).

Ontotext has taken the next logical step and created a ‘Semantic Object Service’
 that automatically generates GraphQL schemas bound to the underlying RDF data
model  in GraphDB, moving us towards the ontology ubiquity I described above,
where your domain ontologies are expressed in the knowledge graph, and across
your wider data-estate, including your GraphQL schemas. Nice work.

Jem concludes by saying :

> GraphQL is a very popular graph query language, well used, known and liked by
developers. The Ontotext Platform uses GraphQL to lower the barrier of entry to
knowledge graph data, whilst still providing the full expressivity and power of
SPARQL.
and this is a key message, which equally applies to Stardog and any other graph
database that may yet offer a GraphQL client. While GraphQL lowers the barrier
to entry, GraphQL gives you only as much as it is able to with respect to
querying and graph traversal;  to really unlock the power of the graph database,
you will need to use SPARQL.

But if you never plan to, and you are not interested in reasoning either, then
possibly you should ask yourself if you really need a graph database ?

What if you want to use GraphQL as the primary API technology in your stack, but
want to use SPARQL too to access the graph database (amongst other tech)?

A mixed data technology case study
In a very recent engagement, the Data Language team have been working on a Smart
Cities IoT project. The nature of the domain meant the type of data we needed to
deal with was very ‘graphy’ in nature, and the ability to describe our data
semantically and traverse a graph with SPARQL was of high utility. Inference was
less of a concern. The proof of concept for the project was brought together in
AWS with :

 * AWS MSK - a high performance Kafka message bus to consume, orchestrate, and
   distribute IoT sensor events
 * AWS Neptune to give us our scalable RDF based smart city knowledge graph
 * A set of JSON-LD REST API endpoints invoking SPARQL queries on the graph
   running in Kubernetes (EKS)
 * A suite of Kafka event processor consumers and producers deployed into
   Kubernetes mutating the graph, and emitting new events based on our business
   logic rules.
 * AWS AppSync - the GraphQL engine sourcing events and the JSON-LD REST APIs
   from Neptune, providing the primary developer interface to our client
   applications
 * A ReactJS built UI using GraphQL react-hooks to subscribe to real time events
   data, and to query data from the sensors and our knowledge graph (ultimately
   via expressive SPARQL queries).

A simplified view of the architecture looks essentially like this :


The kicker was modeling data across the entire stack using a suite of
ontologies. These ontology models describe our knowledge graph data, our IoT
message payloads in JSON-LD and were all federated in the primary application
API expressed as GraphQL schemas in AppSync. Building the ReactJS front end
using the Apollo GraphQL client and react-hooks was a very nice developer
experience.

Dessert
While GraphQL technically has little to do with Graph Databases, they are not a
perfect match like sausages and mash, GraphQL is just another API / query
technology, and not as expressive as the native graph database query languages
like SPARQL, but in itself is a high-utlity, very sweet technology.

There are some clear synergies, and do provide some of that warm comfort food
feeling when used together. GraphQL does give you an easily understood and very
developer friendly way into a graph database. Once you add the secret sauce of
domain ontologies describing both your knowledge graph and possibly even your
entire data-estate, expressed through your GraphQL schemas, then you have a very
nice pattern for a decoupled data architecture with well structured, nicely
communicated data models across your technology stack, and out to your consumers
too - making for efficient application development using the GraphQL ecosystem.
Get stuck in.

Graph databases

GraphQL and Graph Databases

Julian Everett

In this post we are going to look at some of the ways in which AI and data
science is impacting the practical delivery of technology solutions today. To be
clear from the outset, this will not include a discussion of future-looking
topics that have been getting coverage recently such as fully automated code
generation or algorithmic product development, etc. Rather, we will be looking
the impacts which are already happening right now, the consequences of which are
often just as significant albeit in different ways.

Strategic Alignment Model
To get a clearer idea of what is going on, it will be helpful to start by
reviewing the dramatic shift that has occurred recently within the discipline of
data engineering. In order to do that, we are going to use a version of Niel
Nickolaisen’s Strategic Alignment Model
[http://www.accelinnova.com/strategy.html], which is one of our most frequently
utilised tools in the inception phases of data programs.

It splits an organisation’s capabilities in two core dimensions:

1.     Whether or not they are market differentiating

2.     Whether or not they are mission critical core competencies

This then informs technology delivery strategy as follows:

·      Systems which deliver business capabilities which are mission-critical
market-differentiators you are going to need to build - as by definition there
will be no commoditised solution available yet otherwise it wouldn’t be a
differentiator - and you are going to need to build them very well whilst
effectively managing risk and uncertainty, as this is where your future revenue
streams are coming from (as a side note, this quadrant represents almost all of
our consulting work).

·      Systems which deliver business capabilities which are mission-critical
but not market-differentiators should be bought in. That’s because they are
generally commoditised solutions to common problems (ERP, payroll, invoicing,
etc), hence re-inventing those solutions is a pointless waste of money. In some
cases, organisations might have non-standard workflows which don’t match
commodity solutions, but in the long run it is almost always better to change
those workflows rather than build bespoke software.

·      Systems which deliver business capabilities which are not
mission-critical but are market-differentiating are generally good candidates
for partnerships, i.e. with businesses for whom the newly developed capabilities
will become core competencies.

·      Systems which deliver business capabilities which are neither
mission-critical nor market-differentiating should generally be avoided, as
their value is as best highly questionable.

Such analysis frequently provides very useful insight to organisations. However,
a critical point to remember is that markets are dynamic, so such
categorisations are never fixed. Indeed, it is interesting to see how
technologies traverse this diagram over time. The most common transition is from
market-differentiator to non-market differentiator as technologies mature: new
features and capabilities that were once enough to attract new customers and
generate additional revenue become widely adopted, turning into commodity
features that are required by any product attempting to enter the market or
protect its existing revenue streams. There are also examples, such as online
video within media organisations, which start out as non-mission critical
differentiators before becoming mission-critical differentiators and then
non-differentiating core competencies (as a side note, the other important shift
as functionality matures from differentiating to non-differentiating is the
associated switch in customer priorities from functional to non-functional
requirements – what matters for non-differentiating systems is
standards-compliance, portability and the avoidance of vendor lock-in, etc).

Data Engineering As Market Differentiator
So what does this have to do with data engineering? Well, the really interesting
thing about the recent history of data engineering is that is represents the
much less common example of a technology moving in the opposite direction –
upwards, from non-differentiating to differentiating:

Data engineering used to be the domain of the back office: of ETLs, PL/SQL
programming and cron jobs. It was one of the areas of software development most
insulated from the market pressures which shaped agile software delivery
techniques and lean product development. It was also one of the least impacted
as a result: rare indeed was it to find an ETL developer writing their code
test-first.

However, the revolutions in firstly data analytics and statistical software
development, and then in machine learning techniques, have radically transformed
the discipline and moved it front and centre in the new era of data-driven
product innovation. This has created a sudden critical demand firstly for access
to these new technologies, a need which has driven the recent adoption and
growth of data science as a commercial capability; and secondly for the software
development skills that allow innovative product ideas to be tested and iterated
rapidly under conditions of uncertainty without incurring crippling amounts of
technical debt.

One of the most prevalent organisational responses to these demands is for
data-driven product delivery programs to employ three discrete technology
functions:

1.     Data science teams: typically, recent PhDs with good theoretical
knowledge but little experience of commercial software development. They often
work in what is effectively an R&D function developing prototype implementations
in their preferred language.

2.     Software delivery teams: where these function best, they are most
typically staffed by experienced/senior developers well versed in extreme
programming and continuous delivery techniques and ideally who have worked
together previously (the importance of this last point is so often overlooked –
no matter how good the individual members are, if the team itself is new then
you are going to burn significant time and cash whilst relationships, trust and
delivery flow are negotiated and established
[https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development]).

3.     Machine learning engineering teams: who are responsible for taking the
outputs of data science and adapting them into something which can be integrated
by software delivery. I use the term “adapting” rather than “refactoring” here
intentionally, as due to the “specialised” nature of the data science algorithms
the aim is normally to add runtime tooling into a software execution wrapper
that allows the prototype implementation to be invoked concurrently or to handle
larger datasets. Most typically this will be suites of map-reduce and derivative
tooling (not least because these teams are also the most natural home for data
engineers who invested in Hadoop stack upskilling during the first data
analytics revolution).

Implications for Software Architecture
This differentiation is having profound, and most highly undesirable, results.
My colleague Fabio Colasanti has previously written persuasively
[https://datalanguage.com/features/data-science-is-about-building-software] 
about the consequences at the code level. In the rest of this article, we will
explore the implications for software architecture, and the resulting impacts on
the business value actually being realised by these programs of work.

Arbitrary Service Boundaries
Conway’s Law [http://wiki.c2.com/?ConwaysLaw] states that the architecture of
systems tends to reflect the architecture of the organisations that build them.
The most reliable way to mitigate this tendency is to create a single
cross-functional delivery team that includes domain experts and works as close
to customers as possible. That’s because the domain model and business
capabilities encoded into software create the channels along which information
flows and business value is then generated and delivered.

In the absence of such an integrated, cross-functional approach, the likely
outcome will instead be an architecture where service boundaries reflect the
arbitrary divisions of labour in the construction of the codebase. Is that
really so bad? Well, yes actually - it is. Once the code sequences required to
implement and integrate the business capabilities are defined, it becomes
immediately apparent that the natural execution flow is being critically
disrupted by meaningless service interfaces that require mapping layers,
additional infrastructure complexity, kill performance, undermine cacheability,
expose security vulnerabilities, and introduce a host of other avoidable
problems.

In doing so, they also undermine the chances of a successful delivery.
Successful technology delivery is ultimately a question of risk management: of
identifying risks and then managing or mitigating them so that their cumulative
impact is as low as possible. That means ruthlessly removing all avoidable
sources of risk and noise - so that the inherent risks of the market (changing
customer demand, competitor behaviour, etc) can then be observed, responded to,
and their associated opportunities exploited.

Self-generating Complexity
In response to the implementation barriers created by arbitrary service
boundaries, delivery teams then typically attempt to work around them via the
lowest friction route available. Rather than conducting a politically
unpalatable root cause analysis and proposing a simplified approach that
conflicts with the structure of the overall delivery program, it is easier to
try using additional software and infrastructure complexity to address the
problem. This results in an escalating feedback loop of bad design: not only is
the architecture unnecessarily complex already, but additional complexity is
then introduced to manage that complexity.

Unfortunately, the only way to break out of this loop is for some brave soul to
put their hand up and conduct a “survival of the fittest” design cull, where
every component of the architecture is ripped out and tested for critical system
viability until what remains is the minimal skeleton actually required to
deliver the baseline functionality.

Machine Learning as Universal Panacea
A more subtle variant of self-generating complexity relates to the application
of machine learning itself. Given the relative immaturity and vast potential of
the field of data science, there is now the additional lurking temptation to try
using ML to work around any problem rather than analysing and solving its root
cause. However, as Robert Frost said “the best way out is always through” – a
phrase which captures the essence of how to avoid incurring crippling technical
debt. Applied to data-driven product innovation, that means conducting patient
and precise analysis to identify underlying causes of problems and solving them
at source rather than building an ever-expanding shanty town of workarounds that
is liable to catastrophic and terminal failure.

Effective Mitigation Strategies
What techniques can be employed to mitigate against these problems and create a
reliable foundation on which to base your delivery? In our experience the two
most valuable tools are a domain model and sets of BDD scenarios (or terse use
cases) which describe the state and behaviour on which the anticipated business
value depends:

Domain Model: this is a collaboratively defined conceptual model of the problem
space shared between the subject matter experts and technical staff. Its key
importance is that it explicitly names the entities of the business domain and
their relationships to each other. In doing so, it creates a worldview and
language that is shared by all stakeholders/members of the delivery team (a
purposefully limited and relativistic scope, which is enough to ensure use cases
can be fulfilled whilst avoiding the many pitfalls and dead-ends of enterprise
data modelling). It is not a knowledge graph, ontology or database schema.
Rather it is something which SMEs might express as a frequently iterated
whiteboard diagram of labelled entities and connectors, and developers might
describe using an ERD or high-level class diagram. This foundation is what
allows code to be written in a way that precisely and unambiguously matches the
collective intent of the team. Without it, you are building on quicksand.

BDD Scenarios: once a shared language has been established which describe the
entities in the domain and how they relate to each other, the state transitions
involved in the information flows and generated business value can then be
captured (in practice this tends to be highly iterative, with new scenarios
being sought that break the model and thereby drive its improvement). The
emphasis here is not on automated testing capability – although that can be a
very helpful by-product - but the communication of state transitions and
workflows [https://lizkeogh.com/2013/10/24/bdd-before-the-tools/]. 

Once the workflows and information flows have been defined via which business
value will be delivered, questions concerning the optimal deployment of human
and machine agents can then be correctly and meaningfully addressed. At this
point, it is then really important to return to the Strategic Alignment Model
for one more pass. Newly equipped with a powerful model that clearly describes
the state/data and behaviour of your domain, you can review each entity and
function in turn to create an unambiguous and precise definition of what your
market differentiators really are. Is it really the bespoke algorithm that your
data scientists are working on, or is it in fact the data that the algorithm is
outputting? If the latter case, is it really justifiable to keep investing in
developing something in-house which ought to be (and will almost certainly end
up eventually) a commodity purchase? These considerations will then be primary
factors informing product roadmap planning – a subject beyond the scope of this
post.

The result should be a system architecture that is the simplest thing possible
[https://www.jamesshore.com/Agile-Book/simple_design.html] to deliver the
identified business value, and as a result is the cheapest to build, contains
the minimum number of things that can go wrong, and is fastest to adapt to
future market change.

AI is a core discipline at Data Language. We have an expert team skilled in machine learning and deep learning, and importantly we also have the engineering and deployment skills to deliver AI at scale for your production systems.

AI is a core discipline at Data Language: Our expert team is skilled in machine learning and the engineering skills needed to deliver AI at scale.

Artificial Intelligence delivered at scale - our core discipline.

Artificial Intelligence

Current impacts of AI on Technology Delivery

Silver Oliver

An introductory guide to tagging: part 4

In the previous post I talked about the challenges of tagging
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites].
Here I will start to look at some approaches that can help.

One technique that I will always come back to is domain-driven design (DDD). The
principles behind domain-driven design go back to Eric Evans and his book 
Domain-Driven Design: Tackling Complexity in the Heart of Software
[https://www.oreilly.com/library/view/domain-driven-design-tackling/0321125215/]
.  The starting point for domain-driven design has similarities to domain
analysis in Library Science. 

In domain analysis Albrechtsen and Hjørland make the case for domain knowledge
being given greater importance in the role it plays in Information Science.

> Many approaches to information science and Knowledge Organisation may be
understood as attempts to pass over subject knowledge (or at least not make
subject knowledge explicit in their methodologies). Domain analysis, on the
other hand, makes subject knowledge an explicit and important part of the
methodologies of information science and knowledge organization.
Towards a new horizon in Information Science
[https://www.researchgate.net/publication/220434297_Toward_a_New_Horizon_in_Information_Science_Domain-Analysis]

DDD, though aimed at software developers, comes to a similar conclusion.  This
is that many software projects fail due to passing over subject knowledge and
not making understanding the domain a central part of the process. The problem
being that everyone on a team goes away and builds according to their private
view of the domain. Then when it comes to putting the design pieces together
they don't fit.

DDD ensures a team spend time together at the outset of a project to align their
understanding.  This is before they design or build anything.  The success of
this approach is that it supplements existing expertise with a strong and shared
knowledge of the domain.

The aligning of language during these sessions is crucial.  Participants talk to
each other, draw for each other and do everything they can to understand where
differences of understanding or terminology are lurking. 

Diagrams and word documents have the habit of hiding our differences as opposed
to surfacing them. True shared understanding is something that can only really
be achieved by talking about the domain from as many angles as possible. 

It is as simple as that. 

When starting on a project to design and build anything, I would suggest the
following principles:

 * Recognise that caring about and understanding the subject domain is
   important.
 * Domain experts are a critical part of the team.
 * Everyone will have different views, preconceptions, simplifications they
   bring to a domain and this manifests itself in the language they use.
 * Learning where these differences are will be expensive if learnt through
   writing code,  building a taxonomy or designing a user interface. Better to
   learn this cheaply in front of a whiteboard.
 * The best way to surface these differences and align language is in talking, a
   lot, together as a team.
 * Simple domain model diagrams are a great way to share and challenge
   understanding of the domain.
   

The main outcome of all this is the shared language during conversations and in
design decisions.  There will likely also be some domain model diagrams.  An
example of this is the classic Polar Bear presentation
[https://www.slideshare.net/reduxd/beyond-the-polar-bear/22-stadiumbr_competitionbr_teambr_matchbr_roundbr] 
by Mike Atherton of a sports website. 


Credit Mike Atherton [https://twitter.com/mikeatherton] and Paul Rissen
[https://twitter.com/r4isstatic] (model design)This represents the project
team's shared understanding of the domain and consensus regarding how to talk
about it. In this model are all the types of tags we might need. As we
understand more about the website's goals and the plan for which pages are to be
built first we can ensure tagging will support them.

For example we might decide we want to capture tags for teams and matches. Why
not stadiums? If we tag with a match we already have the stadium from the model.
 So we can reduce the burden on taggers to that which is absolutely necessary.
We can also see we get competition round from the model as well. Each tag is
part of a larger model and we adjust our tagging strategy based on our
understanding of the model.

Why not tag with player?  Analysis may show that there is little value in
including players. For example we have seen little use of player pages in other
parts of the site. Then there is no justification in tagging them. Some might
argue that we need to preempt any possible use of the system in the future, so
we should tag players even though this doesn't add value now.  In my experience
this is where we get into trouble.  Keeping a tight focus on the value and the
tagging needed to create that value is crucial. The domain model will help us
map out possibilities but an understanding of our goals helps up pick the right
path through that map.

To continue this example, if we identify that tagging with teams and matches
will  support the objectives of the site then we have an idea of what good looks
like. This informs how we develop the tools, vocabularies and guidelines to
support this goal. An understanding of the domain, the websites goals and its
evolving scope have defined how we design our tagging.

That is a simple example but there are a whole range of more complex case
studies to support this from Wellcome
[https://stacks.wellcomecollection.org/building-a-free-and-unrestricted-digital-museum-and-library-6cdf28784068] 
, BBC [https://www.bbc.co.uk/blogs/radiolabs/2009/01/how_we_make_websites.shtml] 
and Parliament
[https://pds.blog.parliament.uk/2016/09/09/modelling-parliaments/].

I often get asked if this goes against user-centred design philosophies? Yes and
no. 

When working at its best DDD is complementary and works hand in hand with user
centred approaches.  At its core is the idea that the domain as a key organising
principle.  This can still sit within a user-first approach.  However, on the
whole I feel a team aligning its understanding and language within a domain
before engaging with users can only be a good thing.

In summary, one part of the solution to better tagging is to design with a
collective understanding of your systems domain. 

Part 1: Why the web turned its back on the librarians. And why we need them
back.
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]

Part 2: The importance of tagging when publishing on the web
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]

Part 3: Why good subject tagging is hard
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites]

We use Domain-Driven Design (DDD) as our fundamental foundation for most projects. The skilled facilitation of domain modelling workshops is used to reach a domain model. This helps teams understand core business entities, relationships, and to target areas of crucial strategic value and corresponding pain points.

Domain-Driven Design is a collaborative process that leads to a Domain Model – a point of reference to inform all strategic data and technology decisions.

Domain-driven Design (DDD) - Creating the map of your business

Domain-driven Design

How to use domain modeling to improve subject tagging

An introductory guide to tagging: Part 3

It has been my experience that many organisations struggle with high quality
tagging. In this post I am going to reflect on some of the causes of this
problem. 

As discussed in earlier posts, the value of tagging is often not properly
understood
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]
.  Most often it is considered to be there to support keyword search alone.
 Even the taggers themselves are not aware of how the tags are going to be used,
or why they need to be of good quality to add value.

The next issue is around the guidelines given to taggers.  Typically they are
extremely poor, and go along the lines of “tag what the content is about”.  This
question of “what is it about?” is harder than it seems at first glance and not
very helpful. It usually results in a different answer from each tagger.  To get
good quality tagging you need to ask good quality questions.

As the tagging quality goes down, the ability to drive useful features from the
tags also diminishes. This in turn does nothing to reassure people of the
importance of good tags and you end up in a vicious cycle of ever decreasing
quality and value. 

What good guidance on tagging looks like should be the point where Library
Science can come to the rescue.  But the challenge of how to identify the
subject of a work is a surprisingly under represented topic in Library Science
literature. 

Back in the the 1970s, Hutchins observed:

> THE LITERATURE OF INDEXING AND CLASSIFICATION contains remarkably little
discussion of the processes of indexing and classifying...we find very little
about how indexers and classifiers decide what the subject of a document is, how
they decide what it is 'about'.
The concept of 'aboutness' in subject indexing:
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.584.8116&rep=rep1&type=pdf] 
 W. J. Hutchins

Little has changed.  Taggers are often told to imagine "how might the user
search for this piece of content" or "take the author's view and characterise
what they are trying to say".  Neither is very useful as a framing question. 

Hutchins, in his paper, tackles this problem and introduces a distinction
between summarisation and aboutness. Summarisation is the representation of the
total subject content of a document and the statements made therein. While
aboutness is about capturing the thing that has been written about.

A very simple example might be a sport story about a player and whether they
will move clubs, their struggles with injury and their performance this year.
Many arguments are developed, making up the narrative of the story.
Summarisation would capture not only the player as a tag but also attempt to
capture something about the claims made about the subject of fitness,
performance and likely future at a club.  On the other hand, aboutness only
attempts to capture the thing being talked about and not what claims are made
about them. In this case just the player.

This distinction is rarely made in tagging guidelines and the tagger is left to
balance the level of summarisation and aboutness on gut feel.

Hutchins goes on to characterise this distinction in another interesting way.
The aboutness represents those things that are expected to already be part of
the readers knowledge. Every piece of content assumes a level of knowledge of
the reader with which the narrative then builds. Summarisation on top of this
attempts to capture what is new and builds on that existing knowledge in the
narrative of the content.  The readers are familiar with the player, but not
about the subtle changes in his fitness and future prospects.

With this in mind, the questions we might ask a tagger to answer are: What is
the thing being talked about? What claims or facts are made about a thing in the
content? What might the reader be presumed to know before reading this?

Whether we choose to do summarisation or capture aboutness is a decision related
to the desired outcome of the system. But we should be clear about which one it
is.

A more recent analysis by Birger Hjorland (The concept of subject in
Information
Science [https://www.emeraldinsight.com/doi/abs/10.1108/eb026895]) revisits the
problem.  He cites many examples of Guidelines expecting taggers to do little
more than perform "mind reading" on prospective future consumers of the content.

Instead, he argues, we must think about tags as a tool in enabling the full
potential of the document within the context of the overarching system.  The
desired outcomes of this system should then dictate the tagging strategies and
guidelines. 

It is this domain and system that must be crystal clear in taggers' minds as
opposed to the idea of aligning subject to a theoretical searcher.  They need to
also understand the value that they are creating and how.

Sadly, this is the opposite of what we see in many tagging tools and guidelines
today.

I would argue that tagging should be a contract between taggers and the system
 much in the way we see in the design of data services
[https://medium.com/@diogo.lucas/https-medium-com-diogo-lucas-contract-breaks-are-kinda-of-deal-breakers-447762de20c4]
.  Tagging guidance needs to be managed with a vision of the desired outcomes
combined with asking clear and simple questions of the taggers. This is as true
for humans as machines.

In summary, tagging outside of libraries and archives is often of poor quality.
Value is rarely understood, content creators are often asked to tag in ways that
are a chore and cognitively awkward. Worst of all, the question of what good
tagging looks like has rarely been asked.

In the next post I will look at some potential solutions to the challenge of
better tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]
.

Part 1: Why the web turned its back on the librarians. And why we need them
back.
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]

Part 2: The importance of tagging when publishing on the web
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]

Part 4: How to use domain modeling to improve subject tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]

Why good subject tagging is hard

An introductory guide to tagging: part 2

In the previous post we made the case for why organisations need to rediscover
fundamental library science skills
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]
.

Since the 1990s, library science has been written off by many as a relic of the
dark ages. The perception has been that the gap could be filled by search
engines, machine tagging or folksonomy
[http://shirky.com/writings/herecomeseverybody/ontology_overrated.html].  
However, increasingly organisations are starting to talk about the future in
terms of Knowledge Graphs and metadata. The skills of the librarians are back in
demand and needed more than ever.

There are a range of skills needed to support a comprehensive information
management strategy, but in this post I'm going to examine one:- Subject tagging
(more accurately subject indexing).

Why tag?
The first challenge is changing the perception about why to tag content. It is
often perceived as being about enabling better free text search.  However, this
is probably the least important outcome of tagging.

In fact there is nothing new here.

Since their inception 200 years ago, subject indexing in card catalogues has
been about more than keyword search. Instead these cards represent the domain of
knowledge to be explored providing many contextual entry points. Concepts (or
headings) are navigated rather than documents themselves.  This age-old context
based navigation was rapidly lost as library systems became computerised and
keyword search dominated.

Karen Coyle describes some of the problems of keyword search based catalogues:

> Once a keyword search is done the resulting items are retrieved without any
correlation to headings. It may not even be clear which headings one would use
to create a useful order. The set of retrieved bibliographic resources from a
single keyword search may not provide a coherent knowledge graph.
Karen Coyle Libraryland we have a problem
[https://kcoyle.blogspot.com/2018/08/libraryland-we-have-problem.html]

The situation still prevails today and results in massive lost opportunities for
organisations as they have unwittingly allowed this context to reside within the
search engines and not in their own publishing systems.

This misunderstanding of the value of subject indexing can mean organisations
miss out on the huge benefits.  Without structured subject metadata you limit
your options when publishing, using analytics and monetising content.

Clearly tagging content comes at a cost.  Is it outweighed by the benefits?  For
organisations spending large sums on producing articles, recording podcasts and
shooting video, the associated tagging and other structured descriptive data
often represents a tiny cost in percentage terms of creating the original asset.
 

This is perfectly illustrated by a push in the music industry for better music
metadata
[https://www.theverge.com/2019/5/29/18531476/music-industry-song-royalties-metadata-credit-problems]
.

> Metadata sounds like one of the smallest, most boring things in music. But as it
turns out, it’s one of the most important, complex, and broken, leaving many
musicians unable to get paid for their work.
Metadata is the biggest little problem plaguing the industry
[https://www.theverge.com/2019/5/29/18531476/music-industry-song-royalties-metadata-credit-problems] 
 

So what value can be derived from better tagging?  This clearly depends on the
specific business model of the organisation.  Some examples would be bringing
readers to your site and getting them to spend more time there. Others are
driven by analytics and user insight. Or it might be about revenue from better
targeted advertising or driving paid subscriptions.

In summary, subject tagging provides opportunities to increase the value and
impact of content and data. In the next post I am going to look at why subject
tagging is not as easy as it seems
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites].

Part 1: Why the web turned its back on the librarians. And why we need them
back.
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]

Part 3: Why good subject tagging is hard
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites]

Part 4: How to use domain modeling to improve subject tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]

The importance of tagging when publishing on the web

An introductory guide to tagging: Part 1

This is the first in a series of short blog posts looking at the relationship
between information management and the web. 

This post looks at how developments in search technology between the late
eighties and today created the perception that good information management
practices were redundant. 

A conversation I had a few years ago with Edmund Weiner of the Oxford English
Dictionary has stayed with me ever since.  We talked about the 1980s and how,
during that decade, some publishers started to structure print for computers. He
recalled how questions began to be asked about what documents were really about
and how to represent that knowledge in a database.   

Then came the advent of search technologies. Edmund reflected on how this
resulted in us throwing away hard-won information management skills in favour of
black box technology we didn’t control:

> The march of progress swept us in a different direction, one that offered great
prizes in a much more immediate way. Our partners at the University of Waterloo
had developed one of the first rapid textual search engines, named Pat. It could
search simply-structured text of any kind almost instantaneously and with
well-tagged text it was miraculously powerful. As was said at the time, we
didn’t need to build the intelligence into the database as it could be built
into the search engine.
Modelling Lexical Structures in the Oxford English Dictionary  by Edmund Weiner
[https://brill.com/view/book/edcoll/9789004361133/BP000002.xml?lang=en]  [2008]

This situation still prevails today.  Intelligence resides within the search
engines and not in an organisation's own publishing systems.

The web followed the same path. Powerful search engines and poorly described
pages. Content would express its subject to bots through search engine optimised
copy and links. The abuse of meta tags with keyword stuffing only helped to give
subject tagging a bad name.

These technology developments removed the incentive to invest in doing your own
information management and many organisations unintentionally outsourced their
business intelligence and value creation to search providers. 

However, industry attention has recently shifted back towards subject metadata,
tagging and knowledge graphs as an area of investment. Google have long since
moved to advocating a more structured approach to publishing on the web using
schema.org to provide a baseline data model and Wikidata to provide a baseline
knowledge graph.

What is the reason for the change in perception and the renewed interest in
information management? Organisations are starting to wake up to the fact they
no longer want to outsource business intelligence to Google and other third
parties. To get the most value out of their content they need the ability to
query and aggregate content themselves.

Refitting the plumbing to tag content and manage controlled vocabularies is
potentially a large and complex investment. But there are principles and ideas
that information scientists have been practicing for years that can help,
regardless of the size of the organisation. These challenges will not be solved
by technology alone. Library skills are needed to fill these gaps in expertise. 

In the next post I am going to look at the  importance of subject tagging
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]
.

NOTE: If you want to learn more about Library Science and its the family of
associated disciplines ISKO UK [https://www.iskouk.org/]  and Taxonomy Bootcamp
[http://www.taxonomybootcamp.com/London/2019/Default.aspx]  is a great place to
start (UK focus). They run a brilliant series of events as well as bringing
together the community of practitioners.

Part 2: The importance of tagging when publishing on the web
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]

Part 3: Why good subject tagging is hard
[https://datalanguage.com/features/why-tagging-tends-to-be-bad-on-websites]

Part 4: How to use domain modeling to improve subject tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]

Why the web turned its back on the librarians. And why we need them back.

In most cases when working with or designing an ontology I like to be able to
visualise the model. When designing an ontology from scratch I typically draw
out the model on a whiteboard or directly into a drawing tool, for example 
LucidChart [http://www.lucidchart.com] (my current favourite) or OmniGraffle.
This approach lets us rapidly iterate the model visually before getting our
hands dirty with OWL and RDF. It also allows us to collaborate on the model in a
much more effective way, and engage domain experts whom might not be familiar
with RDF.

While researching public domain models for a recent project, I explored FIBO
[http://www.edmcouncil.org/financialbusiness], the Financial Industry Business
Ontology. FIBO is a suite of building block ontologies engineered by the 
Enterprise Data Management Council [http://www.edmcouncil.org/] to

> “standardize the language used to precisely define the terms, conditions,
and characteristics of financial instruments; the legal and relationship
structure of business entities; the content and time dimensions of market
data; and the legal obligations and process aspects of corporate actions.”


The FIBO v1.0 ontologies currently consist of a set of foundation models
[https://www.omg.org/spec/EDMC-FIBO/FND/] and a set of models describing the
characteristics of business entities [http://www.omg.org/spec/EDMC-FIBO/BE].
Future FIBO ontologies (yet to be published by EDM) will model the domain of
financial instruments and indexes.

While the FIBO ontologies are well defined and the RDFS/OWL definitions are
available for download, there is little in the way of visualisation for
consumers and developers with respect to how the ontologies work together. So it
made sense for me to create visualisations of the models to aid my own
evaluation. The first models I have visualised are the Person domain model, and
the Organization domain model. In these visualisations the symbology used is as
follows :

 * Rectangles = RDFS Datatypes (FIBO defined and XSD datatypes)
 * Hexagons = Predicates
 * Ovals = Classes

Many of the foundation predicates from the FIBO Relations ontology are specified
without formal RDFS domains and ranges in order to enable re-use. Their usages
within each more specific domain ontology are then defined using local OWL
Restrictions on classes. On the diagrams I have indicated these as virtual
domains and ranges of the predicates, clearly distinct from the explicit
rdfs:domain and rdfs:range predicate definitions that can be reasoned upon.

The name prefixes displayed follow the FIBO prefix convention and reference each
FIBO ontology as follows :

prefixnamespacefibo-fnd-rel-relhttp
://www.omg.org/spec/EDMC-FIBO/FND/Relations/Relations/fibo-fnd-utl-bthttp
://www.omg.org/spec/EDMC-FIBO/FND/Utilities/BusinessFacingTypes/fibo-fnd-aap-agt
http://www.omg.org/spec/EDMC-FIBO/FND/AgentsAndPeople/Agents/fibo-fnd-gao-glhttp
://www.omg.org/spec/EDMC-FIBO/FND/GoalsAndObjectives/Goals/fibo-fnd-plc-ctyhttp
://www.omg.org/spec/EDMC-FIBO/FND/Places/Countries/fibo-fnd-aap-pplhttp
://www.omg.org/spec/EDMC-FIBO/FND/AgentsAndPeople/People/fibo-be-le-lphttp
://www.omg.org/spec/EDMC-FIBO/BE/LegalEntities/LegalPersons/fibo-fnd-pty-ptyhttp
://www.omg.org/spec/EDMC-FIBO/FND/Parties/Parties/fibo-fnd-pty-rlhttp
://www.omg.org/spec/EDMC-FIBO/FND/Parties/Roles/fibo-fnd-org-orghttp
://www.omg.org/spec/EDMC-FIBO/FND/Organizations/Organizations/fibo-fnd-org-fm
http://www.omg.org/spec/EDMC-FIBO/FND/Organizations/FormalOrganizations/
fibo-fnd-org-lghttp
://www.omg.org/spec/EDMC-FIBO/FND/Organizations/LegitimateOrganizations/
fibo-be-le-fbohttp
://www.omg.org/spec/EDMC-FIBO/BE/LegalEntities/FormalBusinessOrganizations/The
first ontology tackled is the Person domain model and is visualised below :

The next ontology visualised is the Organization domain model as follows :

I hope these visualisations are helpful. Both images can be seen in full
resolution if opened in a new browser tab, or downloaded.
In a future post I hope to tackle visualisations for Functional Business
Entities, Contracts, Accounting Equity and the Legal domains once these models
are released.

Information Architecture

Information Architecture Blog Posts