Paul Wilton

A case study on a recent engagement in the publishing industry using Data Graphs
for centrally managing cross-publication content metadata combined with our Text
AI NLP service for auto-tagging content to provide a high-utility information
backbone.

The Snap
Tagging content has been a publishing staple for two decades, helping publishers
organise content, perform rudimentary content analytics, and also deliver tag
based user journeys and content aggregations. In many cases publishers use the
tag features built into their CMSs. WordPress and its derivatives are so widely
used, yet still have a very weak content tagging capability. This lack of good
information management and structured metadata often leads to a tag mess over
time, as tags cannot be properly disambiguated, duplicates invariably get
created, and journalists apply tags inconsistently. Silver Oliver has written an 
excellent series of articles
[https://datalanguage.com/blog/why-the-web-turned-its-back-on-the-librarians] on
this subject that are worth revisiting.

This is where more contemporary information management technology and AI can be
a game changer. I outline below how we have used our own Text AI and Knowledge
Graph products to solve these issues, and drastically improve how structured
metadata is used at a large global news publisher.

The Back Story
In the context of this project, we were not dealing with a single CMS, but
multiple news publications, each with their own publishing technical stack, CMS,
mechanism for tagging, and entirely separate tag sets. The ambition was to
centralize the metadata used to tag on each publication, and use AI to aid the
journalists, essentially raising the consistency level of tagging across news
desks, and streamlining and optimising the tagging process. Downstream, this
would unlock better user journeys for consumers, better content analytics, and
provide for better and more rapid cross-publication innovation.

To achieve this, we integrated:

 1. Our product Data Graphs to manage a large corpus of the cross-publication
    tags, constructed as a light-weight knowledge graph of news concepts,
    including concept types such as People, Locations, Organisations, Events,
    Brands, Sports teams and athletes, Categories and Topics.
 2. Our AI Text Classification service, Tagmatic - which predicts the
    “aboutness” of content by tagging using the concepts managed in Data Graphs,
    continuously learning on-the-fly as new articles are published.

The Storyline
Our two products Data Graphs and the Text Classification AI work seamlessly
together as an ideal solution for an information backbone in the new generation
of publishing technology ecosystems.

We move from the legacy pattern of tagging with flat tags, to tagging with well
modelled and organised structured data, where each concept we tag with has a
globally unique, persistent and durable identifier (a URI / URN), and has its
own metadata properties that can be used to provide more information about that
concept to help users disambiguate (types, thumbnails, images, descriptions,
categories etc) and also provide better user experiences and journeys with these
concept tags.

Managing this metadata centrally outside of your CMS provides a number of
advantages:

 1. Your tags are not tightly coupled to one CMS (with respect to identity,
    structure, access and availability etc) - giving you an easy upgrade path or
    migration to a different CMS in the future
 2. The ability to easily use that metadata in other parts of your publishing
    stack or share with other CMSs. For example, the same metadata can then be
    used to tag or classify source feeds upstream from content authoring such
    that external content and your own content are tagged using the same
    classifications and concepts.
 3. Decouple the management of the metadata from the CMS, such that a librarian
    or information specialist takes on that task, leaving your journalists to do
    what they do best and author content.

To do this, Data Graphs allows the publisher to:

 1. Create a domain model of their core entities (People, Places, Orgs, Sports,
    Events, Topics, Content Types, and any other metadata they need to drive
    their publishing processes and user experiences)  and how they prefer to
    classify and relate these concepts to each other.
 2. Ensure these concepts are modelled with all the structured data you need for
    concept disambiguation, categorisation, and concept driven user journeys.
 3. Easily populate this model with the instances (taxonomies) of this metadata
    (essentially, publishing reference data) using either the Data Graphs API or
    CSV Uploaders in the Data Graphs web app.
 4. Manage that metadata with a dedicated application, with an intuitive user
    experience.
 5. Integrate the metadata with any of your business systems using its APIs and
    Web Hooks, including your CMSs and other data or content stores.

Using machine intelligence to automatically tag concepts and classifications of
your articles, also provides a number of advantages:

 1. It makes the task of tagging and classifying content easy and faster. Many
    journalists don’t like having to tag their articles, and may not understand
    the value of tagging well. Because the AI learns from all journalists, and
    from golden documents, this means the quality level is raised for all
    authors, and consistency is improved across all news desks.
 2. The AI can be used to tag content outside of the CMS, completely offloading
    this task from the journalists - either using a human-in-the-loop pattern
    (where a totally different person/role validates the quality of the metadata
    the AI is applying), or in a fully automated mode, for say long tail content
    where you may not be so concerned about occasional mistakes being made,
    where the cost of human supervision might exceed the value realised.
 3. The AI can be used to tag content upstream from your CMS, tagging external
    feeds (Reuters, AP, Social feeds etc) with the same metadata you apply to
    your own content. This can deliver much more efficient tooling for the
    processing of this large content funnel on your news desks.

The Data Language Text Classification AI (Tagmatic)  is designed for exactly
this task. It is a high-performance scalable service that learns how you
classify and tag your content, and then applies metadata to new content that is
sent to it for prediction. The service is highly scalable, fully private, and
designed to adapt to each publisher’s unique tagging style. Its REST API allows
for very easy integration with your CMS and publishing stack, such that it
learns continuously on-the-fly, reacting quickly to new concepts that arise in
the ever changing window of news.

Data Graphs and Tagmatic work seamlessly together, through a dedicated
connector, such that when a concept is merged or removed in Data Graphs,
Tagmatic automatically adjusts its models’ weights to reflect the new state.

The Explainer
Technically Data Graphs and our Auto-tagging Text AI work together as shown in
the diagram below. In this case study a custom WordPress metadata plugin was
created that integrated with both the Data Graphs and Tagmatic APIs to provide a
human-in-the-loop tagging UI directly in the publisher’s CMS:

Where:

 1. The article is sent to the Tagmatic Predict service API to suggest concepts
    to tag with, and these are presented to the user in the WordPress plugin.
    The classes of the concepts are used to present these in a tagging template
    suitable for the type of content being authored (News, Sport, Politics etc).
    The user can then keep or remove the tags that seem incorrect.
 2. A user can lookup missing or new concepts in Data Graphs via
    typeahead/autocomplete widgets in the WordPress plugin by using the Data
    Graphs Search API
 3. When the article and metadata is saved / published, WordPress sends the
    content and tags back to Tagmatic to its Train API, so it can use this for
    continuous improvement of its prediction models.  This allows it to learn
    brand new tags from their first appearance in  an article precluding the
    need for scheduled long running training cycles, and instead learns
    on-the-fly.
 4. If the librarian or information specialist removes a tag concept in Data
    Graphs, it automatically fires a webhook to the Tagmatic connector, so
    Tagmatic can disable the classifier model for that concept. If a user merges
    two concepts in Data Graphs, the connector will trigger Tagmatic to re-point
    its classifier from the source concept to the target concept of the merge
    operation - and adjust its weights accordingly.

In the more detailed images below you can see how this WordPress metadata plugin
might look, and how a concept is presented to the user with enough information
from Data Graphs to correctly disambiguate the tag.

Once tags are sent back to the training API with the content, Tagmatic makes
tagging and classification metrics available via another service API that
includes the precision, recall, F1 scores, support for each tag, and also the
date the tag was last trained. These metrics can then be used to provide
insights including usage analytics and patterns to the editorial floor.

The Finished Article
Bringing these technologies together creates a high-utility information backbone
for the modern newsroom, streamlining processes, automating others, and
unlocking publishing innovation.

By centralising and decoupling metadata from your CMS, and introducing machine
intelligence, you unlock a number of key capabilities: 

 * Better information and data governance, the importance of which cannot be
   understated
 * The ability to cross-pollinate your reference data across third party content
   and other internal systems, including both content and user analytics.
 * The ability to automate the classification of upstream content feeds with the
   same metadata, which allows you to scale out the funnel of source material
   coming into the news desks, and provide better search and discovery
   capabilities to those journalists.
 * Making the journalists job easier - letting them focus more on authoring good
   content
 * The ability to automate the aggregation of content, for example for topic
   pages, using structured through metadata driven content query APIs
 * Let the CMS do what it does best: the authoring and structuring of the
   content itself

Ultimately, this technology makes tags and concepts management a first class
citizen in the publishing stack. No longer just an afterthought inherited from
and tightly coupled with the CMS of the day - but something that is built to
stay even when the authoring tooling changes, unlocking the true value of your
content metadata.

Publishing Technology is a core discipline at Data Language. Our AI for Auto-tagging content and text classification combined with your product Data Graphs can provide a rich information backbone for the modern newsroom.

Publishing Tech

Case Study: How Data Graphs and AI Auto-Tagging transforms Publishing

Jim Howard

Get top tips and insights from Jim Howard on valuable use cases for your Sports Data, and avoid the common pitfalls.

Getting the most from your Sports Data - Insights and Best Practice

My name is Jim Howard and I am a freelance Product Manager specialising in
developing products using data and content. After working at the BBC for a
number of years, I have worked on projects for a number of clients including
Government Digital Service, Eurosport and Stats Perform.

I got to know Data Language after working with some of the team on big
publishing and audience-facing Sport products (World Cups, Olympics etc), when I
was at the BBC and The Telegraph.

We have been discussing some of the issues that we have had to deal with over
the years and the opportunities that are still there for companies to make more
of the data that they consume. In this post, I will try and summarise some of
the key areas we have identified and some areas that could still be developed
further.

Sport, data and content in publishing

Now that the restrictions on Sport are gradually being eased, we thought it
would be an opportunity to review some of the areas where Sport-related products
have been developed and business issues have been addressed with a more
innovative approach to the use of data (not analytics) as a core ‘building
block’ to development. 

As always in organisations, there are lots of great ideas around innovation but
the business needs a focus - and sport is an ideal test subject. As we have
found out, you can look at a major sporting event in isolation (Olympics,
Football World Cup etc), review the outcomes, decide on what areas worked and
then integrate into wider product development strategies. 

I will try and break this down into areas that will be relevant to other
products, not just Sport.

Buying in sports data 
First off, you should look at refocusing Sport as a data proposition and not
just a content one.

Most major publishers typically buy in fixtures, results and tables from a
specialist sports data company. Users, and editorial teams expect that.
Depending on the amount and scale of data, this contract is usually paid
annually and is an ongoing cost. So how do you get maximum value from the data
you are buying?

The data usually contains unique IDs for teams, players and competitions, as
well as additional metadata such as dates, times, locations, rounds
(semi-finals, round 13 etc) and venues. These data feeds also contain names of
players, teams and competitions. A whole wealth of information that we could
exploit much better than we were. 

When we started looking in a lot more detail at the data that was being bought,
you could start to see the opportunities to properly join the dots between the
content proposition, the technical challenges, and what we wanted to build as a
Product. 

It wasn’t perfect the first time round we tried this, although we did get to the
stage very quickly of having a process to tag content and create automated pages
for all players, teams and groups at the 2010 World Cup
[https://www.bbc.co.uk/blogs/bbcinternet/2010/07/bbc_world_cup_2010_dynamic_sem.html]
.

We then realised that as Sport is so consistent, once you had a basic domain
model in place, then you could approach development and products in a different
way. 

Working up a Sport domain modelUsing the data to help your tagging 
Workflow is always a challenge, whether you are introducing a whole new system
or just tweaking an existing one. 

From a product perspective, I want consistent and accurately tagged content to
avoid issues with SEO, duplication, URLs and historical inconsistency across
different systems within a business. 

Other business aspects to consider were ensuring we could align the editorial
style guide, improving editorial team efficiency, enabling consistent reporting
of traffic and ad revenue, allow for cleaner syndication of content and at
times, manage tags in multiple languages (Bayern Munich and Bayern München are
the same team), and managing alternate names and synonyms for your tags.

When we first introduced manual tagging of content, we had issues with
consistency and accuracy. Even with guidance, individuals approached relevance
and accuracy in slightly different ways. 

Taking feedback from this, our next iteration was a semi-automated tagging
process using NLP techniques, where we focussed tagging only on headlines and
first 4 or 5 paragraphs of the content. The copy was cross referenced with the
sport vocabulary we had created from the bought in data and tags suggested to
journalists. 

This created a much quicker and accurate workflow for the editorial teams. 

There were unexpected challenges too, such as when a new CMS allowed creation of
multiple tags by content teams, with no editorial oversight on tag quality. The
CMS allowed audience-facing pages to be created automatically once a tag had
been used several times - even if they were incorrect. This caused major issues
for the editorial team, the SEO strategy and User Experience. 

We revisited the key terms around sport that the data supplier had delivered,
used that as a starting point for a controlled vocabulary for the editorial
team, and then used machine intelligence to automatically retag the archive of
existing articles, video and other content in the CMS, to ensure consistency. 

Linking a content model and data 
Another key part of this process is understanding the editorial workflow around
fixtures and events and where a level of automation can help save time and
effort. 

There is a considerable amount of consistency around how events are approached
within newsrooms, with editorial teams creating match previews, live coverage,
reports, photo galleries, post-match analysis etc, all of which fit within a 
content model
[https://www.cmswire.com/content-strategy/content-modeling-what-it-is-and-how-to-get-started/] 
and data strategy approach. This thinking equally applies to video and audio,
live and clips. 

As part of a workflow review we could see that the editorial teams were taking
time to manually create pages in the CMS in the days before a big weekend of
sport. 

We suggested using the bought in fixture list to drive automated creation of
pages in the CMS for journalists. We identified what templates needed creating
around a match - which links nicely with the content model work - and spun-up
pre-tagged templates in the CMS when the fixtures were announced. 

Before a match starts, we already know the teams, competition, venue, historical
records etc.  Depending on the sport, we will also know the athletes taking part
in a match (football tends to announce starting line-ups about 45 mins before
kick off) although other sports make announcements further in advance. 

Silver Oliver’s workshop flipchart diagram of editorial workflowScaling Up
This data-driven approach allows you to scale quickly. New competitions or
events can be approached from a ‘we just need some new data’ perspective and
this should fit with your ongoing data management strategy. 

This is especially relevant around major events where you may want to cover new
teams or events as part of the Olympics, Commonwealth Games or research has
shown you that your audience wants more detailed club level NFL or NRL coverage. 

If you want to create a new type of written content, such as a match report,
that is an extension of your existing content model. 

When you start adding in multiple media types used for coverage such as
photographs and social media, you can begin to see where consistency of tagging
and annotation is key to creating product propositions. 

These methods can also be used for content gathering as well - from external
suppliers to content supplied by correspondents and social media sources. 

Data visualisation
One of the challenges every media business has, is how do you stand out from the
crowd around Sport? Everyone has the same fixture list (that you have no control
over) therefore a focus on the user experience and a unique proposition helps
you stand out. 

Depending on the amount of data that you are buying, there is a lot of mileage
in offering a unique insight, and deeper analysis into the events or
competition. 

I worked on the Roboblogger project
[https://www.niemanlab.org/2016/06/the-telegraph-is-trying-to-streamline-soccer-live-blogging-with-an-automated-graphic-system/]
, part funded by the Google Digital News Innovation fund
[https://newsinitiative.withgoogle.com/dnifund/]. We really dug into the details
of what was in the football data that we bought and pushed the automated
publishing of infographics as far as we could.

It helped that we had already put in place a controlled vocabulary of sport
terms, which made the consistency between systems much easier. 

We ensured that any graphics created fitted with the visual style guide of The
Telegraph and were getting them published in under a few minutes after we
received data about the events. 

It’s not all about the website or app
Once you start digging into how a publishing business works, you can start to
see how progress could be made by thinking about an ‘open data strategy’ within
your organisation and allowing different systems to use the same vocabulary to
make life easier, although that does start changing the business focus to manage
data better - still a very overlooked area at the moment.

There are still multiple opportunities to join the data dots in businesses.
Especially linking back end systems and archiving with audience facing products.

 * Unlocking content archives. Always handy for Olympics, World Cups, and major
   events.
 * Ensuring you can map multiple, external suppliers to a single tag or ID.
 * Advertising and commercial consistency for teams and competitions and even
   individuals athletes.
 * Providing consistency between content and tags after mergers between
   companies.
 * Electronic Programme Guides (EPG).
 * Video tagging - enhancing video with structured data - both your own video
   and inbound social media video
 * Analysis and reporting.

Sport - or even an individual sport such as football - is a very consistent
domain which makes it easier to integrate into your business model, or at the
very least, look at Sport, as your first step towards utilising data and content
in a structured and consistent way.

The M word
I have avoided writing metadata all the way through this so far. 

Data suppliers provide very consistent products and can help you solve some
business and product issues. Linking together data and content to create better
products both for internal processes and audience-facing or syndication
purposes.

For example, you shouldn't have to rely on editorial teams to rebuild a section
or feature. We can do this based on the data we have. If google enables a new
rich snippet type we can add this markup etc. 

Most media organisations buy in content feeds, and although the assets are
tagged, they won’t be quite the same as the vocabulary that you use within your
organisation. Clarifying that vocabulary will save time and money between
systems and your own team.

The key to structured data is a well structured content model, a data management
strategy and metadata.

If you want to extend the model, then you can have a list of your contributors
and journalists as well. This would enable me to create a feed or even a
subscription based-product based on a question such as:

> “Give me all match reports (content type) written by Joe Bloggs (Author) about
Melchester Rovers (team) in the past 6 months (date range)”
Are you making the most of your bought data and content?
Data management is a challenge and unfortunately it does tend to get lumped in
with data science and analytics within a business.

I prefer to look at the reference data as a core building block for the Product.
I want to look at the details of the data contract to see what is being bought.
Does it fit with the current editorial strategy or is it just a historical add
on? Is this what the audience is looking for or expecting? Do they come to your
site or buy your paper for a specific type of coverage?

The cost of the data contract varies from company to company and of course also
depends on the amount of data bought. One thing in my experience that is
consistent though, is that usually only a small amount of the data is being used
by the business.

The coronavirus pandemic has caused a raft of fixture cancellations,
rearrangements, postponements and economic challenges across the board for
industries associated with sport. 

Meeting new business challenges is going to be about agility and speed of
response to new opportunities. 

Companies will have to be more agile in the future and the key to this agility
is having scalable and well marked up content that can be repurposed. 

The future
Hopefully 2021 will be a big year for Sport with a still to be defined weekly
sporting calendar, a rescheduled Olympics in Japan, a rescheduled European
football championships, a Lions Tour to South Africa and lots more. 

Companies will want to make the most of these opportunities for engaging
audiences and hopefully these examples can start some processes we have been
involved in:

 * Advertising and commercial consistency - for teams and competitions and even
   individual athletes
 * Unlocking under utilised content archives
 * Providing consistency across content and data after mergers between
   organisations
 * Electronic programme guides - a key syndication product and internal tool for
   some organisations
 * Video clips and live coverage

Around major events, other departments within your organisation that don’t
usually do sport coverage, may well want to start creating content about the
event such as documentaries etc that you may want to leverage.

Many of these ideas are not just relevant to Sport, the patterns I describe can
be applied to many domains and publishing any form of content, whether that is
in news, sport, entertainment, healthcare, science, or finance. Adding
geospatial metadata into the mix adds another dimension, and more avenues for
adding value.

Every domain has its own challenges but Sport is a very nice place to start. If
you would like to know more, please do get in touch.

Data Language is an industry leader in Information Architecture, ontology design and practical knowledge graph implementation. We use Domain Driven Design practises and user-centred design to focus information and data design on core business needs.

Find Information Architecture news and product innovation updates here, including ontology design, vocabulary management, domain & data modelling and more.

Information Architecture - News, Product Innovation and Technology

Information Architecture

Getting the most from your Sports Data

Silver Oliver

An introductory guide to tagging: Part 3

It has been my experience that many organisations struggle with high quality
tagging. In this post I am going to reflect on some of the causes of this
problem. 

As discussed in earlier posts, the value of tagging is often not properly
understood
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]
.  Most often it is considered to be there to support keyword search alone.
 Even the taggers themselves are not aware of how the tags are going to be used,
or why they need to be of good quality to add value.

The next issue is around the guidelines given to taggers.  Typically they are
extremely poor, and go along the lines of “tag what the content is about”.  This
question of “what is it about?” is harder than it seems at first glance and not
very helpful. It usually results in a different answer from each tagger.  To get
good quality tagging you need to ask good quality questions.

As the tagging quality goes down, the ability to drive useful features from the
tags also diminishes. This in turn does nothing to reassure people of the
importance of good tags and you end up in a vicious cycle of ever decreasing
quality and value. 

What good guidance on tagging looks like should be the point where Library
Science can come to the rescue.  But the challenge of how to identify the
subject of a work is a surprisingly under represented topic in Library Science
literature. 

Back in the the 1970s, Hutchins observed:

> THE LITERATURE OF INDEXING AND CLASSIFICATION contains remarkably little
discussion of the processes of indexing and classifying...we find very little
about how indexers and classifiers decide what the subject of a document is, how
they decide what it is 'about'.
The concept of 'aboutness' in subject indexing:
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.584.8116&rep=rep1&type=pdf] 
 W. J. Hutchins

Little has changed.  Taggers are often told to imagine "how might the user
search for this piece of content" or "take the author's view and characterise
what they are trying to say".  Neither is very useful as a framing question. 

Hutchins, in his paper, tackles this problem and introduces a distinction
between summarisation and aboutness. Summarisation is the representation of the
total subject content of a document and the statements made therein. While
aboutness is about capturing the thing that has been written about.

A very simple example might be a sport story about a player and whether they
will move clubs, their struggles with injury and their performance this year.
Many arguments are developed, making up the narrative of the story.
Summarisation would capture not only the player as a tag but also attempt to
capture something about the claims made about the subject of fitness,
performance and likely future at a club.  On the other hand, aboutness only
attempts to capture the thing being talked about and not what claims are made
about them. In this case just the player.

This distinction is rarely made in tagging guidelines and the tagger is left to
balance the level of summarisation and aboutness on gut feel.

Hutchins goes on to characterise this distinction in another interesting way.
The aboutness represents those things that are expected to already be part of
the readers knowledge. Every piece of content assumes a level of knowledge of
the reader with which the narrative then builds. Summarisation on top of this
attempts to capture what is new and builds on that existing knowledge in the
narrative of the content.  The readers are familiar with the player, but not
about the subtle changes in his fitness and future prospects.

With this in mind, the questions we might ask a tagger to answer are: What is
the thing being talked about? What claims or facts are made about a thing in the
content? What might the reader be presumed to know before reading this?

Whether we choose to do summarisation or capture aboutness is a decision related
to the desired outcome of the system. But we should be clear about which one it
is.

A more recent analysis by Birger Hjorland (The concept of subject in
Information
Science [https://www.emeraldinsight.com/doi/abs/10.1108/eb026895]) revisits the
problem.  He cites many examples of Guidelines expecting taggers to do little
more than perform "mind reading" on prospective future consumers of the content.

Instead, he argues, we must think about tags as a tool in enabling the full
potential of the document within the context of the overarching system.  The
desired outcomes of this system should then dictate the tagging strategies and
guidelines. 

It is this domain and system that must be crystal clear in taggers' minds as
opposed to the idea of aligning subject to a theoretical searcher.  They need to
also understand the value that they are creating and how.

Sadly, this is the opposite of what we see in many tagging tools and guidelines
today.

I would argue that tagging should be a contract between taggers and the system
 much in the way we see in the design of data services
[https://medium.com/@diogo.lucas/https-medium-com-diogo-lucas-contract-breaks-are-kinda-of-deal-breakers-447762de20c4]
.  Tagging guidance needs to be managed with a vision of the desired outcomes
combined with asking clear and simple questions of the taggers. This is as true
for humans as machines.

In summary, tagging outside of libraries and archives is often of poor quality.
Value is rarely understood, content creators are often asked to tag in ways that
are a chore and cognitively awkward. Worst of all, the question of what good
tagging looks like has rarely been asked.

In the next post I will look at some potential solutions to the challenge of
better tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]
.

Part 1: Why the web turned its back on the librarians. And why we need them
back.
[https://datalanguage.com/features/why-the-web-turned-its-back-on-the-librarians]

Part 2: The importance of tagging when publishing on the web
[https://datalanguage.com/features/the-importance-of-tagging-when-publishing-to-the-web]

Part 4: How to use domain modeling to improve subject tagging
[https://datalanguage.com/features/how-to-use-domain-modeling-to-improve-subject-tagging]

Publishing Tech

Publishing Tech Blog Posts