Fabio Colasanti

We have recently completed some new "Inline Entity Extraction" benchmarks. This
is a sophisticated "Named Entity Recognition" (NER) product that we launched as
part of our Text AI Services suite in 2020
[https://datalanguage.com/news/inline-entity-extraction-text-ai-ner]. It is
optimized for Knowledge Graph platforms in high-value knowledge organizations.
It identifies entities inline within text media and attributes rich metadata.

The Problem: Automated Tagging in Custom Domains
Our customers are high-value media publishers, knowledge organizations, and DaaS
providers. 

 * A huge portion of their business asset value is locked in their unstructured
   mixed media assets.
 * To liberate this value, accurate inline text classification of 100s of 1,000s
   of entities is required to enable structured connection with bespoke
   information architecture models and/or knowledge graphs.
 * Generic off-the-shelf NLP SaaS solutions do not recognize these custom
   entities, so a mapping exercise is required, and is often a too low
   resolution to provide value.
 * Many NLP solutions do not provide the rich, domain model-aware metadata,
   because they are not trained for the custom domain.
 * Other custom NLP solutions require larger training sets and golden corpora
   that cover the entire entity set.
 * "Build it yourself" becomes very expensive, with a large data scientist and
   engineering overheads, while organizations find themselves re-solving the
   same challenges that we have already solved.
 * "Manage it yourself" has a surprisingly large data science and engineering
   overhead, to overcome production challenges that we have already solved.

Testing methodology
We trained our Inline Entity Extraction SaaS to understand around 180,000
entities from a custom domain.  We then benchmarked it in a controlled
environment where two teams of annotators were asked to annotate 500 documents:-

 * The first team used the old-fashioned method of manually annotating a
   document.
 * The second team moderated the output of the model that had already scanned
   the documents and picked up entities, only needing to confirm or reject
   pre-determined results.

Results: High F1 Score
Here’s how our Data Language Text AI Services
[https://datalanguage.com/products/datalanguageai/text-ai-services] Inline
Entity Extraction SaaS performed:

 * Precision: 0.94
 * Recall: 0.90
 * F1: 0.92

We have proven we can train a highly accurate model, specialized for a custom
domain, that can cope with hundreds of thousands of entities with very
high-performing metrics, using a golden corpus that covers only a small subset
of the entities. 

Achieving such high results on the classification of such a high number of
classes is very rare for a model like this in the wild.

Results: Huge Workflow Efficiency Gains
The findings of this exercise were that the error rate of the team that used the
Data Language AI model to pre-annotate the documents was substantially lower
than the team that worked without the model. 

 * The team using "Data Language AI" demonstrated a 57% drop in error rate.
 * The team using "Data Language AI" demonstrated a 53% increase in annotation
   coverage.

The measurable benefits were clearly efficiency and accuracy and added to the
coverage of entities being substantially larger, it’s clear that using our model
was a great success.

Should we keep Humans in the loop, or not?
We were also analyzing whether there is benefit in keeping humans in the loop
and this result proved surprising. While it was clear that helping humans out in
entity extractions tasks by pre-running the content through a model could
significantly assist their operation, we actually found that the error rate of
the model was actually slightly higher when humans were involved than when the
results were totally automated. In this case, humans reviewing results actually
makes things slightly worse!

There are some cases where perfect results are not mission-critical - e.g.
upstream automated tagging to simplify downstream workflow, where "the machines
do the heavy lifting". In these cases a certain amount of error is acceptable
and a totally automated process will work. 

In other cases, where errors are less acceptable, the confidence and evidence
metadata in our SaaS can be used to allocate "less certain" predictions to SMEs
and expert analysts for decisions and feedback loop optimizations.

Comparison to other Text Analytics NER Approaches
So what does our competition offer to compete with our results in the Named
Entity Recognition (NER) space? 

Well, it’s difficult to compare apples to apples in this scenario, as our
competitors suffer from the same problem outlined above: Complex custom domains
with small training corpora.

This is the exact advantage of our SaaS that these benchmarks emphasize. 

To put these results into the context of standardized testing of NER performance
elsewhere in the wider world and industry, one usually finds that most tests of
F1 approach the value of 1, only where these models work by identifying a small
handful of classes. Have a look - you will likely see "People, Places,
Organizations, Events"... the usual suspects, and this problem itself has been
solved daily for the last 12 years by almost every computer science student.

The Benefits of Data Language Text AI Services
The fact that our model can cope with a far greater number of classes than most
other providers, and still see such impressive results, is a strong
differentiator.

Another advantage of our Data Language AI Services
[https://datalanguage.com/products/datalanguageai] that we offer is that they
work out of the box at scale for your production platforms.

Our Inline Entity Extraction SaaS comes with all these benefits immediately: 

 * Ability to classify text content to 100s of 1,000s of entities in a custom
   domain.
 * A rich API for interacting with the machine learning service metadata.
 * Automatic management of machine learning model ‘fallout’.
 * Self-optimisation of the machine learning models, on the fly.
 * Capability to cope with an extremely high number of entities.
 * The ability to scale up indefinitely.
 * Production-ready engineering, out of the box.

All the engineering is done, and the security & maintenance are on us!

AI Relationship Extraction for Knowledge Graph Platforms
Later in 2022, we will launch a new commercial Knowledge Graph Relationship
Extraction service as part of Text AI Services - early testing has impressive
benchmarks for high-value knowledge organizations. More to come soon.

For more information on Data Language AI Services, check out the links below.

Data Langauge AI [https://datalanguage.com/products/datalanguageai] | Text AI
Services [https://datalanguage.com/lp/text-ai-services]

New Benchmarks from our Named Entity Recognition (NER) Text AI SaaS Product

Alexi Buckingham

We are proud to announce the new member of our Text AI Services
[https://datalanguage.ghost.io/products/datalanguageai/text-ai-services] family:
Inline Entity Extraction. This new AI NLP service was created for high-value
knowledge organizations who want to accurately classify their text media at a
granular, inline-level.

The Problem: Automated Tagging in Custom Domains
We created this service to solve the problems that other Named Entity
Recognition (NER) NLP SaaS - and "build it yourself" approaches - could not
solve.

Our customers are high-value media and knowledge organizations. A huge portion
of their business asset value is locked in their unstructured media assets,
which they must leverage to add value to their subscription, DaaS, targeted
advertising, and other business models. This unstructured content must be
connected to their custom business domain via knowledge graphs and other
structured data systems:-

 * Complexity: To liberate this value, accurate inline text classification of 
   100s of 1,000s of entities is required to enable structured connection with
   bespoke information architecture models and/or knowledge graphs.
 * Custom Domain: Generic off-the-shelf NLP SaaS solutions do not recognize
   these custom entities, so a mapping exercise is required, and is often a too
   low resolution to provide value.
 * Rich Metadata: Other NLP solutions do not provide the rich, domain
   model-aware metadata, because they are not trained for the custom domain.
 * Small Training Sets: Other custom NLP solutions require larger training sets
   and golden corpora that cover the entire target entity set.
 * Cost of Build: "Build it yourself" becomes very expensive, with a large data
   scientist and engineering overheads, while organizations find themselves
   re-solving the same challenges that we have already solved.
 * Cost of Maintenance: "Manage it yourself" has a surprisingly large data
   science and engineering overhead, to overcome production challenges that we
   have already solved.

Our Solution: Text AI Services Inline Entity Extraction
Our Inline Entity Extraction SaaS handles a far greater number of classes than
other NER NLP solutions, while delivering impressive performance results. It
also works out of the box at scale for your production platforms.

Our Inline Entity Extraction SaaS comes with all these benefits immediately: 

 * Custom Domains at Scale: It classifies text content to 100s of 1,000s of
   entities in a custom domain.
 * Powerful API: It provides a rich API for interacting with the machine
   learning service metadata.
 * Does its own ML housekeeping: It automatically manages machine learning model
   ‘fallout’.
 * Self-Optimizing: It self-optimizes its machine learning models, on the fly.
 * Scalable: It is able to scale up indefinitely.
 * Production-Ready: It is Production-ready, out of the box.

All the engineering is done, and the security & maintenance are on us!

For more information on Data Language AI Services, check out the links below.

Data Langauge AI [https://datalanguage.com/products/datalanguageai] | Text AI
Services [https://datalanguage.com/lp/text-ai-services]

Product Launch: Inline Entity Extraction SaaS - AI Custom Domain Auto-Tagging

Entity Extraction, and Named Entity Recognition (NER) are NLP capabilities used in media analytics. Find out about our NER product updates and features here.

Inline Entity Extraction & Named Entity Recognition

News about Inline Entity Extraction & Named Entity Recognition