We've just benchmarked our Text AI Inline Entity Extraction SaaS. The results show a very high F1, and huge workflow efficiency improvements.
We have recently completed some new "Inline Entity Extraction" benchmarks. This is a sophisticated "Named Entity Recognition" (NER) product that we launched as part of our Text AI Services suite in 2020. It is optimized for Knowledge Graph platforms in high-value knowledge organizations. It identifies entities inline within text media and attributes rich metadata.
Our customers are high-value media publishers, knowledge organizations, and DaaS providers.
We trained our Inline Entity Extraction SaaS to understand around 180,000 entities from a custom domain. We then benchmarked it in a controlled environment where two teams of annotators were asked to annotate 500 documents:-
Here’s how our Data Language Text AI Services Inline Entity Extraction SaaS performed:
We have proven we can train a highly accurate model, specialized for a custom domain, that can cope with hundreds of thousands of entities with very high-performing metrics, using a golden corpus that covers only a small subset of the entities.
Achieving such high results on the classification of such a high number of classes is very rare for a model like this in the wild.
The findings of this exercise were that the error rate of the team that used the Data Language AI model to pre-annotate the documents was substantially lower than the team that worked without the model.
The measurable benefits were clearly efficiency and accuracy and added to the coverage of entities being substantially larger, it’s clear that using our model was a great success.
We were also analyzing whether there is benefit in keeping humans in the loop and this result proved surprising. While it was clear that helping humans out in entity extractions tasks by pre-running the content through a model could significantly assist their operation, we actually found that the error rate of the model was actually slightly higher when humans were involved than when the results were totally automated. In this case, humans reviewing results actually makes things slightly worse!
There are some cases where perfect results are not mission-critical - e.g. upstream automated tagging to simplify downstream workflow, where "the machines do the heavy lifting". In these cases a certain amount of error is acceptable and a totally automated process will work.
In other cases, where errors are less acceptable, the confidence and evidence metadata in our SaaS can be used to allocate "less certain" predictions to SMEs and expert analysts for decisions and feedback loop optimizations.
So what does our competition offer to compete with our results in the Named Entity Recognition (NER) space?
Well, it’s difficult to compare apples to apples in this scenario, as our competitors suffer from the same problem outlined above: Complex custom domains with small training corpora.
This is the exact advantage of our SaaS that these benchmarks emphasize.
To put these results into the context of standardized testing of NER performance elsewhere in the wider world and industry, one usually finds that most tests of F1 approach the value of 1, only where these models work by identifying a small handful of classes. Have a look - you will likely see "People, Places, Organizations, Events"... the usual suspects, and this problem itself has been solved daily for the last 12 years by almost every computer science student.
The fact that our model can cope with a far greater number of classes than most other providers, and still see such impressive results, is a strong differentiator.
Another advantage of our Data Language AI Services that we offer is that they work out of the box at scale for your production platforms.
Our Inline Entity Extraction SaaS comes with all these benefits immediately:
All the engineering is done, and the security & maintenance are on us!
Later in 2022, we will launch a new commercial Knowledge Graph Relationship Extraction service as part of Text AI Services - early testing has impressive benchmarks for high-value knowledge organizations. More to come soon.
For more information on Data Language AI Services, check out the links below.