Paul Wilton

Data Language has been accepted onto the Digital Catapult Machine Intelligence
Garage [https://www.digicatapult.org.uk/projects/machine-intelligence-garage] 
program.

This is fantastic news for us. Through MI Garage, Digital Catapult is going to
provide us with plenty of high performance infrastructure and technical support
to power our machine learning and AI product R&D throughout 2019.

This will allow us to rapidly advance innovation and development across all our
products, running our ML models at scale, and on GPU kit. It will enable us to
expand our Tagmatic [https://datalanguage.com/tagmatic] ML product suite for
automated content classification; our deep learning image classification
platform, SCOPA; and help with automation on our Contextual Commerce platform, 
Luminery [/2019/02/06/digital-catapult-machine-intelligence-garage/luminery.com]
.

2019 is going to be a great year for Data Language!

AI is a core discipline at Data Language. We have an expert team skilled in machine learning and deep learning, and importantly we also have the engineering and deployment skills to deliver AI at scale for your production systems.

AI is a core discipline at Data Language: Our expert team is skilled in machine learning and the engineering skills needed to deliver AI at scale.

Artificial Intelligence delivered at scale - our core discipline.

Artificial Intelligence

Digital Catapult - Machine Intelligence Garage

Tagmatic is a service for automatically tagging your content.
It uses sophisticated machine learning techniques to learn how you tag your
content on-the-fly and then will tag new content automatically.
Tagmatic uses your own tag dictionaries. Whether you tag with simple text terms,
identifiers, taxonomies, or linked data, Tagmatic will tag with those. There is
no data mapping exercise needed, just send your content & tags to Tagmatic and
it gets to work. Tagmatic quickly learns new tags too.
It is incredibly easy to integrate into your CMS or editorial workflows via two
simple web service interfaces.
If you are looking for an automated tagging solution, then look no further.
If you are developing your own automated tagging solution, then pause, give
Tagmatic a try - and be up and running quickly.

If you would like to try Tagmatic on your own content, get in touch
[https://datalanguage.com/tagmatic]  for a free trial.
For more information go to https://datalanguage.com/tagmatic

News, Blog Posts and Innovation updates about Tagmatic - our sophisticated AI automated content classification SaaS product. We have taken care of all the engineering and data science for you: Tagmatic learns on the fly, is self-optimising, and language-agnostic.

Tagmatic is an AI automated text classification SaaS product by Data Language. It learns on the fly, is self-optimising, and language-agnostic.

Tagmatic - News and Product Updates about our Text Classification AI

Tagmatic

Introducing Tagmatic

Fabio Colasanti

Earlier this week, this tweet
[https://mobile.twitter.com/random_walker/status/1196870349574623232] appeared
on my feed.

> Much of what’s being sold as "AI" today is snake oil. It does not and cannot
work. In a talk at MIT yesterday, I described why this happening, how we can
recognize flawed AI claims, and push back. Here are my annotated slides: 
https://t.co/iCpyFw5urN pic.twitter.com/pmOTI3vq8p [https://t.co/pmOTI3vq8p]

— Arvind Narayanan (@random_walker) November 19, 2019
[https://twitter.com/random_walker/status/1196870349574623232?ref_src=twsrc%5Etfw]

Prof Narayanan's deck
[https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf] has
reached a relatively sizeable audience and received widespread positive
commentary. After all, anyone who has had any involvement with AI in the last
few years can 100% relate with the catchy title of his deck.
Hey - we're just about starting to see a feeble light at the end of the
blockchain tunnel, right?

But, but...
Something doesn't quite sit well with me. So I'm going to stick my neck out and
respectfully reject, if not the conclusions, the way Prof Narayanan has chosen
to make his point. Or rather, the way he has chosen to make his real point.

I find this deck deceptively coy, as the actual message is not "How to recognize
AI snake oil" but rather "Why I don't want AI anywhere near socially sensitive
topics", which would have been a perfectly valid argument, and made for a
fantastic set of slides, but required a slightly different set of arguments and,
most importantly, to be explicit about it.

At the end of the day ML/AI is about identifying patterns in data and draw
inferences, so of course it can "predict the future". Google Maps does it all
the times when it gives you an ETA for your journeys. In Prof Narayanan's own
examples, bail decisions and border control are not performed by machines, but
based on the prediction of machines. It's not about predicting the future: it's
called decision support, performed at speed that is beyond human capabilities.
Is the way banks and insurance companies make decision on mortgages and premiums
snake oil too? 
Prof Narayanan seems exceedingly preoccupied about "massive transfer of power
from [...] workers to unaccountable tech companies" and points at "enormous
commercial interests" that "we must resist" and "push back". He is of course
completely entitled to give the whole argument a political spin, but I'm not
sure this helps his arguments.

The real issue here is that some problems, and especially socially impactful
ones, carry high personal risks if an inference is wrong (problem 1). Therefore,
in order to minimise that, large amounts of potentially sensitive data may be
required (problem 2) and in most, if not all, cases there is a strong argument
for transparency, accountability and human-in-the-loop that makes such
propositions much more expensive and challenging from both a technical (problem
3) and governance (problem 4) perspective. 
Bias poses no doubt massive challenges (problem 5), not just in its
identification and removal, but also in its very definition, as what turns
common sense into bias is a function of our set of social and cultural values,
which greatly varies in time and space. Should history be changed because it
doesn't fit our Western 21st century view of the world? I don't have an answer,
but at least I'm asking the question.
Also, there is the topic of regulation which can't be ignored in these cases:
who establishes the boundaries? Who is it meant to enforce them? Do we need a
universal legislation, or would locally applicable frameworks suffice? Who makes
a call on what is an acceptable risk vs potentially beneficial outcomes -
supranational bodies, governments, individuals, etc.. ? (problem 6)

So the better discussion to have is around the risks involved in deploying AI in
socially sensitive fields, what is the minimum level of benefits it must bring
about to make it worth us facing the challenges above, and how do we go about
having that conversation.

Just branding the whole thing as snake oil is sidestepping the issue and largely
unfair, because it's bypassing the actual conversations to be had and going
straight for the outcome that Prof Narayanan - in arguably a biased opinion -
has decided is what's best for us all.

We need to talk about AI

It may be argued that customers' satisfaction is the only and ultimate KPI that
really matters when judging the success of a product. In which case Tagmatic
would pass with flying colours, given the fantastic feedback we are receiving
from clients. 

Remarkably enough, we found that sheer inferencing performances are often at the
edges of our clients' considerations: this is partly due to the intrinsically
tailored nature of Tagmatic's models, but mostly attributable to a plethora of
invaluable qualities that go beyond the naked F1 scores: being able to quickly
spin up a model on your own content and vocabulary, and then interact with it
via a minimalistic HTTP interface, zero maintenance, on-the-fly learning of new
classes, low latency, security, scalability and high availability out of the
box, are crucial factors to enterprise-scale deliveries. 

Your Intellectual Property lies in your content, your data, your data models,
and how you use the classifications. It is not in commoditised compute
solutions.
Your resources are thus more effectively spent developing and exploiting your
market differentiating IP, rather than spending way too much time and money on
trying to replicate the engineering complexities of off-the-shelf MLaaS
solutions which may not be your technical team's core competence.

This is where Tagmatic comes in. The hard work and complex engineering has been
done, just bring your content!

The datasets
RCV1 is arguably the most popular dataset for multi-label classification,
comprising of 800,000 newswires by Reuters tagged with 103 classes. Oddly enough
for such a totemic dataset, the breadth of the vocabulary is anachronistic when
compared to some of our clients, modern global publishers who routinely maintain
taxonomies with thousands of topics.
The dataset is available here
[http://www.ai.mit.edu/projects/jmlr/papers/volume5/lewis04a/lyrl2004_rcv1v2_README.htm] 
although unfortunately not in its original XML format with full HTML markup,
which we would normally leverage as part of the classification task. Also the
corpus comes already tokenised and train/test datasets pre-canned. Besides
taking away part of the fun, this has also negative implications on the
vectorisation logic Tagmatic implements, making it largely ineffective
unfortunately. For the purpose of this benchmark, we have used the full train
set (23,000) and a reduced test set (100,000).

AAPD (Arvix Academic Paper Dataset) is a collection of research papers covering
a number of subjects organised in several classification systems. The dataset 
here [https://www.kaggle.com/neelshah18/arxivdataset] is annotated with a number
of classification schemes, of varying level of quality. We have carved out a
subset corpus in the Computer Science domain, comprising of 32,000 papers and 39
subjects in total.

The benchmark
We have considered a number of models amongst the most widely employed for
multi-label classification tasks. The reference evaluation metric is
weighted-F1.

RCV1 AAPD One-vs-Rest - LinearSVC 0.78 0.71 One-vs-Rest - SGD 0.74 0.62 
One-vs-Rest - XGBoost 0.75 0.64 Classifier Chain - LinearSVC 0.79 0.71 Label
Powerset - MultinomialNB 0.55 0.55 Label Powerset - RandomForest 0.60 0.55 
Tagmatic 0.80 0.74 
Tagmatic offers out-of-the-box improvement over a number of basic
implementations. Granted, a team of Data Scientists might possibly come up with
better optimised flavours of such models, pushing performances up. But at what
cost? And by how much? And what then? There is still the much bigger task of
bringing that model to fruition in a real world environment, requiring
non-trivial development effort. Is that really worth it? 
Unless text classification is either your USP or a strong market differentiator,
then the answer ought to be a resounding no. 

Tagmatic is production-ready, enterprise-grade, and available to you right now
[https://datalanguage.com/tagmatic].

Lastly...
We have also benchmarked Tagmatic against Hansard data: a corpus of ~200,000
parliamentary interventions for the 2005/6 period , classified with ~7200
categories, scoring a whopping 0.83. More on this on the next blog post. 
Make sure you sign up for our newsletter [ https://datalanguage.com/contact-us] 
and follow @Tagmatic [https://twitter.com/tagmatic] on Twitter for all the
latest.

Benchmarking Tagmatic

For Data Science to truly live up to expectations, it must embrace software craftsmanship.

Data Science is about building software

Earlier this month I was chatting with my old friend Pierluigi Casale
[https://www.linkedin.com/in/pierluigi-casale-41271430/] (Data Scientist at
TomTom, AI lecturer and researcher) about what seems to be the question of the
century: "What is Data Science"? Tongue in cheek, I offered my very own answer:
"Data Science is that thing you get when someone with a PhD plays with a Jupyter
Notebook, and a technical architect refuses to let it anywhere near production
because it doesn't have a single unit test". 
Although sporting a PhD himself, Pierluigi agreed that my slightly caustic
definition was indeed something he observed himself, and that he too was wary
of. He mentioned how he often concludes the many talks he gives at conferences
around the world with a diagram from the notorious paper "Hidden Technical Debt
in Machine Learning Systems
[https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf]
" by D. Scully et al.


There is indeed a widespread assumption that Data Science and Machine Learning
belong into a hazy limbo somewhere between data analysis, analytics, and R&D.
Within this mystical space, a set of practices otherwise regarded as inefficient
seem to be accepted as the norm: manual mining and manipulation of data; one-off
transformations virtually impossible to replicate; casual scripting of code with
total disregard of any development standard; artefacts laying around and code
running here or there...
When Scully talks about "glue code" and "plumbing", he is pointing at the right
problems but possibly underplaying the issue by being too humble about it. So
forgive my bluntness and let me put that straight: Data Science is about
building software.

Whether the context is academia or enterprise, research or commercial alike, the
adoption of Software Development best practices, tools and methodologies is the
key for efficient delivery of outcomes. At Data Language we have worked with a
number of clients at different levels of "enlightenment", and witnessed first
hand how injecting software craftsmanship into Data Science can lead to
profoundly transformational effects. We can also testify how the development of
our Tagmatic [https://datalanguage.com/tagmatic] service has been indeed 10%
Machine Learning and 90% software engineering: not much we could have done with
our shiny model without things like containerisation, high availability, high
throughput, load balancing, scaling, automation, security, governance,
monitoring, and many others. 

The adoption of established best practices in Software Development will boost
productivity, shorten the feedback cycles, and produce more robust code that is
easier to change, maintain, and ship.

Six things we at Data Language do and you should too
1. Ditch the darn notebooks
I'm fully aware this is highly controversial bordering sacrilegious for many,
but the truth is these type of editors foster bad habits and are holding you
back. They are like comfy, safe, cute trikes you got used to riding when you
were three; but if you actually want to go somewhere other than your garden,
you’ve got to move on and get yourself a bike. 
At this year's JupyterCon in NY, the talk widely regarded as best of the lot was
"I don't like notebooks.
[https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68282]"
by Joel Grus. I wholeheartedly invite you to check it out.

2. Version control
A common refrain in favour of notebooks, is that they enable collaboration.
Oddly enough, when further engaged on this topic and asked about which version
control tool they use, what's their branching strategy, how often they merge,
etc... it all unravels. 
Collaboration is mighty important, so make sure you get on top of your version
control strategy. (And ditch the darn notebooks)

3. Test automation and CI
Arguments for Test Driven Development and Continuous Integration are widely
available and universally recognised as the single crucial factor that enables
faster, safer, better code. The very same arguments apply to Data Science. The
ability to move faster with your code enables quicker feedback and shorter
iterations: you will be able to experiment and prove ideas quicker, boosting
productivity. You will also end up with code that is designed better, and it can
be transitioned into a production environment more easily.

4. Infrastructure as code
In order to take full advantage of the ongoing democratisation of computing
power, it is crucial to write code that is portable and can be deployed wherever
is faster to run it. In a typical development setting, we want be able to
provision computing resources in the cloud, deploy our code there, execute it,
collect outputs and dispose of the remote resources. And we want to be able to
do this in a fully automated fashion, at the push of a button.
Many ways to skin this cat, but if I were to single out one thing you should do,
and learn to do well, is containerisation. Plenty of great Docker learning
resources around: get stuck in!

5. Data management
As in any other scientific setting, it is mighty important for experiments to be
repeatable under the same conditions. We want to make sure immutability of
inputs is preserved through transformations. Versioning and artefacts management
tools play an invaluable role in the development of Data Science solutions. This
goes hand in hand with practices discussed in points 2) and 3), and falls
naturally from there.

6. Metadata management
Often the journey is just as valuable as the destination. As much as possible
(and to the extent that is useful, of course!) we want to be able to retain key
information about the process at hand, so that we can better interpret, analyse
and enhance its outputs. The spectrum here is broad, from basic logging to
systematic capturing and analysis of metadata. The point is: we want to be able
to fulfil this function in the way that maximises the insights offered by the
data being collected. 
Metadata management is at the core of many activities that are either routine
(eg compare performance of models), or required for production (eg backout
strategy for a failing data pipeline), or further reaching into the realms of
ethics and accountability (eg quantitative justification of a Machine Learning
prediction). It is important we do it well.

Conclusions
I fully appreciate some of the recommendations above may feel daunting to the
more traditional Data Scientists, due to such activities not (yet!) being
typically associated with their profession. To some extent, we witnessed
something similar in the past with network and infrastructure specialists, now
evolved in DevOps that design and build infrastructure solutions through code,
fully embedded in the fabric of development teams from the onset, sharing tools
and methodologies. 
Similarly, the key for enabling the evolution of Data Science from R&D practice
to fully fledged enterprise function, lays in bringing data scientists and
developers closer: before long the distinction will blur, and specialist
knowledge will thrive on a common substrate of working practices. 

At the end of the day, remember, Data Science is about building software.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. We deliver practical data science implementations for both our own SaaS products, and for our clients' bespoke platforms and products, for the analysis of data and media at scale.

We are specialists in the design, implementation and deployment of Scalable Data Science solutions. Find updates, News and insights on Data Science here.

Data Science - our Work, Features, News and Updates

Data Science

Julian Everett

Over the last couple years, one of the more interesting architectural patterns
to have emerged on deliveries for our media customers has internally acquired
the name ‘daisy architecture’. It evolves as organisations acquire the ability
to scale their premium content production processes via the use of machine
learning, text analytics and annotation techniques, and then extend the coverage
of that tooling over as much of their remaining content geography as possible.
In workflow terms, this entails a switch from content production pipelines that
are a.) push-based; b.) manual and c.) linear; to content enrichment cycles that
are a.) pull-based; b.) automated; and c.) on-demand and iterative:



Enabling factors
The pattern is typically the result of three technical enablers:

 1. The increasing influence of Kanban on software design, (i.e. principles
    taken from how applications are delivered and applied to how they work, in
    particular pull-based systems and queue balancing/optimisation techniques).
 2. The scaling of automated content analysis and rich metadata creation via
    text analytics and machine learning.
 3. The adoption of micro-service architectures, allowing business functions to
    be recombined much more easily to serve the individual needs of specific
    product contexts.

1. Kanban
The influence of Kanban techniques not only on the process of application
delivery but on the outputs of application design (i.e. on business processes
implemented in software) is arguably a return to its roots in some ways. We
perceived that as a direct transfer of principles, but what also seems to be
going on here is that the influence of Kanban is spreading out of software
delivery into business operations more generally (we are definitely seeing
increased adoption of pull-based value streams measured via cycle time and queue
buffer monitoring) and that is then being reflected back into the styles of
software design that are required to support those pull-based business
processes.

2. Machine Learning
There has been without doubt an aggressive increase in the use of machine
learning and AI techniques within media organisations to automate content
metadata creation and indexing (see our machine learning cloud service
[https://datalanguage.com/tagmatic] for a typical example). Critical differences
between automated and manual task processing are firstly that automation has
essentially negligible operational cost, and secondly it has a larger
variability in the quality of its output. This has a major impact on the
economics of workflow design. Historically, content production workflows were
designed as linear pipelines because every step was manual and therefore costs
were within similar boundaries of magnitude. Once certain steps incur negligible
cost however, new opportunities open up to process much larger content volumes
at basic quality levels. From a brand/product risk management perspective, this
then needs to be complimented by quality assurances steps, i.e. manual review.
Those manual steps are where the costs are incurred, which is why it is highly
desirable for them to do so just-in-time i.e. once product opportunities have
been targeted which clearly outweigh that investment.

Interestingly, for these reasons we are seeing a significant impact of
automation on changing job roles in this space, but a negligible impact of
automation on actual jobs - apart from some impact to very repetitive,
rule-based work (i.e. a prime candidate for a machine) that has most often
already been outsourced. Instead we are seeing job roles switching from
production at lower volumes to the curation of production at higher volumes.
This typically takes the form of:

a.Quality Assurance of machine output, the results of which can then be used as
ongoing training data to refine machine statistical models.
b.Sampling Tuning. Not all content is equal: the quality of machine output will
depend on both its source and its subject area (e.g. the simple factual
statements of a football match report will typically be easier for a machine to
analyse successfully than an in-depth opinion piece). That means different QA
sampling rates will be required for content in different subject areas and from
different sources: e.g. new syndication source vs in-house editorial, and those
rates will then need to be managed by people over time: e.g. as the new
syndication source becomes a trusted partner they will need less manual
oversight so the sampling rate should be reduced.
c.Terminology management: to ensure that controlled vocabularies are used to
encode the core entities and relationships of the business domain thereby
maximising data interoperability across the organisation. (This is one of the
core benefits of using domain-driven design
[https://datalanguage.com/services/information-architecture] to define the
information architecture)

3. Micro-services
The adoption of micro-service architectures has unlocked access to discrete
units of business functionality, allowing them to be recombined in novel ways
according to the emerging needs of new product contexts. This doesn’t mean a
return to the dark days of untestable spaghetti service orchestrations in
black-box ESBs. Choreography logic should still be pushed into endpoints (or
else into a standalone endpoint if justified), but it should be implemented as a
strategy pattern – where the content enrichment strategy is determined by a
specific product need rather than hardcoded.

Services and Routing
Product-specific service choreography basically means having a routing service
that can execute an enrichment strategy which encodes the content requirements
of a specific product context (for example: it needs subject classification
metadata, time-coded video scene metadata, advertising block-word annotations,
etc). Doing this just-in-time however means that you only know the state of the
required output, not the input state (as you don’t know which enrichment
operations might already have been performed). This has two interesting
consequences:

 1. Routing tends to be both expressed and executed in terms of gap analysis:
    i.e. perform whatever enrichment operations are required to get content
    selection C into state P needed by my product.
 2. Services tend to be partitioned into enrichment services and content
    search/persistence services, with the latter frequently needing to support
    terse query semantics that can be either named or URL encoded and passed
    between other services.

The final point worth mentioning concerns the temptation to implement aggregate
AI solutions, i.e. replacing a workflow that is currently 12 manual
steps/functions with a single neural network solution or similar. The argument
goes that machine systems are fundamentally different from human systems hence
there is no need to decompose them into small steps: you can just train from
required outputs against known inputs and get rid of all the complexity. The
first point to note here is that the complexity isn’t being removed, it’s just
being hidden. Moreover, it is being aggressively hidden as anything encoded in a
neural network is semantically opaque: it is essentially just set of weightings
that produces a desired output, which means there are no conceptually
interoperable subcomponents that can be decomposed back out into something
actionable by humans if anything starts going wrong. Also error margins will
almost certainly be amplified across a monolithic system that has no
intermediary review and correction checkpoints. For these reasons our advice is
to steer clear of AI monoliths. Instead we recommend you continue modelling your
business processes as discrete, atomic functions, and then leave the decision of
human vs machine vs human-assisted machine to be an implementation concern only.
The key task is to ensure your systems are then designed to allow that decision
to be revised according to changing circumstance with minimal operational
impact.

The next generation of adaptive businesses will be those that can swap between
outsourced, insourced, human, machine and human-assisted machine implementations
of their core business functions with minimal impact to their ongoing
operational capabilities. That will be the true test of how cohesive and
loosely-coupled the service-based architectures of today really are.

Machine Learning

Machine Learning Blog Posts

News about Machine Learning