Paul Wilton

Ear of bat, blood of frog, rabbit’s tail. Stir them all up in your graph
database, utter the magic words, and lo, with a sprinkling of sparql dust, gold
speweth forth.

After working with graph databases for 10 years now, it troubles me that this is
still how many implementers expect things to work. Graph databases are not magic
cauldrons. They really are not.

A graph database is no different from any other database. If you put in data
mess, do not expect order to be created from chaos. If you then combine this
with over constrained and over-engineered inference and reasoning you end up
with an even bigger mess. A popcorn machine.

You will not be able to build robust practical business or enterprise grade
applications by treating your graph as some dirty place where ontologists (who?)
 can dip their fingers in, mess around ad-hoc, and make magic happen. “let’s
just add another inference rule, that will sort it..”

Just like any other practical, robust data architecture, data governance is
essential. Just like any other database you would not expect end users to
interface directly with it via a SQL endpoint, or a JDBC connector? No, you
would build robust, well tested APIs and Services to interface with your
database, and let your application builders interface with these services. End
users would then interact with these applications. These API services are the
gatekeepers of good data governance.  No mess should pass this point. In
production, no human hands should *ever* pass this point. 

Any changes to models should only be done to meet some business need or
requirements. Any changes to models should be tested.

Inference and reasoning should be included in your models *only* to meet some
practical value-deriving business need.  If the inference does not deliver any
meaningful or practical value to your end users, it should not be there. 

If you can deliver a business requirement effectively and easily without
inference, do it without inference.

Your models should be as simple as possible to deliver your business
requirements. A SQL data architect would not just add additional random tables
into their SQL model to deliver their requirements, so when you develop a
practical ontology model for your domain, only model what you need to deliver
those requirements. You do not need to model extra stuff. 

While your domain model may extend well beyond the bounds of your business
requirements (in order to better inform requirements and business landscape),
your detailed ontology models should not. By only engineering the very minimum
of the model needed to deliver your current requirements, your platform is
easier to evolve, easier to maintain, and at a much lower total cost of
ownership.

Finally - end users should not have access to your production graph database
sparql or cypher endpoint. Never. 

Stop pretending you are witches. Start thinking like traditional data
architects.

Graph databases

Graph Databases are not Magic Cauldrons

A discussion on the synergies between GraphQL and Graph Databases, data architecture and how ontology models provide the special ingredients.

The synergies between GraphQL and Graph Databases

5 latin characters
Until recently, my understanding was that apart from 5 latin characters GraphQL
has got little or nothing to do with Graph Databases. But things are changing,
Graph Database vendors have started bringing these two technologies together. A
recent project has given me some insights.

For some background, GraphQL is a contemporary API interface standard used as an
alternative to REST. GraphQL was originally developed by Facebook. It provides a
structured query language for querying (and mutating) data, along with a runtime
implementation to invoke those queries upon your data. In pretty much all
implementations it uses JSON as its serialization format.

REST (I suspect most readers know this) is a lightweight low-level API standard
for performing uniform and a predefined set of stateless operations on data
aligned with the HTTP verbs. With REST you have to design and  build your API
interface contract for that set of operations - and while of course there are
common patterns in REST APIs, this is very much arbitrary. REST makes no
assumptions about the high level API contract or the underlying data model. The
developer defines both. With GraphQL you only define the data model (the GraphQL
schema), exposing to the GraphQL runtime and your consumers what the shape of
the data is.

With a GraphQL query you can request only the minimum data required to fulfil
your current need. Of course you can do this with a REST API too - in fact it is
a common pattern with JSON REST APIs to filter response data by JSON fields
using dot notation. But of course you need to build this yourself.

GraphQL does a lot more for you to boot, as well as out of the box query
patterns for mutating data, it also supports subscriptions (pub/sub style) - so
your GraphQL client can subscribe to server side events, and have data pushed to
it.

The developer ecosystem around GraphQL is superb now - The Apollo GraphQL
[https://www.apollographql.com] implementation works beautifully with React -
integrating with React Hooks for query and subscriptions. Apollo also has
in-browser developer tools akin to the redux-devtools for GraphQL debugging, and
you can manage local app state via Apollo using the GraphQL schemas / queries
etc.

Caching and performance
Requests to GraphQL are typically all POST requests - not great for caching you
ask ? Indeed, this is where REST has excelled for years. But the GraphQL server
implementations have come to the rescue. The Apollo GraphQL server has an
internal caching layer that supports full response caching, and fine grained
field-level caching (read more here
[https://www.apollographql.com/docs/apollo-server/performance/caching/] )

Amazon’s AWS AppSync is a fully managed GraphQL service built upon the Apollo
technologies - and can source data out of the box from other services such as
DynamoDB, ElasticSearch, REST APIs etc. With fully managed caching and real-time
data subscriptions, it really does make your developer life easy.

On the surface it still appears GraphQL has little to do with Graph Databases.
Yet increasingly we are hearing about how they should go together like bread and
butter, sausage and mash.

Not all graph databases are equal
I won't go into depth about what a Graph Database is, but am going to explore
some of the options available, and the relationship between Graph Databases and
GraphQL.

Graph databases come in several flavour combinations. First up you have RDF
graphs vs Labelled-Property graphs. The first big choice. Both are very
different [https://en.wikipedia.org/wiki/Graph_database#Graph_models]. Your
primary use case is important here. Essentially if you primarily need to walk,
query and analyse paths through your graph, then Labelled-Property graphs are
probably the way to go. If you are more interested in the nature of the
relationships between things, and creating rich semantic representations of
things, then RDF graphs hit the spot. Both types have their own well-developed
query language. Labelled-Property graphs are typically queried with Cypher or
Gremlin (eg Neo4J), RDF graphs are queried with SPARQL.

While most mainstream RDF based graph databases are fully compliant to the RDF
and SPARQL W3C standards, not all RDF graph databases are equal. Each has its
own implementation nuances and come with their own added extras (decisions,
decisions!):

 * An RDF graph database may or may not support reasoning (to some degree of the
   RDFS and OWL standards).
 * Reasoning may be at query time (backward chaining) or at write-time
   (forward-chaining).
 * The way reasoning is implemented can dramatically impact performance.
 * Add-ons and extensions for GeoSPARQL, Path analysis, Full text search,
   connections to other datastores etc.

To give some examples (DISCLAIMER - apols to those vendors not mentioned, I am
only referencing those I have worked with properly hands-on, and this is not an
exhaustive features list - it’s just a flavour) :

GraphDB [https://www.ontotext.com/products/graphdb/] (Ontotext)

 * Fully compliant with RDF/SPARQL standard;
 * Highly compliant and configurable OWL / RDFS forward-chaining reasoning
   (materialises inferred knowledge at write time);
 * Unparallelled full-text-search capability using lucene, or via bi-drectional
   connectors to external SOLR or ElasticSearch clusters.
 *  Ability to materialise post-inference fully structured JSON or JSON-LD
   objects into ElasticSearch indices or MongoDB
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI
 * UPDATED May 2022 - Path query capability is now avaialble using a SPARQL
   extension plugin

Neptune [https://aws.amazon.com/neptune/] (Amazon AWS)

 * Fully compliant with RDF/SPARQL standards;
 * No reasoning capability
 * Fully managed cloud service
 * Highly, highly horizontally scalable (primarily due to lack of reasoning)
 * Full text search capability via connector to ElasticSearch - materialisation
   into elastic is constrained (you cannot materialise user-modelled JSON
   objects into an elastic index), but delivers good full-text search utility.
 * You use this either as a Labelled-Property graph (using Gremlin) or an RDF
   graph (with SPARQL)
 * No query workbench/admin UI - you need to install your own 

Stardog [https://stardog.com]

 * Fully compliant with RDF/SPARQL standards;
 * Highly compliant and configurable OWL / RDFS backward-chaining reasoning
   (materialises inferred knowledge at query time);
 * Extremely fast online write times (due to backward-chaining model)
 * Adequate full-text-search capability using internal indexing
 * An outstanding SPARQL extension for PATH analysis (Pathfinder) - you can
   query for paths, shortest paths with weighted edges, normally only possible
   with labelled-property graphs
 * Geospatial querying and other extensions..
 * Rich visual query and management workbench UI

So you can see that each of these three are fully W3C compliant RDF / SPARQL
graph databases. But each has one or more killer features, that the others do
not. 

GraphDB - the post-inference connectors to ElasticSearch and MongoDB unlock some
outstanding technical and data architecture patterns out of the box -UPDATED May
2022 - GraphDB now also has comprehensive Path query capability using sparql
extension plugin

Stardog - combines the semantics of RDF with sophisticated Path querying and
analysis.

Neptune - fully managed cloud service with unparalleled horizontal scalability,
but no reasoning.

The upshot - choose wisely based on your requirements now, and what is likely to
matter most in the medium term.

..and then enter stage right, GraphQL

Sausage and Mash or Pineapple on Pizza ?
How do GraphQL and Graph Databases fit together? As we learned earlier, GraphQL
is a REST API alternative with a structured query language combined with a
runtime for query processing and serving data to GraphQL clients (and it is
almost always implemented with JSON in mind).

So why are Graph Database vendors getting involved and what do you get for your
money?

The cynical marketer in me says those 5 latin characters definitely count for
something. Which sales manager wouldn’t love the potential energy pressure in
the ready-to-explode singularity of Graph Databases and GraphQL (while humming
‘Don’t stop me now’).

However, GraphQL does only provide another query mechanism for your graph
database alongside SPARQL or Gremlin.  Moreover it is much less expressive, and
most definitely not designed for exploiting the rich semantics of RDF and OWL. 

In fact if you go all in on a michelin-starred Graph Database, and only use
GraphQL to query it, I would argue you really probably don’t need a Graph
Database - a NOSQL JSON store would probably suffice. Of course, if you have
forward-chaining inference in your graph database (as with GraphDB), then your
GraphQL queries will benefit from the RDFS/OWL semantics you have modelled your
data with, which does add weight to the value-proposition.

Still hungry ?
Ontologies are an important part of the picture. In the RDF world an ontology
semantically describes the model and nature of your domain, and ultimately the
data structures stored in your knowledge graphs, and if inference-aware, they
describe the meaning of the data, and how we reason upon it. Typically these
models are communicated up and down our tech stack (and the responsible
developers), and often all the way out to consumers of your APIs (public or
enterprise-internal).  While we typically move data around using JSON the
linked-data aficionados will know well that if we do this using JSON-LD, we can
use the same ontology models from the knowledge graph (or a subset) for
describing the JSON-LD payloads we ship around outside of our graph database.

So do we need GraphQL to take advantage of these ontological graph models ?.  No
we don’t, we can do it with normal REST APIs (querying our graph with SPARQL)
and using JSON-LD payloads - but of course we have to build more API code to do
this.

With GraphQL though we can (we must) formally describe our data structures as
GraphQL schemas, and so we have a great opportunity here to align our GraphQL
schemas with those same ontology models that describe the data in our knowledge
graph -  this could be either a subset of our ontology landscape, or possibly
the superset, if we are using other GraphQL (served) data sources aside from the
data in the graph database. This is starting to look like very nice decoupled
data architecture with well structured cohesive data models, with possibly the
most efficient mechanism for consuming data in our web/mobile applications (a
GraphQL client).

It is looking more tasty
Adding GraphQL client interfaces to a Graph Database does make life easier - if
you are rapidly prototyping straight from the graph, or federating data into an
external GraphQL server there is a nice pattern for making a common set of well
described data models.

The Graph Database vendors have indeed jumped on-board the GraphQL sushi train
(and it’s not all about those 5 latin characters). Stardog and GraphDB (from
Ontotext) both offer GraphQL clients now.

At this point I was going to show an example of how GraphQL might be used to
query a Graph Database, but Jem Rayfield at Ontotext has already done the job in
his superb Star Wars blog post
[https://www.ontotext.com/blog/the-rise-of-the-knowledge-graph/] (if you have
got this far, then this is a must read).

Ontotext has taken the next logical step and created a ‘Semantic Object Service’
 that automatically generates GraphQL schemas bound to the underlying RDF data
model  in GraphDB, moving us towards the ontology ubiquity I described above,
where your domain ontologies are expressed in the knowledge graph, and across
your wider data-estate, including your GraphQL schemas. Nice work.

Jem concludes by saying :

> GraphQL is a very popular graph query language, well used, known and liked by
developers. The Ontotext Platform uses GraphQL to lower the barrier of entry to
knowledge graph data, whilst still providing the full expressivity and power of
SPARQL.
and this is a key message, which equally applies to Stardog and any other graph
database that may yet offer a GraphQL client. While GraphQL lowers the barrier
to entry, GraphQL gives you only as much as it is able to with respect to
querying and graph traversal;  to really unlock the power of the graph database,
you will need to use SPARQL.

But if you never plan to, and you are not interested in reasoning either, then
possibly you should ask yourself if you really need a graph database ?

What if you want to use GraphQL as the primary API technology in your stack, but
want to use SPARQL too to access the graph database (amongst other tech)?

A mixed data technology case study
In a very recent engagement, the Data Language team have been working on a Smart
Cities IoT project. The nature of the domain meant the type of data we needed to
deal with was very ‘graphy’ in nature, and the ability to describe our data
semantically and traverse a graph with SPARQL was of high utility. Inference was
less of a concern. The proof of concept for the project was brought together in
AWS with :

 * AWS MSK - a high performance Kafka message bus to consume, orchestrate, and
   distribute IoT sensor events
 * AWS Neptune to give us our scalable RDF based smart city knowledge graph
 * A set of JSON-LD REST API endpoints invoking SPARQL queries on the graph
   running in Kubernetes (EKS)
 * A suite of Kafka event processor consumers and producers deployed into
   Kubernetes mutating the graph, and emitting new events based on our business
   logic rules.
 * AWS AppSync - the GraphQL engine sourcing events and the JSON-LD REST APIs
   from Neptune, providing the primary developer interface to our client
   applications
 * A ReactJS built UI using GraphQL react-hooks to subscribe to real time events
   data, and to query data from the sensors and our knowledge graph (ultimately
   via expressive SPARQL queries).

A simplified view of the architecture looks essentially like this :


The kicker was modeling data across the entire stack using a suite of
ontologies. These ontology models describe our knowledge graph data, our IoT
message payloads in JSON-LD and were all federated in the primary application
API expressed as GraphQL schemas in AppSync. Building the ReactJS front end
using the Apollo GraphQL client and react-hooks was a very nice developer
experience.

Dessert
While GraphQL technically has little to do with Graph Databases, they are not a
perfect match like sausages and mash, GraphQL is just another API / query
technology, and not as expressive as the native graph database query languages
like SPARQL, but in itself is a high-utlity, very sweet technology.

There are some clear synergies, and do provide some of that warm comfort food
feeling when used together. GraphQL does give you an easily understood and very
developer friendly way into a graph database. Once you add the secret sauce of
domain ontologies describing both your knowledge graph and possibly even your
entire data-estate, expressed through your GraphQL schemas, then you have a very
nice pattern for a decoupled data architecture with well structured, nicely
communicated data models across your technology stack, and out to your consumers
too - making for efficient application development using the GraphQL ecosystem.
Get stuck in.

GraphQL and Graph Databases

Julian Everett

One the most important tasks for software architects in the early stages of any
technology delivery program is to reach “architecture conceivability”, or
consensus agreement with senior developers that the technical proposition could
actually conceivably be workable. Starting from a position of defensive
scepticism, it typically entails an outside-in narrative exploration of the
potential end-to-end program execution flows at a level of detail between the
Component and Code levels in the C4 Architecture Model [https://c4model.com/] 
for key use cases.

The granularity of the discussion will switch depending on the level of
technical risk entailed by the component. If it is providing a standard
capability that is self-evident to to those present how to code, then there is
no point doing the more detailed code discussion. If it is encoding anything
with a non-trivial level of technical risk, then dropping down to the next level
will be required until everyone is roughly on the same page about how it needs
to work. In this way it acts a bit like a code preview (as opposed to a code
review), functioning in the same way that a delivery prespective
[https://www.malotaux.eu/nrmc.php?id=prespectives] is more efficient than a
retrospective in terms of mitigating risk.

Once and only once this has been completed, you will have established baseline
viability for the program. The narrative described will be unquestionably wrong,
inefficient, full of flaws – that’s not the point. The point is that it will
represent a first, stuttering, throwaway sketch out which a simple, resilient
system design can eventually be evolved. Gaping holes and pointless layers of
cruft will already be visible. Beyond that, most of the selective pressures
driving that evolution will be created later by writing tests and the feedback
from coding itself. 

That’s all fine. However, until you’ve established architecture conceivability,
in reality you’ve got nothing. It is hard to over-emphasise the importance of
this point. Context and Container-level views are meaningless slideware (or at
very best, vague indicators of business process alignment) until they are backed
up by Component and Code-level views. That is solely how they gain their value,
by proxy as maps to decomposing and navigating the the Code-level views of the
platform. Without the code, they are essentially maps to nothing.

Technical Architecture

Architecture Conceivability

In this post we are going to look at some of the ways in which AI and data
science is impacting the practical delivery of technology solutions today. To be
clear from the outset, this will not include a discussion of future-looking
topics that have been getting coverage recently such as fully automated code
generation or algorithmic product development, etc. Rather, we will be looking
the impacts which are already happening right now, the consequences of which are
often just as significant albeit in different ways.

Strategic Alignment Model
To get a clearer idea of what is going on, it will be helpful to start by
reviewing the dramatic shift that has occurred recently within the discipline of
data engineering. In order to do that, we are going to use a version of Niel
Nickolaisen’s Strategic Alignment Model
[http://www.accelinnova.com/strategy.html], which is one of our most frequently
utilised tools in the inception phases of data programs.

It splits an organisation’s capabilities in two core dimensions:

1.     Whether or not they are market differentiating

2.     Whether or not they are mission critical core competencies

This then informs technology delivery strategy as follows:

·      Systems which deliver business capabilities which are mission-critical
market-differentiators you are going to need to build - as by definition there
will be no commoditised solution available yet otherwise it wouldn’t be a
differentiator - and you are going to need to build them very well whilst
effectively managing risk and uncertainty, as this is where your future revenue
streams are coming from (as a side note, this quadrant represents almost all of
our consulting work).

·      Systems which deliver business capabilities which are mission-critical
but not market-differentiators should be bought in. That’s because they are
generally commoditised solutions to common problems (ERP, payroll, invoicing,
etc), hence re-inventing those solutions is a pointless waste of money. In some
cases, organisations might have non-standard workflows which don’t match
commodity solutions, but in the long run it is almost always better to change
those workflows rather than build bespoke software.

·      Systems which deliver business capabilities which are not
mission-critical but are market-differentiating are generally good candidates
for partnerships, i.e. with businesses for whom the newly developed capabilities
will become core competencies.

·      Systems which deliver business capabilities which are neither
mission-critical nor market-differentiating should generally be avoided, as
their value is as best highly questionable.

Such analysis frequently provides very useful insight to organisations. However,
a critical point to remember is that markets are dynamic, so such
categorisations are never fixed. Indeed, it is interesting to see how
technologies traverse this diagram over time. The most common transition is from
market-differentiator to non-market differentiator as technologies mature: new
features and capabilities that were once enough to attract new customers and
generate additional revenue become widely adopted, turning into commodity
features that are required by any product attempting to enter the market or
protect its existing revenue streams. There are also examples, such as online
video within media organisations, which start out as non-mission critical
differentiators before becoming mission-critical differentiators and then
non-differentiating core competencies (as a side note, the other important shift
as functionality matures from differentiating to non-differentiating is the
associated switch in customer priorities from functional to non-functional
requirements – what matters for non-differentiating systems is
standards-compliance, portability and the avoidance of vendor lock-in, etc).

Data Engineering As Market Differentiator
So what does this have to do with data engineering? Well, the really interesting
thing about the recent history of data engineering is that is represents the
much less common example of a technology moving in the opposite direction –
upwards, from non-differentiating to differentiating:

Data engineering used to be the domain of the back office: of ETLs, PL/SQL
programming and cron jobs. It was one of the areas of software development most
insulated from the market pressures which shaped agile software delivery
techniques and lean product development. It was also one of the least impacted
as a result: rare indeed was it to find an ETL developer writing their code
test-first.

However, the revolutions in firstly data analytics and statistical software
development, and then in machine learning techniques, have radically transformed
the discipline and moved it front and centre in the new era of data-driven
product innovation. This has created a sudden critical demand firstly for access
to these new technologies, a need which has driven the recent adoption and
growth of data science as a commercial capability; and secondly for the software
development skills that allow innovative product ideas to be tested and iterated
rapidly under conditions of uncertainty without incurring crippling amounts of
technical debt.

One of the most prevalent organisational responses to these demands is for
data-driven product delivery programs to employ three discrete technology
functions:

1.     Data science teams: typically, recent PhDs with good theoretical
knowledge but little experience of commercial software development. They often
work in what is effectively an R&D function developing prototype implementations
in their preferred language.

2.     Software delivery teams: where these function best, they are most
typically staffed by experienced/senior developers well versed in extreme
programming and continuous delivery techniques and ideally who have worked
together previously (the importance of this last point is so often overlooked –
no matter how good the individual members are, if the team itself is new then
you are going to burn significant time and cash whilst relationships, trust and
delivery flow are negotiated and established
[https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development]).

3.     Machine learning engineering teams: who are responsible for taking the
outputs of data science and adapting them into something which can be integrated
by software delivery. I use the term “adapting” rather than “refactoring” here
intentionally, as due to the “specialised” nature of the data science algorithms
the aim is normally to add runtime tooling into a software execution wrapper
that allows the prototype implementation to be invoked concurrently or to handle
larger datasets. Most typically this will be suites of map-reduce and derivative
tooling (not least because these teams are also the most natural home for data
engineers who invested in Hadoop stack upskilling during the first data
analytics revolution).

Implications for Software Architecture
This differentiation is having profound, and most highly undesirable, results.
My colleague Fabio Colasanti has previously written persuasively
[https://datalanguage.com/features/data-science-is-about-building-software] 
about the consequences at the code level. In the rest of this article, we will
explore the implications for software architecture, and the resulting impacts on
the business value actually being realised by these programs of work.

Arbitrary Service Boundaries
Conway’s Law [http://wiki.c2.com/?ConwaysLaw] states that the architecture of
systems tends to reflect the architecture of the organisations that build them.
The most reliable way to mitigate this tendency is to create a single
cross-functional delivery team that includes domain experts and works as close
to customers as possible. That’s because the domain model and business
capabilities encoded into software create the channels along which information
flows and business value is then generated and delivered.

In the absence of such an integrated, cross-functional approach, the likely
outcome will instead be an architecture where service boundaries reflect the
arbitrary divisions of labour in the construction of the codebase. Is that
really so bad? Well, yes actually - it is. Once the code sequences required to
implement and integrate the business capabilities are defined, it becomes
immediately apparent that the natural execution flow is being critically
disrupted by meaningless service interfaces that require mapping layers,
additional infrastructure complexity, kill performance, undermine cacheability,
expose security vulnerabilities, and introduce a host of other avoidable
problems.

In doing so, they also undermine the chances of a successful delivery.
Successful technology delivery is ultimately a question of risk management: of
identifying risks and then managing or mitigating them so that their cumulative
impact is as low as possible. That means ruthlessly removing all avoidable
sources of risk and noise - so that the inherent risks of the market (changing
customer demand, competitor behaviour, etc) can then be observed, responded to,
and their associated opportunities exploited.

Self-generating Complexity
In response to the implementation barriers created by arbitrary service
boundaries, delivery teams then typically attempt to work around them via the
lowest friction route available. Rather than conducting a politically
unpalatable root cause analysis and proposing a simplified approach that
conflicts with the structure of the overall delivery program, it is easier to
try using additional software and infrastructure complexity to address the
problem. This results in an escalating feedback loop of bad design: not only is
the architecture unnecessarily complex already, but additional complexity is
then introduced to manage that complexity.

Unfortunately, the only way to break out of this loop is for some brave soul to
put their hand up and conduct a “survival of the fittest” design cull, where
every component of the architecture is ripped out and tested for critical system
viability until what remains is the minimal skeleton actually required to
deliver the baseline functionality.

Machine Learning as Universal Panacea
A more subtle variant of self-generating complexity relates to the application
of machine learning itself. Given the relative immaturity and vast potential of
the field of data science, there is now the additional lurking temptation to try
using ML to work around any problem rather than analysing and solving its root
cause. However, as Robert Frost said “the best way out is always through” – a
phrase which captures the essence of how to avoid incurring crippling technical
debt. Applied to data-driven product innovation, that means conducting patient
and precise analysis to identify underlying causes of problems and solving them
at source rather than building an ever-expanding shanty town of workarounds that
is liable to catastrophic and terminal failure.

Effective Mitigation Strategies
What techniques can be employed to mitigate against these problems and create a
reliable foundation on which to base your delivery? In our experience the two
most valuable tools are a domain model and sets of BDD scenarios (or terse use
cases) which describe the state and behaviour on which the anticipated business
value depends:

Domain Model: this is a collaboratively defined conceptual model of the problem
space shared between the subject matter experts and technical staff. Its key
importance is that it explicitly names the entities of the business domain and
their relationships to each other. In doing so, it creates a worldview and
language that is shared by all stakeholders/members of the delivery team (a
purposefully limited and relativistic scope, which is enough to ensure use cases
can be fulfilled whilst avoiding the many pitfalls and dead-ends of enterprise
data modelling). It is not a knowledge graph, ontology or database schema.
Rather it is something which SMEs might express as a frequently iterated
whiteboard diagram of labelled entities and connectors, and developers might
describe using an ERD or high-level class diagram. This foundation is what
allows code to be written in a way that precisely and unambiguously matches the
collective intent of the team. Without it, you are building on quicksand.

BDD Scenarios: once a shared language has been established which describe the
entities in the domain and how they relate to each other, the state transitions
involved in the information flows and generated business value can then be
captured (in practice this tends to be highly iterative, with new scenarios
being sought that break the model and thereby drive its improvement). The
emphasis here is not on automated testing capability – although that can be a
very helpful by-product - but the communication of state transitions and
workflows [https://lizkeogh.com/2013/10/24/bdd-before-the-tools/]. 

Once the workflows and information flows have been defined via which business
value will be delivered, questions concerning the optimal deployment of human
and machine agents can then be correctly and meaningfully addressed. At this
point, it is then really important to return to the Strategic Alignment Model
for one more pass. Newly equipped with a powerful model that clearly describes
the state/data and behaviour of your domain, you can review each entity and
function in turn to create an unambiguous and precise definition of what your
market differentiators really are. Is it really the bespoke algorithm that your
data scientists are working on, or is it in fact the data that the algorithm is
outputting? If the latter case, is it really justifiable to keep investing in
developing something in-house which ought to be (and will almost certainly end
up eventually) a commodity purchase? These considerations will then be primary
factors informing product roadmap planning – a subject beyond the scope of this
post.

The result should be a system architecture that is the simplest thing possible
[https://www.jamesshore.com/Agile-Book/simple_design.html] to deliver the
identified business value, and as a result is the cheapest to build, contains
the minimum number of things that can go wrong, and is fastest to adapt to
future market change.

AI is a core discipline at Data Language. We have an expert team skilled in machine learning and deep learning, and importantly we also have the engineering and deployment skills to deliver AI at scale for your production systems.

AI is a core discipline at Data Language: Our expert team is skilled in machine learning and the engineering skills needed to deliver AI at scale.

Artificial Intelligence delivered at scale - our core discipline.

Artificial Intelligence

Current impacts of AI on Technology Delivery

Over the last couple years, one of the more interesting architectural patterns
to have emerged on deliveries for our media customers has internally acquired
the name ‘daisy architecture’. It evolves as organisations acquire the ability
to scale their premium content production processes via the use of machine
learning, text analytics and annotation techniques, and then extend the coverage
of that tooling over as much of their remaining content geography as possible.
In workflow terms, this entails a switch from content production pipelines that
are a.) push-based; b.) manual and c.) linear; to content enrichment cycles that
are a.) pull-based; b.) automated; and c.) on-demand and iterative:



Enabling factors
The pattern is typically the result of three technical enablers:

 1. The increasing influence of Kanban on software design, (i.e. principles
    taken from how applications are delivered and applied to how they work, in
    particular pull-based systems and queue balancing/optimisation techniques).
 2. The scaling of automated content analysis and rich metadata creation via
    text analytics and machine learning.
 3. The adoption of micro-service architectures, allowing business functions to
    be recombined much more easily to serve the individual needs of specific
    product contexts.

1. Kanban
The influence of Kanban techniques not only on the process of application
delivery but on the outputs of application design (i.e. on business processes
implemented in software) is arguably a return to its roots in some ways. We
perceived that as a direct transfer of principles, but what also seems to be
going on here is that the influence of Kanban is spreading out of software
delivery into business operations more generally (we are definitely seeing
increased adoption of pull-based value streams measured via cycle time and queue
buffer monitoring) and that is then being reflected back into the styles of
software design that are required to support those pull-based business
processes.

2. Machine Learning
There has been without doubt an aggressive increase in the use of machine
learning and AI techniques within media organisations to automate content
metadata creation and indexing (see our machine learning cloud service
[https://datalanguage.com/tagmatic] for a typical example). Critical differences
between automated and manual task processing are firstly that automation has
essentially negligible operational cost, and secondly it has a larger
variability in the quality of its output. This has a major impact on the
economics of workflow design. Historically, content production workflows were
designed as linear pipelines because every step was manual and therefore costs
were within similar boundaries of magnitude. Once certain steps incur negligible
cost however, new opportunities open up to process much larger content volumes
at basic quality levels. From a brand/product risk management perspective, this
then needs to be complimented by quality assurances steps, i.e. manual review.
Those manual steps are where the costs are incurred, which is why it is highly
desirable for them to do so just-in-time i.e. once product opportunities have
been targeted which clearly outweigh that investment.

Interestingly, for these reasons we are seeing a significant impact of
automation on changing job roles in this space, but a negligible impact of
automation on actual jobs - apart from some impact to very repetitive,
rule-based work (i.e. a prime candidate for a machine) that has most often
already been outsourced. Instead we are seeing job roles switching from
production at lower volumes to the curation of production at higher volumes.
This typically takes the form of:

a.Quality Assurance of machine output, the results of which can then be used as
ongoing training data to refine machine statistical models.
b.Sampling Tuning. Not all content is equal: the quality of machine output will
depend on both its source and its subject area (e.g. the simple factual
statements of a football match report will typically be easier for a machine to
analyse successfully than an in-depth opinion piece). That means different QA
sampling rates will be required for content in different subject areas and from
different sources: e.g. new syndication source vs in-house editorial, and those
rates will then need to be managed by people over time: e.g. as the new
syndication source becomes a trusted partner they will need less manual
oversight so the sampling rate should be reduced.
c.Terminology management: to ensure that controlled vocabularies are used to
encode the core entities and relationships of the business domain thereby
maximising data interoperability across the organisation. (This is one of the
core benefits of using domain-driven design
[https://datalanguage.com/services/information-architecture] to define the
information architecture)

3. Micro-services
The adoption of micro-service architectures has unlocked access to discrete
units of business functionality, allowing them to be recombined in novel ways
according to the emerging needs of new product contexts. This doesn’t mean a
return to the dark days of untestable spaghetti service orchestrations in
black-box ESBs. Choreography logic should still be pushed into endpoints (or
else into a standalone endpoint if justified), but it should be implemented as a
strategy pattern – where the content enrichment strategy is determined by a
specific product need rather than hardcoded.

Services and Routing
Product-specific service choreography basically means having a routing service
that can execute an enrichment strategy which encodes the content requirements
of a specific product context (for example: it needs subject classification
metadata, time-coded video scene metadata, advertising block-word annotations,
etc). Doing this just-in-time however means that you only know the state of the
required output, not the input state (as you don’t know which enrichment
operations might already have been performed). This has two interesting
consequences:

 1. Routing tends to be both expressed and executed in terms of gap analysis:
    i.e. perform whatever enrichment operations are required to get content
    selection C into state P needed by my product.
 2. Services tend to be partitioned into enrichment services and content
    search/persistence services, with the latter frequently needing to support
    terse query semantics that can be either named or URL encoded and passed
    between other services.

The final point worth mentioning concerns the temptation to implement aggregate
AI solutions, i.e. replacing a workflow that is currently 12 manual
steps/functions with a single neural network solution or similar. The argument
goes that machine systems are fundamentally different from human systems hence
there is no need to decompose them into small steps: you can just train from
required outputs against known inputs and get rid of all the complexity. The
first point to note here is that the complexity isn’t being removed, it’s just
being hidden. Moreover, it is being aggressively hidden as anything encoded in a
neural network is semantically opaque: it is essentially just set of weightings
that produces a desired output, which means there are no conceptually
interoperable subcomponents that can be decomposed back out into something
actionable by humans if anything starts going wrong. Also error margins will
almost certainly be amplified across a monolithic system that has no
intermediary review and correction checkpoints. For these reasons our advice is
to steer clear of AI monoliths. Instead we recommend you continue modelling your
business processes as discrete, atomic functions, and then leave the decision of
human vs machine vs human-assisted machine to be an implementation concern only.
The key task is to ensure your systems are then designed to allow that decision
to be revised according to changing circumstance with minimal operational
impact.

The next generation of adaptive businesses will be those that can swap between
outsourced, insourced, human, machine and human-assisted machine implementations
of their core business functions with minimal impact to their ongoing
operational capabilities. That will be the true test of how cohesive and
loosely-coupled the service-based architectures of today really are.

Daisy Architecture

I recently finished up a lead architect engagement on a very large semantic
publishing implementation for a finance client in the City of London. Digging
out some old notes, I noticed that the delivery has been bookended by
reflections on the pain caused by httpRange-14.

This problem has been debated amongst some very smart people for ten years plus
now without any generally accepted solution being forthcoming, and more recently
I have begun wondering whether the real issue is more a function of the question
itself. So as an experiment, at the start of the program I tried to take a step
back to see if a different perspective might add some clarity. In doing so, I
decided to revisit a book that I last read many years ago, Saul Kripke’s seminal
text Naming and Necessity [http://en.wikipedia.org/wiki/Naming_and_Necessity].

The end result was a draft blog post I wrote early last year that somewhat
optimistically finished with the sentence “It will be interesting to see if
there are any obvious flaws I have overlooked, and what challenges are thrown up
once we proceed to prototyping the solution.” It prompted a lot of interesting
internal discussion at Datalanguage but in the end we decided not to publish
because, as Paul pointed out, the use of blank nodes on which the proposal
depended is problematic for a number
[http://milicicvuk.com/blog/2011/07/14/problems-of-the-rdf-model-blank-nodes/] 
of reasons [http://answers.semanticweb.com/questions/8336/what-is-skolemization]
.

The follow-up I wrote a couple of weeks ago, and it is set out below. It evolved
out of my client handover notes on URI strategy but quickly turned into another
discussion about httpRange-14. Somewhat encouragingly there has been clear
progression in my thoughts particularly with regard to URI binding and
httpRange-14 itself, about which I’ve now reached some concrete conclusions
which seem both coherent and practically viable.

Resource Identity and httpRange-14
As practitioners of Domain Driven Design, one of the early steps we perform in
the modelling of any new domain is the classification of its resources
[http://martinfowler.com/bliki/EvansClassification.html] into entity types and
value types. This is a question of identity and context:

 * Value types are types which are best modelled as data: their “identity” is
   determined by their attributes (and for this reason, in code they are
   generally implemented as immutable). An example might be Postcode: in most
   contexts, changing the string value of a postcode simply means you have a new
   postcode.
 * Conversely, Entity types have an identity beyond their total set of
   attributes. An example might be Person, where - in most contexts, at least -
   it makes sense to model someone as the same person regardless of which
   attributes might change over time (for which reason, a domain where it could
   conceivably make sense to model Postcode as an entity type is workflow
   management of postcode creation and allocation).

This has an impact on URI strategy. Entity types typically require
synthetic/machine identifiers such as GUIDs, Twitter snowflake IDs or similar,
whereas value types will normally use the text or hash value of their
identity-conferring attributes. So in the examples from Jeni Tennision’s
discussion of Opaque URIs [http://www.jenitennison.com/blog/node/114]:

http://id.example.org/house/NG9_3HZ/4
http://id.example.org/house/0aef0218


A domain driven design practitioner might agree that whilst the former has
advantages in terms of hackability, it should only be used in addressing
contexts. If your domain resource is really encapsulating the identity of
physical buildings then you should adopt the latter strategy. That way, your
design will be resilient to high impact edge cases such as house numbers or
postcodes being reallocated. In short,

> when designing URIs, start by analysing resource identity. Once you understand
the > nature of the nodes in your graph, you can implement resource
identification with > some degree of confidence and then move onto modelling the
arcs.
Putting this into practice, a helpful way of resisting the urge to jump straight
into traversing the graph is to start by modelling your domain using ERD
techniques and only switch into linked data graphical modelling once you’ve
nailed your URI strategy.


As regards URI structure, a strictly minimalist policy has a lot to commend it.
In this respect URIs are like code: the less it takes to achieve your goal, the
higher the likelihood of good design and the less there is to go wrong. “If in
doubt, leave it out” definitely applies, and it is then generally worth testing
your confidence in everything that remains as well. The result is normally
something that looks a lot like:

http{s?}://{authority}/{uid}


where

{uid} = {namespace?}{id}


and

{namespace} = the minimal virtual path prefix required to ensure uniqueness within the authority's hostname/port naming scope


For synthetic identifiers a namespace is rarely required, apart from the case of
incremental integers which should normally be avoided for other issues relating
to distributed synchronisation. In the instance where a namespace is necessary,
then the chosen virtual path should either be meaningless or else an intrinsic
and version-independent attribute of the resource, such as type alias/label. The
latter is not an entailed requirement for opaque URIs, but on purely aesthetic
grounds it still seems deeply ugly to me to have URLs such as the above
dereference to things which aren’t in fact representations of houses.

One of the more interesting consequences of this view of resource identity
relates to how reference is assigned. If resource identity is not a function of
attribute values, then clearly identification (i.e. what a URI refers to) can’t
be determined by attribute or property matching. The main alternative model of
reference determination was set out by Kripke in Naming and Necessity
[http://en.wikipedia.org/wiki/Naming_and_Necessity] whereby proper names act as 
rigid designators [http://en.wikipedia.org/wiki/Rigid_designation] the
denotation of which is set by a one-off “baptism event” at the point of
creation/adoption within a social group. By way of summary for those unfamiliar
with the work, Kripke proposes something of a return to John Stuart Mill in
taking on the previously dominant descriptive theories of naming that had been
advocated by Bertrand Russell and Gottlob Frege amongst others (interestingly,
the distinction between sense and content that Jeni Tennison makes in her post
about punning and httpRange-14 [http://www.jenitennison.com/blog/node/170] has
echoes of Frege). Kripke objects to descriptive theories of naming on the
grounds that giving meaning and fixing reference are not the same thing.
Instead, Mill’s position that names have “denotation but not connotation” -
which seems an excellent description of opaque URIs - is developed into the
notion of names as rigid designators, which are effectively opaque identifiers
whose references are bound at the point of social adoption.

Such a viewpoint seems consistent with experience. URIs refer to resources
because social groups (i.e. those performing the URI authority function) decide
they refer to those resources: where those URIs are URLs, that is reinforced by
the behaviour of DNS resolvers, but in the instance that a URL temporarily
dereferences to a wifi login prompt
[http://www.jenitennison.com/blog/node/168#comment-11729] that doesn’t signify a
change in meaning. Isn’t this a contradiction of the notion that URLs are late
bound? No, because the thing they late-bind to is representation, not the
resource/conceptual mapping - which is in fact early bound by the authority for
that URI at point of creation. What about the late binding enforced by the
constraint on using hypermedia as the engine of application state? Again no
contradiction, as that concerns late binding of client to URI rather than
resource identifier to concept:


The notion of resource identity also has important implications in one other
area. For the last couple of weeks I’ve been conducting a technical
retrospective and handover to the client’s architecture team on one of the
largest semantic publishing implementations in the UK right now. Part of this
has been a review of URI strategy and the decisions made around httpRange-14
[http://www.w3.org/2001/tag/group/track/issues/14] (or httpRedirections-57
[http://www.w3.org/2001/tag/group/track/issues/57] as it has become) . Due to
previous experiences of the pain entailed in coding clients against fragment
identifier URIs (namely: the ambiguities introduced about whether a given
fragment identifier denotes a document fragment or an NRI; the problems certain
HTTP request processing frameworks have with fragment identifiers; the lack of
portability to other better solutions that might be devised in future), this
time we chose to adopt the 303 strategy. In doing so, we avoided the ambiguities
and additional programming complexities but at the expense of performance due to
all the additional, and worse uncacheable
[http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html] (a bug in the RFC now
being addressed as part of httpBis
[http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-24#section-6.4.4])
303 roundtrips. For something that seemingly generates no business value to most
customers - i.e. anyone who doesn’t have concrete use cases which depend on a
formal distinction between information resources and non-information resources
being maintained - both strategies [http://www.w3.org/TR/cooluris/] are
definitely a very long way from ideal.

In preparation for the technical retrospective sessions and having now been
bitten by both approaches, I took the opportunity to revisit the recommendations
with a more critical perspective. In doing so, I arrived at the realisation that
there are actually multiple interpretations of the term “resource” in use within
the web community right now. I decided to swap a domain-driven approach for a
constraint-driven approach - the constraint in question being that I’m not using
either 303s or fragment identifiers any more because they both suck :-) - to see
if any of those interpretations offered an alternative way of thinking about the
domain which a.) avoids the problem and b.) is consistent with the way the web
works right now. The use of domain modelling was limited to prioritising the
analysis of those interpretations: by starting from the interpretation of
“resource” which most closely models how language and identity works in the real
world, then there was a good chance that the hard part of actually solving the
problem could effectively be left to Natural Selection meaning scope would be
limited to modelling only:



The diagram above describes how we assign identity as we perceive the world. Our
retinas detect a continual, undifferentiated stream of photons reflected from
the environment around us, which we then filter and interpret using mental
models or templates. For example, I have a mental concept of my cat, and that
allows to me to assign continuity of identity to any photon stream (e.g. cat in
kitchen, cat climbing tree) which I can map to that concept. When I assign a
name to that model, participate in a social group and there is enough
commonality between our named mental models, then language becomes possible:



Now let’s extend the first diagram to include mediated perception:


This shows a sliding scale - from direct perception, live MPEG4 video broadcast,
MP2 accessible audio description, web Flash streaming to web-published text – of
different formats in which the essential information content of the same real
world entity is encoded. Translating the web examples into the terminology of
HTTP, the name N is a URI, the mental model M’ is a resource and the encoded
information formats are representations. This corresponds very encouragingly to
the definition of resource as defined in section 1.1 of the older URI spec RFC
2982 (which has unfortunately been superseded by the open, and arguably less
helpful, definition in RFC 3986: the term “resource” is used in a general sense
for whatever might be identified by a URI):

> A resource can be anything that has identity… Not all resources are network
”retrievable”; e.g., human beings, corporations, and bound books in a library
can also be considered resources. The resource is the conceptual mapping to an
entity.


This interpretation of the term resource I’m going to label concept-centric.
In contrast, the interpretation of resource implied by the Cool URIs
[http://www.w3.org/TR/cooluris/] recommendation I’m going to refer to as 
document-centric: although section 2.1 describes the distinction between
resource and representation and its consequences in relation to content
negotiation, it is apparent from the diagram in the first part of section 3 that
each document has its own URI and is therefore a resource. In section 3.1 this
is then expanded into the requirement for either 303s or fragment identifiers,
on the basis that any real world “Non-Information Resource” needs a
corresponding web document “Information Resource” which describes it:



The concept-centric interpretation views the distinction between Non-Information
and Information Resource differently, and of essentially no value as both
variants are functionally identical:



This view says that RDF and HTML documents are representations, not resources.
They only need URIs in the instance that the separate concept of the
representation’s byte stream is also required. In such cases a Content-Location
header should be exposed with the URI for that concept. On the other hand, a
description of a real-world entity is a resource, because a description of
something is conceptually different to the thing itself. The description’s URI
should simply dereference to some kind of representation in exactly the same way
as a URI for a real-world resource. For example, URIs denoting a.) London and
b.) a description of London, just work as follows:



So, adopting a concept-centric rather than document-centric interpretation of
“resource” has the following consequences:

 * URIs are late binding in all aspects except denotation: what a URI refers to
   is determined at creation time by its naming authority rather than at runtime
   via a dereferenced representation.
 * As with new terms introduced into natural language, ambiguities or
   misunderstandings that arise in the early adoption of a newly published URI
   should be resolved via recourse to the publishing social group – i.e. the
   URI’s naming authority.
 * There is no non-trivial difference between an Information Resource and a
   Non-Information Resource: URIs simply denote concepts in the real or
   imaginary universe, and optionally derefence to one or more informationally
   equivalent representations.
 * A representation is not a description: a representation is a byte stream
   whereas a description of a resource a distinct concept in its own right.
 * A URI which denotes a representation (i.e. via a Content Location header)
   denotes the concept of a byte stream , not the byte stream itself.
 * 303 response codes should be used solely for standard See Other semantics.
 * Fragment identifiers should be used solely to denote resource parts.
 * No change is required from how the web works right now.

In other words by slightly changing our perspective on the problem, it
essentially disappears.

What about the scenarios where a naming authority has not published any/adequate
information to allow URI consumers to disambiguate denotation to the extent
required for the consumers’ use cases? Simple: in the absence of any better
alternatives, consumers should mint their own unambiguous URIs that do meet
their needs. My guess is that this will only be an issue for a few naming
authorities anyway: either those participating in the linked data cloud who are
currently larger enough to be used for general purposes (e.g. dbpedia,
musicbrainz, etc) or else those organisations which have strategically targeted
niche verticals where they perceive there to be longer term commercial
advantages from owning the dominant vocabulary. It seems reasonable to request
of such organisations that in order to realise their goals they conform to a
more formal definition of resource that is consistent with the demands of
semantic reasoners.

In addition to the clear pragmatic benefits of adopting a concept-centric notion
of resource, I then discovered that such a position also seems to be consistent
with the views of one of the main authors of HTTP. That finally sold me on the
viewpoint. In a typically entertaining and strong post
[http://lists.w3.org/Archives/Public/www-tag/2002Jul/0090.html] from Roy
Fielding to the TAG mailing list dating back to 2002 - worth a full read - he
makes the following comments:

> Resources are not transferred, just as identity within the real world
is not transferred when it is accessed. That doesn't stop us from
reasoning about rocks, plants, chairs, or my dog. In fact, the
separation of identity from representations of an identity is
necessary to reason about them at all. That does not mean that the
resource and the representation are both identified by the same URI;
they are not the same resource. It means that you can reason about
resources and reason about representations of resources, even if you
don't know "the most appropriate URI" that does identify the
representation as a separate resource.


> In short, the only reason this gives anyone in RDF land heartburn is
simply because their definition of resource doesn't match that of the
Web, or that of reality.


> I absolutely refuse to consider RDF as a useful technology or the
Semantic Web as the future of human communication if its reasoning
power is incapable of describing one of the fundamental facts of life,
particularly since the only reason it is incapable of doing so is
because a few people suffer from the unfortunate belief that it is
easier to change the Web than it is to change their preconception
of what it means to be a resource in RDF. Change that preconception
and RDF becomes capable of reasoning about both resources and
the representations of resources through one level of indirection,
just like the rest of us mere mortals.


To Jeni Tennison’s observation [http://www.jenitennison.com/blog/node/170] that 
“things recently make me more convinced than ever that the TAG must either
provide some direction to the community, and soon, or get out of the way”, my
view is that the minimum/sufficient direction is as follows:

 * The TAG should provide an unambiguous definition of “resource” to be used by
   anyone building reasoners. This definition should be concept-centric.
 * The TAG should provide a general advisory to anyone publishing linked open
   data that in the instance they wish their data to be used by consumers for
   more than simple graph traversal then they need to understand fully, and
   conform to, the definition of “resource” being implemented in reasoners.
 * The TAG should deprecate Cool URIs [http://www.w3.org/TR/cooluris/]

I think they should then stand back and let normal evolutionary drivers take
care of the rest. It seems unlikely that any organisation whose business cases
depend on the type of market-differentiating graph hops made possible by the
power of semantic inference is going to get their notion of “resource” mixed up
more than once.

Technical Architecture

Technical Architecture Blog Posts