Julian Everett

Over the past five years it has been our privilege to be a technology partner of 
Cochrane [https://www.cochrane.org/]. For anyone unfamiliar with their work,
they are globally recognised as the gold standard in evidence-based healthcare.
Historically their primary published output has been systematic reviews, which
evaluate and summarise the current state of human knowledge about specific
clinical questions. These then inform both healthcare policy and clinical
guidelines for point-of-care decision making around the world. For example, over
90% of WHO guidelines are informed by Cochrane reviews. 

The review production process starts with the formalisation of the clinical
question of interest in terms of the Population it relates to (e.g. age, sex,
condition), the Interventions being investigated (e.g. drugs and/or procedures),
the Comparators against which the Interventions are being evaluated (e.g.
placebo), and the Outcomes against which effects are measured. This is then
translated into search protocol and run against trial registries and journal
databases to identify all relevant clinical study information in the public
domain. The matches are then evaluated for risk of bias, and the subset of
results which meet Cochrane’s stringent data quality thresholds are subsequently
aggregated and analysed to determine the cumulative effect estimates. 

In essence, this is about the identification of causal relationships between
interventions and outcomes. To that end, up until now Cochrane has focussed on
double-blind randomised controlled trials. They provide the highest quality
input data: randomisation greatly reduces the likelihood of alternative
explanations for the observed results as the number of participants increases;
blinding of field researchers and participants (i.e. hiding information about
specific interventions) removes the cognitive and behavioural bias risks that
might arise if, for example, placebos are known. 

Finally, review production is not a static process. New information is being
released into the public domain all the time both in the form of additional
journal articles about completed studies as well as information about new
studies. As a result, ongoing evidence surveillance and timely meta-analysis
updates are important additional requirements.

During our time working with Cochrane, we have built a new data platform using
domain-driven design techniques which has an explicit information model encoded
into a knowledge graph at its core. The model is published here
[https://data.cochrane.org/ontologies/]. This is complemented by a loosely
coupled, service-based architecture which ensures each business capability in
the value stream is cohesively encapsulated. In a follow up article, my
colleague Paul Wilton will review how the technical architecture hangs together
[https://datalanguage.com/blog/how-knowledge-graph-technology-is-helping-cochrane-respond-to-covid-19]
. The platform design allows services to be very quickly deployed in new
choreographies as the business context changes - something which has proved
invaluable over the last few weeks. 

So, what have been the business function and information flow impacts of the
Covid-19 pandemic on Cochrane? Firstly it has created an unprecedented critical
demand for the most up-to-date, reliable evidence to inform frontline clinical
decision-making around the world in order to save as many lives as possible. To
that end, they have been asked by the WHO to produce a precisely scoped,
authoritative register of Covid-19 studies. This essentially inverts their
business model from one based on a wide coverage of clinical questions using
only the highest quality primary evidence to one based on a single clinical
question which urgently extracts every bit of informational value from every
piece of evidence available. That means using all types of RCTs and
interventional studies, as well as observational studies, modelling studies,
diagnostic/prognostic studies and anything else that might improve outcomes. The
surveillance requirement for updates and new evidence switched from months to
days. The cycle time for systematic review production needed to be cut, and a
new Rapid Review format has been developed as a result to ensure baseline
datasets can be pulled together as quickly as possible and then augmented with
streamlined editorial content to meet clinical need. At the same time, the
feedback loop to frontline workers was closed to ensure that their highest
priority clinical questions are the ones being addressed first.

The Covid-19 Study Register [https://covid-19.cochrane.org/] was launched
internally on Monday 30th March and publicly on Monday 6th April. In all, two
back-end platforms were integrated together and a new front-end web application
was designed, built and shipped in just under 3 weeks. It has necessarily been
our fastest ever delivery by some distance, and was only even remotely possible
because of three core enablers:

 1. A data architecture, technical architecture and continuous integration
    infrastructure which minimised the technical risks and impacts of adapting
    to the changing organisational context.
 2. A small number of individually, amazingly talented developers.
 3. The tacit/emergent team structure and pre-existing relationships between
    those developers, established over a long track record of successful
    delivery on other demanding client projects, which meant that as a delivery
    unit they functioned at a truly optimal level from day one.

Since internal launch, our focus has been on getting as much data as possible
into the information model by any and every means we can think of, as quickly as
possible and at the highest possible quality levels. This is covering expert
crowd annotation, machine learning, and other text analysis and informational
reverse engineering techniques. One of the biggest automation/ML challenges
currently is the lack of training data due to the rapid emergence and rate of
evolution of the Covid-19 domain. We have previous experience of this class of
problem from our work with media newsrooms and automated classification of
breaking news stories. It is likely the crisis will drive extensive further
innovation in this space. 

At the same time, we have started receiving comments from stakeholders and end
users across the organisation. It has been some of the most inspiring and
touching feedback we have ever received both as an organisation and as
individuals, especially in light of the wider situation right now. The high
point so far was one very senior Cochrane staff member who, in wonderfully
memorable terms, said he couldn't overstate how useful the register is to all
aspects of his work and that he's now going to shout about it to everyone who
will listen.

A beautiful moment in the midst of demanding times.

Technical Strategy

Covid-19: Rapid Response Technical Delivery at Cochrane

One the most important tasks for software architects in the early stages of any
technology delivery program is to reach “architecture conceivability”, or
consensus agreement with senior developers that the technical proposition could
actually conceivably be workable. Starting from a position of defensive
scepticism, it typically entails an outside-in narrative exploration of the
potential end-to-end program execution flows at a level of detail between the
Component and Code levels in the C4 Architecture Model [https://c4model.com/] 
for key use cases.

The granularity of the discussion will switch depending on the level of
technical risk entailed by the component. If it is providing a standard
capability that is self-evident to to those present how to code, then there is
no point doing the more detailed code discussion. If it is encoding anything
with a non-trivial level of technical risk, then dropping down to the next level
will be required until everyone is roughly on the same page about how it needs
to work. In this way it acts a bit like a code preview (as opposed to a code
review), functioning in the same way that a delivery prespective
[https://www.malotaux.eu/nrmc.php?id=prespectives] is more efficient than a
retrospective in terms of mitigating risk.

Once and only once this has been completed, you will have established baseline
viability for the program. The narrative described will be unquestionably wrong,
inefficient, full of flaws – that’s not the point. The point is that it will
represent a first, stuttering, throwaway sketch out which a simple, resilient
system design can eventually be evolved. Gaping holes and pointless layers of
cruft will already be visible. Beyond that, most of the selective pressures
driving that evolution will be created later by writing tests and the feedback
from coding itself. 

That’s all fine. However, until you’ve established architecture conceivability,
in reality you’ve got nothing. It is hard to over-emphasise the importance of
this point. Context and Container-level views are meaningless slideware (or at
very best, vague indicators of business process alignment) until they are backed
up by Component and Code-level views. That is solely how they gain their value,
by proxy as maps to decomposing and navigating the the Code-level views of the
platform. Without the code, they are essentially maps to nothing.

Technical Architecture

Architecture Conceivability

In this post we are going to look at some of the ways in which AI and data
science is impacting the practical delivery of technology solutions today. To be
clear from the outset, this will not include a discussion of future-looking
topics that have been getting coverage recently such as fully automated code
generation or algorithmic product development, etc. Rather, we will be looking
the impacts which are already happening right now, the consequences of which are
often just as significant albeit in different ways.

Strategic Alignment Model
To get a clearer idea of what is going on, it will be helpful to start by
reviewing the dramatic shift that has occurred recently within the discipline of
data engineering. In order to do that, we are going to use a version of Niel
Nickolaisen’s Strategic Alignment Model
[http://www.accelinnova.com/strategy.html], which is one of our most frequently
utilised tools in the inception phases of data programs.

It splits an organisation’s capabilities in two core dimensions:

1.     Whether or not they are market differentiating

2.     Whether or not they are mission critical core competencies

This then informs technology delivery strategy as follows:

·      Systems which deliver business capabilities which are mission-critical
market-differentiators you are going to need to build - as by definition there
will be no commoditised solution available yet otherwise it wouldn’t be a
differentiator - and you are going to need to build them very well whilst
effectively managing risk and uncertainty, as this is where your future revenue
streams are coming from (as a side note, this quadrant represents almost all of
our consulting work).

·      Systems which deliver business capabilities which are mission-critical
but not market-differentiators should be bought in. That’s because they are
generally commoditised solutions to common problems (ERP, payroll, invoicing,
etc), hence re-inventing those solutions is a pointless waste of money. In some
cases, organisations might have non-standard workflows which don’t match
commodity solutions, but in the long run it is almost always better to change
those workflows rather than build bespoke software.

·      Systems which deliver business capabilities which are not
mission-critical but are market-differentiating are generally good candidates
for partnerships, i.e. with businesses for whom the newly developed capabilities
will become core competencies.

·      Systems which deliver business capabilities which are neither
mission-critical nor market-differentiating should generally be avoided, as
their value is as best highly questionable.

Such analysis frequently provides very useful insight to organisations. However,
a critical point to remember is that markets are dynamic, so such
categorisations are never fixed. Indeed, it is interesting to see how
technologies traverse this diagram over time. The most common transition is from
market-differentiator to non-market differentiator as technologies mature: new
features and capabilities that were once enough to attract new customers and
generate additional revenue become widely adopted, turning into commodity
features that are required by any product attempting to enter the market or
protect its existing revenue streams. There are also examples, such as online
video within media organisations, which start out as non-mission critical
differentiators before becoming mission-critical differentiators and then
non-differentiating core competencies (as a side note, the other important shift
as functionality matures from differentiating to non-differentiating is the
associated switch in customer priorities from functional to non-functional
requirements – what matters for non-differentiating systems is
standards-compliance, portability and the avoidance of vendor lock-in, etc).

Data Engineering As Market Differentiator
So what does this have to do with data engineering? Well, the really interesting
thing about the recent history of data engineering is that is represents the
much less common example of a technology moving in the opposite direction –
upwards, from non-differentiating to differentiating:

Data engineering used to be the domain of the back office: of ETLs, PL/SQL
programming and cron jobs. It was one of the areas of software development most
insulated from the market pressures which shaped agile software delivery
techniques and lean product development. It was also one of the least impacted
as a result: rare indeed was it to find an ETL developer writing their code
test-first.

However, the revolutions in firstly data analytics and statistical software
development, and then in machine learning techniques, have radically transformed
the discipline and moved it front and centre in the new era of data-driven
product innovation. This has created a sudden critical demand firstly for access
to these new technologies, a need which has driven the recent adoption and
growth of data science as a commercial capability; and secondly for the software
development skills that allow innovative product ideas to be tested and iterated
rapidly under conditions of uncertainty without incurring crippling amounts of
technical debt.

One of the most prevalent organisational responses to these demands is for
data-driven product delivery programs to employ three discrete technology
functions:

1.     Data science teams: typically, recent PhDs with good theoretical
knowledge but little experience of commercial software development. They often
work in what is effectively an R&D function developing prototype implementations
in their preferred language.

2.     Software delivery teams: where these function best, they are most
typically staffed by experienced/senior developers well versed in extreme
programming and continuous delivery techniques and ideally who have worked
together previously (the importance of this last point is so often overlooked –
no matter how good the individual members are, if the team itself is new then
you are going to burn significant time and cash whilst relationships, trust and
delivery flow are negotiated and established
[https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development]).

3.     Machine learning engineering teams: who are responsible for taking the
outputs of data science and adapting them into something which can be integrated
by software delivery. I use the term “adapting” rather than “refactoring” here
intentionally, as due to the “specialised” nature of the data science algorithms
the aim is normally to add runtime tooling into a software execution wrapper
that allows the prototype implementation to be invoked concurrently or to handle
larger datasets. Most typically this will be suites of map-reduce and derivative
tooling (not least because these teams are also the most natural home for data
engineers who invested in Hadoop stack upskilling during the first data
analytics revolution).

Implications for Software Architecture
This differentiation is having profound, and most highly undesirable, results.
My colleague Fabio Colasanti has previously written persuasively
[https://datalanguage.com/features/data-science-is-about-building-software] 
about the consequences at the code level. In the rest of this article, we will
explore the implications for software architecture, and the resulting impacts on
the business value actually being realised by these programs of work.

Arbitrary Service Boundaries
Conway’s Law [http://wiki.c2.com/?ConwaysLaw] states that the architecture of
systems tends to reflect the architecture of the organisations that build them.
The most reliable way to mitigate this tendency is to create a single
cross-functional delivery team that includes domain experts and works as close
to customers as possible. That’s because the domain model and business
capabilities encoded into software create the channels along which information
flows and business value is then generated and delivered.

In the absence of such an integrated, cross-functional approach, the likely
outcome will instead be an architecture where service boundaries reflect the
arbitrary divisions of labour in the construction of the codebase. Is that
really so bad? Well, yes actually - it is. Once the code sequences required to
implement and integrate the business capabilities are defined, it becomes
immediately apparent that the natural execution flow is being critically
disrupted by meaningless service interfaces that require mapping layers,
additional infrastructure complexity, kill performance, undermine cacheability,
expose security vulnerabilities, and introduce a host of other avoidable
problems.

In doing so, they also undermine the chances of a successful delivery.
Successful technology delivery is ultimately a question of risk management: of
identifying risks and then managing or mitigating them so that their cumulative
impact is as low as possible. That means ruthlessly removing all avoidable
sources of risk and noise - so that the inherent risks of the market (changing
customer demand, competitor behaviour, etc) can then be observed, responded to,
and their associated opportunities exploited.

Self-generating Complexity
In response to the implementation barriers created by arbitrary service
boundaries, delivery teams then typically attempt to work around them via the
lowest friction route available. Rather than conducting a politically
unpalatable root cause analysis and proposing a simplified approach that
conflicts with the structure of the overall delivery program, it is easier to
try using additional software and infrastructure complexity to address the
problem. This results in an escalating feedback loop of bad design: not only is
the architecture unnecessarily complex already, but additional complexity is
then introduced to manage that complexity.

Unfortunately, the only way to break out of this loop is for some brave soul to
put their hand up and conduct a “survival of the fittest” design cull, where
every component of the architecture is ripped out and tested for critical system
viability until what remains is the minimal skeleton actually required to
deliver the baseline functionality.

Machine Learning as Universal Panacea
A more subtle variant of self-generating complexity relates to the application
of machine learning itself. Given the relative immaturity and vast potential of
the field of data science, there is now the additional lurking temptation to try
using ML to work around any problem rather than analysing and solving its root
cause. However, as Robert Frost said “the best way out is always through” – a
phrase which captures the essence of how to avoid incurring crippling technical
debt. Applied to data-driven product innovation, that means conducting patient
and precise analysis to identify underlying causes of problems and solving them
at source rather than building an ever-expanding shanty town of workarounds that
is liable to catastrophic and terminal failure.

Effective Mitigation Strategies
What techniques can be employed to mitigate against these problems and create a
reliable foundation on which to base your delivery? In our experience the two
most valuable tools are a domain model and sets of BDD scenarios (or terse use
cases) which describe the state and behaviour on which the anticipated business
value depends:

Domain Model: this is a collaboratively defined conceptual model of the problem
space shared between the subject matter experts and technical staff. Its key
importance is that it explicitly names the entities of the business domain and
their relationships to each other. In doing so, it creates a worldview and
language that is shared by all stakeholders/members of the delivery team (a
purposefully limited and relativistic scope, which is enough to ensure use cases
can be fulfilled whilst avoiding the many pitfalls and dead-ends of enterprise
data modelling). It is not a knowledge graph, ontology or database schema.
Rather it is something which SMEs might express as a frequently iterated
whiteboard diagram of labelled entities and connectors, and developers might
describe using an ERD or high-level class diagram. This foundation is what
allows code to be written in a way that precisely and unambiguously matches the
collective intent of the team. Without it, you are building on quicksand.

BDD Scenarios: once a shared language has been established which describe the
entities in the domain and how they relate to each other, the state transitions
involved in the information flows and generated business value can then be
captured (in practice this tends to be highly iterative, with new scenarios
being sought that break the model and thereby drive its improvement). The
emphasis here is not on automated testing capability – although that can be a
very helpful by-product - but the communication of state transitions and
workflows [https://lizkeogh.com/2013/10/24/bdd-before-the-tools/]. 

Once the workflows and information flows have been defined via which business
value will be delivered, questions concerning the optimal deployment of human
and machine agents can then be correctly and meaningfully addressed. At this
point, it is then really important to return to the Strategic Alignment Model
for one more pass. Newly equipped with a powerful model that clearly describes
the state/data and behaviour of your domain, you can review each entity and
function in turn to create an unambiguous and precise definition of what your
market differentiators really are. Is it really the bespoke algorithm that your
data scientists are working on, or is it in fact the data that the algorithm is
outputting? If the latter case, is it really justifiable to keep investing in
developing something in-house which ought to be (and will almost certainly end
up eventually) a commodity purchase? These considerations will then be primary
factors informing product roadmap planning – a subject beyond the scope of this
post.

The result should be a system architecture that is the simplest thing possible
[https://www.jamesshore.com/Agile-Book/simple_design.html] to deliver the
identified business value, and as a result is the cheapest to build, contains
the minimum number of things that can go wrong, and is fastest to adapt to
future market change.

AI is a core discipline at Data Language. We have an expert team skilled in machine learning and deep learning, and importantly we also have the engineering and deployment skills to deliver AI at scale for your production systems.

AI is a core discipline at Data Language: Our expert team is skilled in machine learning and the engineering skills needed to deliver AI at scale.

Artificial Intelligence delivered at scale - our core discipline.

Artificial Intelligence

Current impacts of AI on Technology Delivery

Over the last couple years, one of the more interesting architectural patterns
to have emerged on deliveries for our media customers has internally acquired
the name ‘daisy architecture’. It evolves as organisations acquire the ability
to scale their premium content production processes via the use of machine
learning, text analytics and annotation techniques, and then extend the coverage
of that tooling over as much of their remaining content geography as possible.
In workflow terms, this entails a switch from content production pipelines that
are a.) push-based; b.) manual and c.) linear; to content enrichment cycles that
are a.) pull-based; b.) automated; and c.) on-demand and iterative:



Enabling factors
The pattern is typically the result of three technical enablers:

 1. The increasing influence of Kanban on software design, (i.e. principles
    taken from how applications are delivered and applied to how they work, in
    particular pull-based systems and queue balancing/optimisation techniques).
 2. The scaling of automated content analysis and rich metadata creation via
    text analytics and machine learning.
 3. The adoption of micro-service architectures, allowing business functions to
    be recombined much more easily to serve the individual needs of specific
    product contexts.

1. Kanban
The influence of Kanban techniques not only on the process of application
delivery but on the outputs of application design (i.e. on business processes
implemented in software) is arguably a return to its roots in some ways. We
perceived that as a direct transfer of principles, but what also seems to be
going on here is that the influence of Kanban is spreading out of software
delivery into business operations more generally (we are definitely seeing
increased adoption of pull-based value streams measured via cycle time and queue
buffer monitoring) and that is then being reflected back into the styles of
software design that are required to support those pull-based business
processes.

2. Machine Learning
There has been without doubt an aggressive increase in the use of machine
learning and AI techniques within media organisations to automate content
metadata creation and indexing (see our machine learning cloud service
[https://datalanguage.com/tagmatic] for a typical example). Critical differences
between automated and manual task processing are firstly that automation has
essentially negligible operational cost, and secondly it has a larger
variability in the quality of its output. This has a major impact on the
economics of workflow design. Historically, content production workflows were
designed as linear pipelines because every step was manual and therefore costs
were within similar boundaries of magnitude. Once certain steps incur negligible
cost however, new opportunities open up to process much larger content volumes
at basic quality levels. From a brand/product risk management perspective, this
then needs to be complimented by quality assurances steps, i.e. manual review.
Those manual steps are where the costs are incurred, which is why it is highly
desirable for them to do so just-in-time i.e. once product opportunities have
been targeted which clearly outweigh that investment.

Interestingly, for these reasons we are seeing a significant impact of
automation on changing job roles in this space, but a negligible impact of
automation on actual jobs - apart from some impact to very repetitive,
rule-based work (i.e. a prime candidate for a machine) that has most often
already been outsourced. Instead we are seeing job roles switching from
production at lower volumes to the curation of production at higher volumes.
This typically takes the form of:

a.Quality Assurance of machine output, the results of which can then be used as
ongoing training data to refine machine statistical models.
b.Sampling Tuning. Not all content is equal: the quality of machine output will
depend on both its source and its subject area (e.g. the simple factual
statements of a football match report will typically be easier for a machine to
analyse successfully than an in-depth opinion piece). That means different QA
sampling rates will be required for content in different subject areas and from
different sources: e.g. new syndication source vs in-house editorial, and those
rates will then need to be managed by people over time: e.g. as the new
syndication source becomes a trusted partner they will need less manual
oversight so the sampling rate should be reduced.
c.Terminology management: to ensure that controlled vocabularies are used to
encode the core entities and relationships of the business domain thereby
maximising data interoperability across the organisation. (This is one of the
core benefits of using domain-driven design
[https://datalanguage.com/services/information-architecture] to define the
information architecture)

3. Micro-services
The adoption of micro-service architectures has unlocked access to discrete
units of business functionality, allowing them to be recombined in novel ways
according to the emerging needs of new product contexts. This doesn’t mean a
return to the dark days of untestable spaghetti service orchestrations in
black-box ESBs. Choreography logic should still be pushed into endpoints (or
else into a standalone endpoint if justified), but it should be implemented as a
strategy pattern – where the content enrichment strategy is determined by a
specific product need rather than hardcoded.

Services and Routing
Product-specific service choreography basically means having a routing service
that can execute an enrichment strategy which encodes the content requirements
of a specific product context (for example: it needs subject classification
metadata, time-coded video scene metadata, advertising block-word annotations,
etc). Doing this just-in-time however means that you only know the state of the
required output, not the input state (as you don’t know which enrichment
operations might already have been performed). This has two interesting
consequences:

 1. Routing tends to be both expressed and executed in terms of gap analysis:
    i.e. perform whatever enrichment operations are required to get content
    selection C into state P needed by my product.
 2. Services tend to be partitioned into enrichment services and content
    search/persistence services, with the latter frequently needing to support
    terse query semantics that can be either named or URL encoded and passed
    between other services.

The final point worth mentioning concerns the temptation to implement aggregate
AI solutions, i.e. replacing a workflow that is currently 12 manual
steps/functions with a single neural network solution or similar. The argument
goes that machine systems are fundamentally different from human systems hence
there is no need to decompose them into small steps: you can just train from
required outputs against known inputs and get rid of all the complexity. The
first point to note here is that the complexity isn’t being removed, it’s just
being hidden. Moreover, it is being aggressively hidden as anything encoded in a
neural network is semantically opaque: it is essentially just set of weightings
that produces a desired output, which means there are no conceptually
interoperable subcomponents that can be decomposed back out into something
actionable by humans if anything starts going wrong. Also error margins will
almost certainly be amplified across a monolithic system that has no
intermediary review and correction checkpoints. For these reasons our advice is
to steer clear of AI monoliths. Instead we recommend you continue modelling your
business processes as discrete, atomic functions, and then leave the decision of
human vs machine vs human-assisted machine to be an implementation concern only.
The key task is to ensure your systems are then designed to allow that decision
to be revised according to changing circumstance with minimal operational
impact.

The next generation of adaptive businesses will be those that can swap between
outsourced, insourced, human, machine and human-assisted machine implementations
of their core business functions with minimal impact to their ongoing
operational capabilities. That will be the true test of how cohesive and
loosely-coupled the service-based architectures of today really are.

Daisy Architecture

I recently finished up a lead architect engagement on a very large semantic
publishing implementation for a finance client in the City of London. Digging
out some old notes, I noticed that the delivery has been bookended by
reflections on the pain caused by httpRange-14.

This problem has been debated amongst some very smart people for ten years plus
now without any generally accepted solution being forthcoming, and more recently
I have begun wondering whether the real issue is more a function of the question
itself. So as an experiment, at the start of the program I tried to take a step
back to see if a different perspective might add some clarity. In doing so, I
decided to revisit a book that I last read many years ago, Saul Kripke’s seminal
text Naming and Necessity [http://en.wikipedia.org/wiki/Naming_and_Necessity].

The end result was a draft blog post I wrote early last year that somewhat
optimistically finished with the sentence “It will be interesting to see if
there are any obvious flaws I have overlooked, and what challenges are thrown up
once we proceed to prototyping the solution.” It prompted a lot of interesting
internal discussion at Datalanguage but in the end we decided not to publish
because, as Paul pointed out, the use of blank nodes on which the proposal
depended is problematic for a number
[http://milicicvuk.com/blog/2011/07/14/problems-of-the-rdf-model-blank-nodes/] 
of reasons [http://answers.semanticweb.com/questions/8336/what-is-skolemization]
.

The follow-up I wrote a couple of weeks ago, and it is set out below. It evolved
out of my client handover notes on URI strategy but quickly turned into another
discussion about httpRange-14. Somewhat encouragingly there has been clear
progression in my thoughts particularly with regard to URI binding and
httpRange-14 itself, about which I’ve now reached some concrete conclusions
which seem both coherent and practically viable.

Resource Identity and httpRange-14
As practitioners of Domain Driven Design, one of the early steps we perform in
the modelling of any new domain is the classification of its resources
[http://martinfowler.com/bliki/EvansClassification.html] into entity types and
value types. This is a question of identity and context:

 * Value types are types which are best modelled as data: their “identity” is
   determined by their attributes (and for this reason, in code they are
   generally implemented as immutable). An example might be Postcode: in most
   contexts, changing the string value of a postcode simply means you have a new
   postcode.
 * Conversely, Entity types have an identity beyond their total set of
   attributes. An example might be Person, where - in most contexts, at least -
   it makes sense to model someone as the same person regardless of which
   attributes might change over time (for which reason, a domain where it could
   conceivably make sense to model Postcode as an entity type is workflow
   management of postcode creation and allocation).

This has an impact on URI strategy. Entity types typically require
synthetic/machine identifiers such as GUIDs, Twitter snowflake IDs or similar,
whereas value types will normally use the text or hash value of their
identity-conferring attributes. So in the examples from Jeni Tennision’s
discussion of Opaque URIs [http://www.jenitennison.com/blog/node/114]:

http://id.example.org/house/NG9_3HZ/4
http://id.example.org/house/0aef0218


A domain driven design practitioner might agree that whilst the former has
advantages in terms of hackability, it should only be used in addressing
contexts. If your domain resource is really encapsulating the identity of
physical buildings then you should adopt the latter strategy. That way, your
design will be resilient to high impact edge cases such as house numbers or
postcodes being reallocated. In short,

> when designing URIs, start by analysing resource identity. Once you understand
the > nature of the nodes in your graph, you can implement resource
identification with > some degree of confidence and then move onto modelling the
arcs.
Putting this into practice, a helpful way of resisting the urge to jump straight
into traversing the graph is to start by modelling your domain using ERD
techniques and only switch into linked data graphical modelling once you’ve
nailed your URI strategy.


As regards URI structure, a strictly minimalist policy has a lot to commend it.
In this respect URIs are like code: the less it takes to achieve your goal, the
higher the likelihood of good design and the less there is to go wrong. “If in
doubt, leave it out” definitely applies, and it is then generally worth testing
your confidence in everything that remains as well. The result is normally
something that looks a lot like:

http{s?}://{authority}/{uid}


where

{uid} = {namespace?}{id}


and

{namespace} = the minimal virtual path prefix required to ensure uniqueness within the authority's hostname/port naming scope


For synthetic identifiers a namespace is rarely required, apart from the case of
incremental integers which should normally be avoided for other issues relating
to distributed synchronisation. In the instance where a namespace is necessary,
then the chosen virtual path should either be meaningless or else an intrinsic
and version-independent attribute of the resource, such as type alias/label. The
latter is not an entailed requirement for opaque URIs, but on purely aesthetic
grounds it still seems deeply ugly to me to have URLs such as the above
dereference to things which aren’t in fact representations of houses.

One of the more interesting consequences of this view of resource identity
relates to how reference is assigned. If resource identity is not a function of
attribute values, then clearly identification (i.e. what a URI refers to) can’t
be determined by attribute or property matching. The main alternative model of
reference determination was set out by Kripke in Naming and Necessity
[http://en.wikipedia.org/wiki/Naming_and_Necessity] whereby proper names act as 
rigid designators [http://en.wikipedia.org/wiki/Rigid_designation] the
denotation of which is set by a one-off “baptism event” at the point of
creation/adoption within a social group. By way of summary for those unfamiliar
with the work, Kripke proposes something of a return to John Stuart Mill in
taking on the previously dominant descriptive theories of naming that had been
advocated by Bertrand Russell and Gottlob Frege amongst others (interestingly,
the distinction between sense and content that Jeni Tennison makes in her post
about punning and httpRange-14 [http://www.jenitennison.com/blog/node/170] has
echoes of Frege). Kripke objects to descriptive theories of naming on the
grounds that giving meaning and fixing reference are not the same thing.
Instead, Mill’s position that names have “denotation but not connotation” -
which seems an excellent description of opaque URIs - is developed into the
notion of names as rigid designators, which are effectively opaque identifiers
whose references are bound at the point of social adoption.

Such a viewpoint seems consistent with experience. URIs refer to resources
because social groups (i.e. those performing the URI authority function) decide
they refer to those resources: where those URIs are URLs, that is reinforced by
the behaviour of DNS resolvers, but in the instance that a URL temporarily
dereferences to a wifi login prompt
[http://www.jenitennison.com/blog/node/168#comment-11729] that doesn’t signify a
change in meaning. Isn’t this a contradiction of the notion that URLs are late
bound? No, because the thing they late-bind to is representation, not the
resource/conceptual mapping - which is in fact early bound by the authority for
that URI at point of creation. What about the late binding enforced by the
constraint on using hypermedia as the engine of application state? Again no
contradiction, as that concerns late binding of client to URI rather than
resource identifier to concept:


The notion of resource identity also has important implications in one other
area. For the last couple of weeks I’ve been conducting a technical
retrospective and handover to the client’s architecture team on one of the
largest semantic publishing implementations in the UK right now. Part of this
has been a review of URI strategy and the decisions made around httpRange-14
[http://www.w3.org/2001/tag/group/track/issues/14] (or httpRedirections-57
[http://www.w3.org/2001/tag/group/track/issues/57] as it has become) . Due to
previous experiences of the pain entailed in coding clients against fragment
identifier URIs (namely: the ambiguities introduced about whether a given
fragment identifier denotes a document fragment or an NRI; the problems certain
HTTP request processing frameworks have with fragment identifiers; the lack of
portability to other better solutions that might be devised in future), this
time we chose to adopt the 303 strategy. In doing so, we avoided the ambiguities
and additional programming complexities but at the expense of performance due to
all the additional, and worse uncacheable
[http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html] (a bug in the RFC now
being addressed as part of httpBis
[http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-24#section-6.4.4])
303 roundtrips. For something that seemingly generates no business value to most
customers - i.e. anyone who doesn’t have concrete use cases which depend on a
formal distinction between information resources and non-information resources
being maintained - both strategies [http://www.w3.org/TR/cooluris/] are
definitely a very long way from ideal.

In preparation for the technical retrospective sessions and having now been
bitten by both approaches, I took the opportunity to revisit the recommendations
with a more critical perspective. In doing so, I arrived at the realisation that
there are actually multiple interpretations of the term “resource” in use within
the web community right now. I decided to swap a domain-driven approach for a
constraint-driven approach - the constraint in question being that I’m not using
either 303s or fragment identifiers any more because they both suck :-) - to see
if any of those interpretations offered an alternative way of thinking about the
domain which a.) avoids the problem and b.) is consistent with the way the web
works right now. The use of domain modelling was limited to prioritising the
analysis of those interpretations: by starting from the interpretation of
“resource” which most closely models how language and identity works in the real
world, then there was a good chance that the hard part of actually solving the
problem could effectively be left to Natural Selection meaning scope would be
limited to modelling only:



The diagram above describes how we assign identity as we perceive the world. Our
retinas detect a continual, undifferentiated stream of photons reflected from
the environment around us, which we then filter and interpret using mental
models or templates. For example, I have a mental concept of my cat, and that
allows to me to assign continuity of identity to any photon stream (e.g. cat in
kitchen, cat climbing tree) which I can map to that concept. When I assign a
name to that model, participate in a social group and there is enough
commonality between our named mental models, then language becomes possible:



Now let’s extend the first diagram to include mediated perception:


This shows a sliding scale - from direct perception, live MPEG4 video broadcast,
MP2 accessible audio description, web Flash streaming to web-published text – of
different formats in which the essential information content of the same real
world entity is encoded. Translating the web examples into the terminology of
HTTP, the name N is a URI, the mental model M’ is a resource and the encoded
information formats are representations. This corresponds very encouragingly to
the definition of resource as defined in section 1.1 of the older URI spec RFC
2982 (which has unfortunately been superseded by the open, and arguably less
helpful, definition in RFC 3986: the term “resource” is used in a general sense
for whatever might be identified by a URI):

> A resource can be anything that has identity… Not all resources are network
”retrievable”; e.g., human beings, corporations, and bound books in a library
can also be considered resources. The resource is the conceptual mapping to an
entity.


This interpretation of the term resource I’m going to label concept-centric.
In contrast, the interpretation of resource implied by the Cool URIs
[http://www.w3.org/TR/cooluris/] recommendation I’m going to refer to as 
document-centric: although section 2.1 describes the distinction between
resource and representation and its consequences in relation to content
negotiation, it is apparent from the diagram in the first part of section 3 that
each document has its own URI and is therefore a resource. In section 3.1 this
is then expanded into the requirement for either 303s or fragment identifiers,
on the basis that any real world “Non-Information Resource” needs a
corresponding web document “Information Resource” which describes it:



The concept-centric interpretation views the distinction between Non-Information
and Information Resource differently, and of essentially no value as both
variants are functionally identical:



This view says that RDF and HTML documents are representations, not resources.
They only need URIs in the instance that the separate concept of the
representation’s byte stream is also required. In such cases a Content-Location
header should be exposed with the URI for that concept. On the other hand, a
description of a real-world entity is a resource, because a description of
something is conceptually different to the thing itself. The description’s URI
should simply dereference to some kind of representation in exactly the same way
as a URI for a real-world resource. For example, URIs denoting a.) London and
b.) a description of London, just work as follows:



So, adopting a concept-centric rather than document-centric interpretation of
“resource” has the following consequences:

 * URIs are late binding in all aspects except denotation: what a URI refers to
   is determined at creation time by its naming authority rather than at runtime
   via a dereferenced representation.
 * As with new terms introduced into natural language, ambiguities or
   misunderstandings that arise in the early adoption of a newly published URI
   should be resolved via recourse to the publishing social group – i.e. the
   URI’s naming authority.
 * There is no non-trivial difference between an Information Resource and a
   Non-Information Resource: URIs simply denote concepts in the real or
   imaginary universe, and optionally derefence to one or more informationally
   equivalent representations.
 * A representation is not a description: a representation is a byte stream
   whereas a description of a resource a distinct concept in its own right.
 * A URI which denotes a representation (i.e. via a Content Location header)
   denotes the concept of a byte stream , not the byte stream itself.
 * 303 response codes should be used solely for standard See Other semantics.
 * Fragment identifiers should be used solely to denote resource parts.
 * No change is required from how the web works right now.

In other words by slightly changing our perspective on the problem, it
essentially disappears.

What about the scenarios where a naming authority has not published any/adequate
information to allow URI consumers to disambiguate denotation to the extent
required for the consumers’ use cases? Simple: in the absence of any better
alternatives, consumers should mint their own unambiguous URIs that do meet
their needs. My guess is that this will only be an issue for a few naming
authorities anyway: either those participating in the linked data cloud who are
currently larger enough to be used for general purposes (e.g. dbpedia,
musicbrainz, etc) or else those organisations which have strategically targeted
niche verticals where they perceive there to be longer term commercial
advantages from owning the dominant vocabulary. It seems reasonable to request
of such organisations that in order to realise their goals they conform to a
more formal definition of resource that is consistent with the demands of
semantic reasoners.

In addition to the clear pragmatic benefits of adopting a concept-centric notion
of resource, I then discovered that such a position also seems to be consistent
with the views of one of the main authors of HTTP. That finally sold me on the
viewpoint. In a typically entertaining and strong post
[http://lists.w3.org/Archives/Public/www-tag/2002Jul/0090.html] from Roy
Fielding to the TAG mailing list dating back to 2002 - worth a full read - he
makes the following comments:

> Resources are not transferred, just as identity within the real world
is not transferred when it is accessed. That doesn't stop us from
reasoning about rocks, plants, chairs, or my dog. In fact, the
separation of identity from representations of an identity is
necessary to reason about them at all. That does not mean that the
resource and the representation are both identified by the same URI;
they are not the same resource. It means that you can reason about
resources and reason about representations of resources, even if you
don't know "the most appropriate URI" that does identify the
representation as a separate resource.


> In short, the only reason this gives anyone in RDF land heartburn is
simply because their definition of resource doesn't match that of the
Web, or that of reality.


> I absolutely refuse to consider RDF as a useful technology or the
Semantic Web as the future of human communication if its reasoning
power is incapable of describing one of the fundamental facts of life,
particularly since the only reason it is incapable of doing so is
because a few people suffer from the unfortunate belief that it is
easier to change the Web than it is to change their preconception
of what it means to be a resource in RDF. Change that preconception
and RDF becomes capable of reasoning about both resources and
the representations of resources through one level of indirection,
just like the rest of us mere mortals.


To Jeni Tennison’s observation [http://www.jenitennison.com/blog/node/170] that 
“things recently make me more convinced than ever that the TAG must either
provide some direction to the community, and soon, or get out of the way”, my
view is that the minimum/sufficient direction is as follows:

 * The TAG should provide an unambiguous definition of “resource” to be used by
   anyone building reasoners. This definition should be concept-centric.
 * The TAG should provide a general advisory to anyone publishing linked open
   data that in the instance they wish their data to be used by consumers for
   more than simple graph traversal then they need to understand fully, and
   conform to, the definition of “resource” being implemented in reasoners.
 * The TAG should deprecate Cool URIs [http://www.w3.org/TR/cooluris/]

I think they should then stand back and let normal evolutionary drivers take
care of the rest. It seems unlikely that any organisation whose business cases
depend on the type of market-differentiating graph hops made possible by the
power of semantic inference is going to get their notion of “resource” mixed up
more than once.

Blogs by Julian Everett