LLMs and Knowledge Graphs - Data Language Spring 2023 Hack Event

Bottom Line Up Front

• chatGPT has many strengths, but...
• There is a large gap in capability if you are seeking repeatable and accurate processes conforming to a structured graph model.
• The LLM-KG combination is a good strategic shape, offering convenience and utility, so we plan to continue pursuing it.

Why LLM / Generative AI?

The team has been playing around with LLM tech for a while, around the edges, so we decided to take a couple of days to look closely at LLM capabilities related to our core Knowledge Graph use cases. We used chatGPTs APIs this time, and plan to try others as they become available.

Hack Format

Some of us gathered in our London office at The Business Design Centre, others worked remotely in Milan, Tokyo and elsewhere.

1. We set a high level challenge
2. We provided APIs and sample knowledge graphs,
3. The team attacked the areas that interested them, alone or in self-organized teams.

The Hack Event Challenge

How might we use LLMs to help with common Knowledge Graph customer use cases?

A. Getting information into a Knowledge Graph, and
B. Getting intelligence out of a Knowledge Graph.

The Results

A. Getting information into a Knowledge Graph

1. Team "Slot 4" explored how chatGPT performed vs our Tagmatic Text Classification product in classifying content against vocabularies. Outcome: chatGPT falls seriously short of our product Tagmatic's capabilities, totally failing in many.
2. Team "MI" explored how chatGPT could be used to load Healthtech industry intelligence from unstructured web content into an instance of our Market Intelligence Platform on Data Graphs. Outcome: chatGPT showed some promising signs, but this was hindered by the unpredictability of hallucinations.
3. Team "Adam" explored how we could use chatGPT to complete information for partial or missing concepts in a graph. Outcome: Very promising results. As expected chatGPT did very well here, populating concepts with well written knowledge from public sources.

B. Getting intelligence out of a Knowledge Graph

1. Team "Humans are not Obsolete" explored using chatGPT as a natural language interface onto a knowledge graph in Data Graphs, so that we could generate stakeholder friendly graph interrogation. Outcome: Promising signs, but still a lot of wrangling and dealing with hallucinations.

Overall: Strengths

• Filling in gaps in knowledge in a knowledge graph.
• Producing well-formed answers to questions, based on knowledge graph data.

Weaknesses

• Hallucinations were ubiquitous, even when we were instructing the chatGPT APIs to stick to known information.
• The presence of hallucinations meant that responses needed to be double checked, which leads to a net negative outcome overall.
• Inability to conform to a scoped, structured model is a problem for our customers.
• Inability to produce repeatable accurate behaviours is an issue for knowledge graph platforms.

What next?

LLMs are clearly an important tool in the knowledge graph toolbox for the coming months and years.

We plan to further develop the "Getting intelligence out of a knowledge graph" side, and keep tabs on the improvements around marshalling unstructured content into a structured graph representation.