Can Conversational AI Shed its Dunce Cap?

Conversational AI is everywhere. (E.g. Alexa, Siri, Google Assistant, as well as thousands of lesser known chatbots). However, as discussed in the last post, these systems currently function well only with simple tasks, because the bots cannot understand normal everyday conversation the way a human assistant does. As a result, we’ve largely given up using them for complex tasks.

Current chatbots are not very smart

Current chatbots are not very smart

As we saw at this year’s Google I/O conference, Google demonstrated that we are on the cusp of change and by using several new techniques, AI is now capable of understanding complex conversations and responding in natural and intelligent ways.

The most popular technique for helping machines understand conversational speech is by using word embeddings, which attempt to encapsulate the “meaning” of a word in a vector after reading massive amounts of text and analyzing how each word appears in various contexts across a dataset. The idea is that words with similar meaning will have similar vectors. Word2Vec & GloVe are currently the most popular word embedding algorithms. But as Sebastian Ruder, Research Scientist at AYLIEN, notes “learning word vectors is like {an image recognition system that} only learning edges.” It works for certain situations where the problem is simple or straightforward, but not when things are complex.

Let’s look at several new techniques that attempt to move beyond Word2Vec’s shallow approach and embed text with meaning in a richer way.

1) Narrow the Scope and Train Intensively

While this technique works, it is more of an “idiot savant” approach, where the bot will be able to converse across a narrow domain quite well but will be really dumb about everything else. This is okay in some situations, but when using this approach, it is especially important that the users know the chatbot is a computer, so that when the bot says something silly the user knows why.

This was a core technique used by Google in its I/O conference demo, when Google Assistant booked an appointment at a hair salon, and then made a restaurant reservation. As Google explained, the training was intensive and narrow in scope. But what would have happened if the human had decided to make small talk and asked, “How about them Red Sox?” Google noted that Google Assistant was not ready to “carry out general conversations,” so the response would probably have been hilarious or embarrassing.

2) Next Generation Word Embeddings

A paradigm shift is occurring within word embeddings by new techniques such as ELMo, ULMFiT, and the OpenAI transformer. As per Sebastian Ruder, if learning word vectors {e.g. Word2vec} is like only learning edges, these approaches are like learning the full hierarchy of features, from edges to shapes to high-level semantic concepts.” In essence these new techniques have a much richer semantic representation of words/sentences and thus enable the bot to understand words in a deeper way.

Possibly even more exciting is the idea that with these newer systems we may be able to build transferable pre-trained universal word/sentence embeddings that we can use with virtually any bot and achieve excellent comprehension and results, which sounds a lot like human intelligence!

3) Use an Ontology and Sentiment detection to Label the Text for Meaning

While word embeddings are one way to embed a text dataset with meaning, data labeling is the tried and true method used in other AI domains such as image recognition. (See our white paper on “Data Labeling Full-Text Datasets for AI Predictive Lift” for more comprehensive treatment of this topic.) The problem with data labeling is that in the past this has been done by humans and thus is very expensive. But automated data labeling for text is now a possibility, using an Entity Ontology and Sentiment Detection.

An entity ontology is like a dictionary and a thesaurus; its job is to define the meaning of words by: a) encoding commonalities between concepts in a specific domain (e.g. both “yellow fever” and “malaria” are “diseases spread by mosquitoes”), and b) encoding how words relate to concepts, when they vary depending upon the context (e.g. that Mercury is sometimes a “metal,” sometimes a “planet” and a sometimes a “Greek god”). Entity ontologies can be created and used to label a dataset with meaning at great cost using humans. But now these tasks can be fully automated. High quality ontologies can be generated using NLP and AI techniques. These ontologies can be further edited by domain experts (“human-in-the-loop”) and then used to label datasets in bulk or in real time (e.g. streaming).

Understanding text also requires a nuanced and micro understanding of sentiment. Document or even sentence level sentiment is essentially useless for AI. For example, “My neighbor’s garden is awesome, the vegetables are really fresh, but they also attract deer, which is how I got Lyme disease.” The bot needs to see the first part of the sentence as positive (e.g. the fresh vegetables produced by my neighbor’s garden are excellent), and the second part of the sentence as negative (e.g. getting Lyme disease because of my neighbor’s garden is awful), rather than as neutral (half good plus half bad).

Labeling datasets with ontologies and sentiment often result in a better chatbot than by using word embedding alone, as the ontology and sentiment detection capture additional meaning allowing the bot to achieve a more human-like understanding of the text.

Losing the Dunce Cap in 2019?

While I cannot be sure these newer techniques will make bots super-smart next month or next year, we do know that they are making conversational AI systems smarter all the time. If you are using these new techniques, we’d love to hear about how it’s working. Or if you want help moving your bot to the head of the class – give us a call.

Secrets to Achieving Predictive Lift When Using AI on Full-Text

As stated in the previous post in this series “Extraction of meaning — or more specifically, semantic relations between words in free text — is a complex task.” Complex enough that machines can find full-text hard to understand and therefore building good models is difficult. Standard NLP techniques (such as tokenization, stemming, parts-of-speech tagging, parsing, named entity recognition, etc.) can improve the model but often prove inadequate.

The following advanced techniques are examples of the types of steps that can be applied to unstructured full-text datasets to add structure and meaning in order to produce better models and significantly increase predictive lift, which go well beyond standard NLP techniques. This post will focus on three examples of advanced techniques:

  • Labeling nouns and noun phrases for meaning
  • Extracting sentiment (most often) from adverbs and adjectives
  • Extracting intent from verbs

The use of these techniques is similar to a person using a compass or GPS, which provide the machine with a way to navigate and understand the text.


1) Labeling nouns and noun phrases for meaning

Much of the meaning in text is stored in nouns and noun phrases. Unfortunately, machines don’t know that dogs and cats are animals often living with humans as pets. However, the machine can learn these types of things via the creation of a semantic ontology of entities.

The purpose of the semantic ontology is to: a) encode commonalities between concepts in a specific domain (e.g. both “ticks” and “mosquitoes” are “disease spreading insects”), and b) to encode that certain words relate to different concepts depending upon the context (e.g. that “mercury” is sometimes a “metal,” sometimes a “planet” and a sometimes a “Greek god”). The entity ontology is a semantically unambiguous dictionary that enables the machine to learn much faster and more accurately than simply processing the ambiguous raw text.

For example, take full-text medical records involving a cardiologist, Dr. Smith. When examining thousands of medical records Dr. Smith frequently occurs in records about heart attacks. When processing raw medical records the AI model will often “overfit” and may think Dr. Smith is a cause of heart attacks, rather than the attending doctor. The entity ontology can prevent this and help the machine to understand the difference between doctors and the factors that can cause heart attacks.

A key problem with building ontologies is that to date most have been created by humans, which is both costly and time-consuming. This is no longer true. Advanced NLP systems can now create a semantic entity ontology in an automated way, simplifying, speeding and reducing the cost of this critical step.

A word of caution, some systems/vendors create what is known as an orthogonal ontology where nouns and noun phrases are only placed in one concept. While this may be acceptable in some applications, in others it may be highly problematic. Please review the type of ontology being created and how it will be used before you invest heavily in its creation.

2) Extracting sentiment (most often) from adverbs and adjectives

Sentiment analysis is the process of determining whether a piece of writing is positive, negative or neutral. Standard NLP systems claim to enable sentiment detection, but most fall far short of the type need to truly help machines learn in all but the most straightforward situations. When using sentiment analysis for AI, document-level sentiment is essentially useless. AI requires sentence and often phrase-level sentiment. For example “The x-ray revealed good news regarding tumor reduction, but also, unfortunately, revealed advanced pneumonia.” For AI, we want to be able to see the first part of the sentence as positive, the second part of the sentence as negative, and we want to identify that “advanced” is an intensifier and therefore really negative. Lastly, we do not want the sentence to be classified as neutral (half good plus half bad), we want to see the sentence as two distinct thoughts one positive and one negative.

Properly detecting sentiment is tricky, but it can change the entire meaning of a sentence and therefore it is often critical in AI applications. For example, a sentence stating “we saw evidence of…” and “we saw scant evidence of” are nearly identical. Yet the entire meaning of the sentence hinges on this one word – “scant.” Note, “scant” would not be labeled by a semantic entity ontology, as scant is an adjective. So while nouns contain most of the meaning, advanced sentiment analysis, e.g. how the nouns (or verbs) are modified, can be critical to the machine truly understanding the full text. Armed with superior sentiment analysis, data scientists can vastly improve the accuracy of their model.

3) Extracting intent from verbs

Datasets are usually about a specific domain (e.g. medical, financial, tech support, e-commerce, etc.). Most domains have a unique and specific set of actions or intents, as Townsend and Bever (2001) remarked, in everyday life, “most of the time what we do is what we do most of the time.” For example, in Weather, the number of user intents is quite low. I want to know the current weather – is by far the most common intent. So training a machine to understand a phrase such as “What is the weather in London?” is easy. In other domains, the number of intents can be quite large and the NLP must be trained more carefully. For example, in e-commerce the intent – “a return” might be characterized many ways – e.g. “take back”, “want my money back”, “don’t want”, “don’t need,” etc. 

The following capabilities are important when evaluating intent detection:

  • Can you train your NLP to understand a domain-specific set of intents?  (By this I mean, not just build a hand-coded set of intents, but build an ML assisted understanding of intents.)
  • Can you build rules with whitelisted or blacklisted words to supplement the learned intents?
  • Does your system have a way to disambiguate intents between the intender and the intendee?

Standard NPL libraries cannot do the above, however, if your NLP system has all of these capabilities you will be able to build a robust understanding of user intents and thus build a better model.

4) Putting it all together

Let's take a look at a few complex sentences.

“The image showed a reduction in the size of the mass. This news came as a devastating blow, as it was understood by the patient that she was in total remission.”

In summary, with the proper pre-processing and NLP feature extraction, the model will be able to understand that:

  • The tumor is getting smaller, which is diagnostically very good news.

Rather than possibly getting tricked into believing that:

  • The patient is in total remission, and the sentiment is neutral.

Let's examine a few more details:

  • An entity ontology will help the machine understand that “mass” is the same as a tumor, even if the word “ mass” is rarely used in the dataset.
  • Sentiment analysis would help the machine understand that we have marginally good medical news (about tumor size) and that the patient is upset.

Advanced NLP techniques such as these are critical to creating better data, from which machines can build better models. Is your model finding full-text hard to understand? Give us a call, we have tools and techniques that can help.

Why Machines (AI) find full-text hard to understand?

In this multi-post topic, we examine the problem and reveal the secrets for successfully training AI models on full-text datasets. First, let’s understand how hard this is and why?

Two people talking v4.jpg

The following statement by Indrek Vainu CEO of AlphaBlues, an enterprise chatbot company, summarizes the current situation. “Extraction of meaning — or more specifically, semantic relations between words in free text — is a complex task. The complexity is mostly due to the rich web of relations between the conceptual entities the words represent.” He goes on to say that machine learning is “largely clueless when fed unstructured data, such as free text.”  

IMImobile, another chatbot company states that “Machine learning is a powerful technology and promises an exciting future where machines can come to understand our needs and our intent, perhaps better than we do ourselves. However, at this moment in time we only recommend machine learning for scenarios where there is little scope for ambiguity, and where vectorisation (converting non-numeric input to numeric inputs) is straightforward.” 

A recent customer engagement at Informatics4AI supports these statements. Our customer was working with a dataset comprised of unstructured doctor's notes. They found their machine learning efforts created a model that was highly effective for straight forward diagnostic situations (e.g. a patient passing a common screening test). But when fed notes relating to complex tests and multiple patient conditions, the model did not produce predictions with the accuracy that they needed. 

As an illustration of the difficulty that AI has with full text (and for a bit of fun) let's take a look at the results that Janelle C Shane got when she trained a neural network on a database of about 30,000 recipes and then asked the machine to produce a new recipe: 


2 pkg hershey’s can be prepared in unpeeled

1 smaller

½ cup yellow onions you may

1 cup egg; chilled, coursely chopped

½ lb bacon, chopped

1 ½ cup sugar, grated

4 oz square oil

Halve the finely chopped fresh garlic salt and pepper. Break the meat into the pineapples and pat them, scraping the room off the skillet. Add ghees and beer and bring to a boil; cover and simmer, uncovered, on High for 20 to 30 minutes or until the onion thickens.

To be fair and to clarify, this model was built by an AI enthusiast and not a AI professional, but I think it illustrates the issue – the machine has no clue what a recipe is really all about. 

However, all is not lost when trying to apply machine learning to full text. The key is adding structure and meaning to the raw data, and by doing so, enable the machine to understand the text and thus begin to learn. We will review these techniques in the next blog post

Information Architecture for AI

A dirty little secret? It's what we do.

The Associated Press released an article on AI's Dirty Little Secret. As the article explains, the secret is that AI is powered by people!  Well, I'm glad the secret is out, as this is what Informatics4AI is all about. Let me explain.


The article discusses how people are needed to tag ("label") images, text, or other data because without the tagging, machines cannot learn. One example is the tagging of traffic images so that self-driving cars can become smart and keep us safe. The "secret" (according to AP) is that the tagging of traffic images by people is now a vibrant worldwide industry. It's important to remember that companies such as Facebook have been using crowd-sourced image tagging (or data labeling) for many years to improve their AI image recognition systems. So the dirty little secret isn't that data labeling is a new blue-collar industry, but rather that machines are actually pretty dumb and they need people to curate data in very specific ways before machine learning can take place.

As David Auerbach said, "Computers are near-omnipotent cauldrons of processing power, but they’re also stupid. They are the undisputed chess champions of the world, but they can’t understand a simple English conversation. IBM’s Watson supercomputer defeated two top Jeopardy! players last year, but for the clue 'What grasshoppers eat,' Watson answered: 'Kosher'.”

How can computers be so stupid and yet so smart at the same time? 

The answer lies in the fact that images and full text are ambiguous and hard to understand (think "captcha"). Unstructured information that you or I can understand makes little or no sense to a machine. However, if the dataset is properly curated ("data wrangling" in AI terminology) and an ontology is built to help the machine learn, then the magic can happen, and the machine can become intelligent. Delivering insights that you or I cannot imagine. Informatics4AI is focused on helping organizations use information architecture to improve the quality of their data, and thus improve the results of natural language AI projects.

So now that the secret is out, let's get to work. Do you need help improving the quality of your data or do you need to develop an information architecture to support your ongoing AI efforts?  Give us a call.