Liberty Chatbot The most common way: not handling it
A common way in today’s industry is to assume that the NLP tools available on the market are not sufficient for Chatbots, and therefore there is no point in providing NLP tools for the Chatbot owner/manager. The solution these platforms offer is to roll the responsibility over to the chatbot owners and leave them to manage the linguistic process manually.
This approach usually leads to producing a chatbot that is capable of producing an active conversation, meaning the chatbot asks questions that practically lead the end user to predefined answers. This is typical for platforms that lack the capacity to analyze natural language, and therefore require less resources to construct. You can read more about the implications on the functioning of the chatbot when using such platforms in the article: "How is asking, and how is answering".
Manually
The simplest and best method is of course the manual one. The Chatbot owner manually indicates what the meaning of every given sentence is. This is how some Chatbot platforms such as api.ai , wit.ai , luis.ai operate. Despite the fact that all three are adorned with domain names that imply artificial intelligence, in practice, at the time of writing these lines, their way of extracting the meaning of a sentence in natural language is to create available tools for the Chatbot owner to do it himself.
The advantages of this method are clear: a human has an amazing ability to understand natural language, so it is reasonable to assume that talented and intelligent developers (and they must be talented and intelligent, because the instructions are so complicated) will perform the task better than any machine could.
However, this method has one significant drawback. There are many ways to say the same thing in natural language, so the chatbot construction will necessarily be tedious and time consuming. Marking the meaning of each and every sentence requires a very significant amount of work, which deems the task unproductive.
The statistical method
The statistical method is broadly familiar, as it is the one used by the Google search engine. The method is based on a statistical analysis of the text. If the word pair "Cherry Blossom" appear more frequently than any other words in the text, it is likely that the text deals in one way or another with the cherry blossom.
The statistical method is the best one existing today, yet it has two major drawbacks:
It is not effective for analyzing a short text (what kind of statistical data could we extract from a sentence like: “I want to know where the birds fly”?) Since there is not even one word that repeats itself in the sentence, the statistical method will not yield any significant insight.
The second problem is that using the statistical method does not enable us to figure out what was actually said about the subject, since it analyzes only the appearance of words, and not the relationship between them. Google's engine will not be capable of distinguishing between a sentence that suggests a positive opinion about "Cherry Blossom" and one that implies a negative opinion. Therefore this method, despite the huge efforts technology giants have made, does not produce impressive results for the Chatbot’s NLP.
In practice, the statistical method isn’t useful because it does not provide a way to know whether the customer wants to see the "cherry blossom", he/she is allergic to the "cherry blossom", or in fact they are looking to stay away from the "cherry blossom" altogether.
Machine learning
The “machine learning” method is an algorithm that receives texts as input and puts out a conclusion based on the relations between the words in the text. In practice, the “learning” is the process of measuring the distances between words, and mathematically modeling the results in order to draw conclusions regarding the specific relationship between the words.
Here's a text for example:
"Cherry blossom in Japan is great."
"The cherry blossom is really so beautiful."
We can examine each word separately by using dictionaries and identify "wonderful" and "beautiful" as positive words.
The learning machine measures the distance between each of the words in the sentence, then examines that particular word’s distance from each of the other words. In the event that only the above two sentences were entered as input, the machine will reveal a simple relationship pattern, and conclude that there is a direct link between "blooming" and "positive".
This is obviously a very simple example attempting to explain a very complex topic. In reality we have millions of words from a vast variety of texts. Using machine learning in combination with other methods in the process of analysis, will improve the results.
The learning machine method is by far superior to the statistical method for Chatbot because it yields results not only about the topic, but also about what was said about the topic. Despite the huge amount of computing power this method consumes (after all, we don't just count words, but check the distance between them) this method is clearly the future in text analytics.
However, this method has two drawbacks regarding NLP for Chatbots.
1 - Ambiguity
Natural ambiguity in the text, meaning that the exact same thing/substance can be positive to one and negative to another. The results we receive from the algorithm are a direct output of the data that was inserted as input, yet the input texts are not necessarily relevant to the end user conversing with the Chatbot. As the following example demonstrates: The machine may conclude that Cherry blossom is a positive thing, but this isn’t necessarily true for the particular individual currently using the Chatbot, if unfortunately he/she were allergic to Cherry blossom.
2 - Unsupervised learning
Unsupervised learning is very complex and it poses a real challenge for the entire field of machine learning: the machine learns by itself, and during that process might reach conclusions that while making total sense from a mathematical perspective, are utterly ridiculous from a human’s point of view.
Think of what might happen should your Chatbot begin to insult customers, because the algorithm identified and thus reflected some of the customer’s impolite language - are you ready to consider this kind of scenario?
The biggest advantage of the “machine learning” method lies in its ability to “learn” by itself. If and when the machine requires supervision by a human, we lose the power and benefits that the method was originally supposed to offer.
Relations between words based on sentence structure
This method is an implementation of Noam Chomsky’s linguistics theory. Chomsky claimed that every language has a “deep structure” that contains the logic of the language and defines the relations between words within the structure (sentences). That explains, according to Chomsky, why children don’t acquire language by learning the meaning of words (as words are super complicated to understand and have multiple meanings.) Instead, children first learn the deep structure of the language, or the way sentences are structured. While deep structure is relatively simple and limited, the meaning of the vast amount of words within it is highly complex.
Although Chomsky’s approach has been considered status quo in the academic world for over forty years, linguists have yet to agree what exactly those “deep structures” are. All things considered, this method appears to prove efficient for extracting the meaning of natural language for Chatbots, because it can analyze a single sentence in and of itself.
Summary
As it appears from the brief overview, there is no one perfect method for NLP. Each method has pros and cons, while in practice most platforms simply choose to ignore the need, and to not implement any of the NLP methods.
We use several different methods combined, which is functionally one of the main advantages of the Chatbot design platform we offer. We have implemented automatic NLP tools into our Chatbot construction process. These tools automatically extract the meaning of a sentence into a short version of a compact response trigger, enabling your Chatbot to respond more relevantly and appropriately to a wider range of possibilities of input data coming from the user in natural language.
