NLP/ NLU Capabilities

1 introduction 1 introduction the engati platform comes pre packaged with its own proprietary nlu engine for making the chatbots built on the platform respond most accurately to the user’s queries while there are a lot of complex computations which happen for determining each response, we have made great efforts to keep the platform simple for building and training the chatbots the following sections will help to throw light on what goes on behind the scenes from an nlp/nlu perspective for resolving each query 2 nlp pipeline 2 nlp pipeline the nlp pipeline can be defined as a series of steps which happen in perfect synchronization to process the query and terms and provide the best response each step or component of the pipeline is focused on performing a particular objective and then all these results are consolidated, reconciled and the best match is returned by the chatbot this pipeline is what orchestrates all the individual aspects let us look at each of the components in more detail 2 1 normalisation to reduce bias in the pattern recognition algorithm tokens are transformed into a consistent format which is lower case this is also necessary since many users prefer lower case text for chatting also noisy characters e g punctuation can also be removed for better downstream processing 2 2 tokenisation input messages are split into sentences, and then sentences are split into tokens/words tokens are the lowest common denominator for further processing 2 3 stop words removal stop words are frequently occurring words like ‘the’, ‘and’, ‘a’, etc that do not contribute greatly to understanding text and are thus removed from the input message this helps in reducing noise and improves accuracy custom stopwords can also be added to enhance the training of the nlp engine 2 4 spell check the platform includes the capability to correct misspelt words in user messages the spell check models continuously learn new words based on the training data added by bot administrator it computes and keeps track of frequencies of words in the system to identify misspelt words it is based on a word distance algorithm called “levenshtein distance” for every user query if we detect any word as a misspelt word we choose the best suggestion based on the above mentioned word distance algorithm the spell check dictionary automatically rebuilds itself on every new statement being added to the system example let’s say the user message is “recharge offers for prepaid with 10gb data and unlimited national calls” this user message is converted to “recharge offers for prepaid with 10gb data and unlimited national calls” before any further processing 2 5 stemming/ lemmatization the root of each word is determined to eliminate affixes from words the approach is different for different languages as well in some cases, the system would perform lemmatization to get to the root word whereas in others it would use the stemming approach these are fundamentally different in their approaches but are used for the same intent of distilling the terms to their base form for example – the word stem of liking liked is determined as like this helps in better semantic understanding 2 6 conversational context nlu capabilities of the chatbot allow it to maintain conversational context while conversing with a user, by keeping track of the entities this makes the conversation easier and quicker so that the user need not mention the entity in subsequent queries carrying the same context the nlu engine remembers the entities used in a session by saving them as conversation context history conversational context is also detailed in its own section 2 7 named entity recognition entities are very important for identifying and extracting useful data from natural language input while intents allow us to understand the motivation behind a particular user input, entities are used to pick out specific pieces of information that users have mentioned any important data to get from a user’s request is an entity there are built in entities like day/date/timequantity/country and many more specific custom entities can be added for a use case, for telecom plans the possible values can be “unlimited national calls”, “unlimited data plan”, “fixed call plan” etc the bot administrator gets to specify the entities expected in user queries and the extraction engine extracts these entities from the user query example “what are pre paid plans with 10gb data and unlimited national calls?” the entity extraction engine can figure out the user is talking about “10gb” data and a plan of type “unlimited national calls” 2 8 figures of speech determination the tokens/words are tagged into various figures of speech like nouns, verbs, adjectives, adverbs, etc these data points are used to tag the query and then give appropriate weightage for these parts of speech in the final representation this influences the responses determined by the nlp engine to result in more relevant matches 2 9 synonyms synonyms are alternate words to denote the same object or action from a bot and context relevancy, typical use cases involve your domain specific synonyms it could also be used for cases of common misspellings, abbreviations, and similar uses you can add more than one synonym by pressing the tab/enter key after each entry the system finds matching synonyms and adds them to your list, you can remove these if you don’t find them relevant 2 10 faq disambiguation related match is a capability which lets the bot provide not just the best response from the training data set but also other options which were under consideration and were close to the top match the number of options which are shown to the user is dependent on how many trained responses matched the query and the distribution pattern of their scores the system also makes a determination of the most well formed statement/phrase from the set of variations and uses that to present as the variation 2 11 semantic match a semantic match is the ability to understand the text semantically and then identify the best response based on the match faqs and documents uploaded into the bot are broken down into multiple logical tokens and then semantically matched for the best response engati’s nlp/nlu pipeline also automatically ranks and identifies the best response to a user query against multiple possible ones in case of issues, feel free to reach out to support\@engati com