This is part three of a three-part blog post. (See part-one and part-two here.)
“Insight Engine” is a new term popularized by research firms such as Gartner and Forrester, and represents the next step in information discovery. The general idea of an Insight engine is to dramatically improve user access to relevant content and to make access to that content frictionless. While definitions vary, I define these engines as using AI techniques to offer three core capabilities generally lacking in search engines:
- Increasing user access to all your content no matter where it resides
- Improving the relevance of search results by using personal relevancy algorithms
- Enabling next generation push/alerts
These capabilities have the ability to move enterprise search/knowledge management from a state where users spend significant time everyday trying to find information, to one where information is almost magically at one's disposal. Let’s take a closer look.
Access to Content
Insight engines dramatically change the dynamics of accessing content through the use of natural language processing. As we have all experienced, every search engine has its own unique syntax. Sometimes that difference is subtle, such as Google vs Bing. But other times the query syntax is incomprehensible to most users (e.g. native SQL queries) and thus the content becomes inaccessible. Insight engines tackle this problem by:
- Allowing users to enter a query using natural language. For example - “How many units did we sell today?”
- Parsing the query (using natural language processing) and then reformulating it in the native query syntax of the repository.
- Returning the results in a format useful to the user (e.g. a spreadsheet).
With this type of power users can query any repository for any data, giving them a true 360-degree view of the available content and thus enabling better decision making.
In the knowledge management world, this is a home run.
Improving Relevance
As we discussed in the previous post, context can be a game changer in relevancy as demonstrated by Google’s use of PageRank to dramatically improve the relevancy of internet search. Unfortunately, PageRank does not apply to enterprise search, and so to date, enterprise search engines have largely ignored context. Insight engines change that by using AI to analyze user behavior (e.g. previous queries, previous downloads, time spent on various articles, etc.) and then incorporating this user-specific context when computing search relevance. The results can be stunning, as each user now has a personal relevancy algorithm, one that is built on an analysis of data they have consumed in the past. This is similar to when you search for “cameras” on the internet and then later in the day you get personalized camera ads, but on steroids.
For you geeks out there, this is a form of Learning to Rank (LTR) where relevancy is customized by an AI model. The training data is your past search and content consumption behavior and the model seeks to predict what content you will like the best.
In the knowledge management world, this is second a home run.
Next Generation Push / Alerts
Insight engines also tackle another huge issue. A great deal of research has been done documenting that knowledge workers spend a significant portion of the day looking for information and often meet with failure. As if that isn’t bad enough, imagine how much information is missed altogether because we didn’t spend all day every day updating our searches and watching newsfeeds for changes in our industry, etc.
Insight engines seek to significantly reduce the need for users to search by enabling next-generation push/alerts. Insight engines use AI to analyze users past content consumption and then, in the background, find new relevant content and deliver it to the user. This means users get awesome, up-to-date, targeted content without even asking for it.
In the knowledge management world, this is a third home run.
A Bit of Caution
Insight engines are definitely here. But please know that:
- Not all these capabilities are available from all Insight engine vendors.
- Not all of these capabilities are as far along as the hype would lead you to believe.
But, if you are looking for that next leap forward, you should certainly look at Insight engines and vendors including: Coveo, Attivio, Sinequa, Lucidworks, as well as offerings from the big boys Microsoft, IBM, and HP.
Have you looked at Insight engines? If so, drop me an email and tell me your story.