Language is probably the only thing that needs it self to be explainable. In other words, you need to use language to explain language. And that is exactly why it is so difficult to have a computer understand language.

But we really would like a computer to be able to understand language. Especially a search engine would benefit if it could understand what we mean with what we type in that little search box.

With current search engines you need to know what you are searching for before you can actually find it. A search query like: “How to clean my kitchen” won’t answer that question, it will merely show you web pages that are about cleaning kitchens. You will most likely find pages that sell cleaning products for in the kitchen.

However, if a human were to select the pages based on the query “How to clean my kitchen”, (s)he would not select cleaning product pages, but informative pages. Why can’t a search engine understand the difference?

The main reason is that a search engine is more a data retrieval and ranking system than an information retrieval and ranking system.

Most search engines ignore “common” words like: how, to, the, a, in, for, etc. Though these words may seem insignificant, they are actually one of the most important words in language. Probably only verbs, and especially all their forms, are more important.

The example “how to clean my kitchen” is an excellent example. A search engine ignores “how” and “to” and basically searches the index based on “clean” and “kitchen”. But “clean kitchen” is not the same as “how to clean my kitchen”. By ignoring “how to” an essential part of the information in the search query is lost.

But how to make a machine know which pages are most related when dealing with a “how to” question? If you look at how the human brain does it, you quickly realize that it cheats. The human brain has a huge database of memories and experience, and all it does is compare the question with the information in the database.

Ask a baby how to clean the kitchen and it won’t even know that you’re talking to it because it hasn’t learned language yet. A 10 year old will know what you want, but lacks experience. A search engine is like a baby that has learned how to walk and recognize words, but has yet to “understand” what a searcher really wants.

The next big step in search engine technology will be the ability to understand language, and most likely it will be a slow learning process. It would not surprise me if this process will take 20 years, about the same time a baby needs to grow up.