Thursday, October 04, 2007

Finding what you're searching for

I do not have precise numbers regarding how much the amount of information on the World Wide Web has been increasing in the past couple of years, but from my searches I get the impression that the same information is available in an exceedingly high number of copies (so to say), be it news, frequently asked questions, or product information, just to name a few.

Every now and then, the magic word natural language search pops up, producing about 104 million hits on Google if I just type in the words, and still 226.000 hits if put into quotation marks - which is way too much to handle. Others have written about the topic before, so I am not going to repeat what has been said before, but the question is what can be done to find the information in something that seems like a huge haystack of Web pages.

Why is everyone using Google? Because it actually does quite well, and I can second that I mostly do not have to go beyond the first couple of result pages to find the information I want. If it does not appear, either my search turns out to have been too unspecific, or the information is not available at all.

Enter startup companies such as Powerset that claim to revolutionize search. I doubt that, given that it is hard to extract any semantics from most searches, which do only contain about three significant search terms or less. I would assume that natural language search may be able to yield decent results, but is the benefit (from the point of view of the users) really as significant as claimed? I am not so sure about this.

The problem is not that our search technologies are not good enough. It is that there is too much information to search within. So, I suspect that the future lies in dedicated search engines for specific domains (e. g. news) rather than a new universal search engine.

No comments: