With search continuing to play such an important role online, and some major upheavals ahead, it seems an appropriate time to cobble together a few of my thoughts on the future of search based on yesterday’s conversation and some recent news stories.

Just two algorithms to rank all sites?

The search engine market is currently dominated by a very small number of players:

Data provided by > HitsLink Market Share

The lack of competition is bad for both innovation and web site owners, and the situation looks to get worse when Bing starts to provide the search results for Yahoo.

From the perspective of web site owners, fewer search engines is bad because they become ever more susceptible to the whims of fewer ranking algorithms. At the moment, with the dominance of Google, if Google change their ranking algorithm in such a way that it penalises certain sites, the businesses behind these sites will suffer.

Bing and Yahoo argue that it should not a choice between three different ranking algorithms or two, but rather two or one; if they down join forces neither can successfully compete with Google individually. Whilst I’m not necessarily convinced by this argument, the latest announcement of personalised search by Google will mean that there is no single Google algorithm, and this may help the Bing/Yahoo cause with the EU competition commission.

From an innovation perspective I was also reminded yesterday that the deal is limited to search results, and much of the current innovation in the search arena is focusing on features built around search – for example the inclusion of real-time features such as Twitter comments – and Yahoo will be able to continue competing with Bing in this area. I was also given assurances regarding the future of Yahoo BOSS.

Search APIs and the future of search

My biggest fear about the Bing/Yahoo collaboration was the effect on the Yahoo API. Yahoo is the last of the major search engines with a decent API: allowing the inclusion of additional operators to collect data about links to a specific web page or web site. This information is very important for understanding the perception of an organisation and its competitors, as well as allowing wider experimentation with the search process. With Bing turning off their link operators I feared that Yahoo would also lose its capabilities.

Rather than a sudden replacement of all the Yahoo search products on a single day, I was assured that Bing would only be replacing Yahoo products where it showed it was capable of doing so, so the changeover with the API wouldn’t come about at the expense of the current users.

Also, the potential of programs like Yahoo BOSS for innovation in search are increasingly recognised by people in Bing. Not only does it offer the opportunity for a range of experimentation not possible in a single organisation, but at relatively little cost to Bing it could a huge amount of good will among technology community. Jordi assured me that he would try to push for a better Bing API.

Conclusions

Overall I was more positive about the state of search at the end of the meeting than I was at the beginning, especially regarding the future of search APIs. At the moment search is very much stuck in the world of web 1.0, failing to tap into the true potential of the wisdom of the crowd, but I have hopes it could offer more in the future.

I’ve just finished writing a program using the Naver API for some colleagues in South Korea:

Whilst it was not an easy task to get an Naver API-key (in the end someone else got it on my behalf), and some of the words were stored as images (i.e., not much use to Google Translate), once the API-key was acquired Naver APIs were very easy to use. Naver has a host of indexes including blogs, images, videos, and academic documents. Unfortunately Naver doesn’t have any link operators, and the index contains less English language pages than would be found in the indexes of the American search engines.

Nonetheless Naver does serve as a reminder that there are possibilities away from Google, Bing, and Yahoo. We just need to start looking at them; even if that does require the help of Google Translate.

• Get the fundamentals right
• Become less US-centric

Whilst it’s good to hear that Microsoft are moving away from a US-centric vision of search, and whilst we understand the importance of getting the fundamentals right, Bing is being built squarely within the existing search paradigm. For Bing to gain a significant market share of search they need to start thinking outside that cliched box.

Whilst O’Reilly described Google as “the standard bearer for Web 2.0“, in many ways search is still stuck in web 1.0; big search engines designed to meet the needs of the average user. Surely search 2.0 is about opening up the data and encouraging the community to create the thousands of search engines that are needed to fill the needs of the extremely long search tail.

Theoretically some of these needed search engines could be built through the Application Programming Interfaces that the search engines currently provide. However these APIs are very limited in comparison to the data the search engines actually have, and Bing’s API is the worst since the removal of some of their commands back in 2007 [Nb. "...back online as soon as possible.." has not yet materialised 2 years later].

If Microsoft want to gain search engine market share, then they need to allow programmers and researchers to work with them. Give us an API (and more) that we can really work with and we will build a million little Bings that won’t just get Microsoft its coveted second place, but first place.

UPDATE: Phil Bradley goes into far more detail of the actual discussion over at his blog.