With search continuing to play such an important role online, and some major upheavals ahead, it seems an appropriate time to cobble together a few of my thoughts on the future of search based on yesterday’s conversation and some recent news stories.

Just two algorithms to rank all sites?

The search engine market is currently dominated by a very small number of players:

Data provided by > HitsLink Market Share

The lack of competition is bad for both innovation and web site owners, and the situation looks to get worse when Bing starts to provide the search results for Yahoo.

From the perspective of web site owners, fewer search engines is bad because they become ever more susceptible to the whims of fewer ranking algorithms. At the moment, with the dominance of Google, if Google change their ranking algorithm in such a way that it penalises certain sites, the businesses behind these sites will suffer.

Bing and Yahoo argue that it should not a choice between three different ranking algorithms or two, but rather two or one; if they down join forces neither can successfully compete with Google individually. Whilst I’m not necessarily convinced by this argument, the latest announcement of personalised search by Google will mean that there is no single Google algorithm, and this may help the Bing/Yahoo cause with the EU competition commission.

From an innovation perspective I was also reminded yesterday that the deal is limited to search results, and much of the current innovation in the search arena is focusing on features built around search – for example the inclusion of real-time features such as Twitter comments – and Yahoo will be able to continue competing with Bing in this area. I was also given assurances regarding the future of Yahoo BOSS.

Search APIs and the future of search

My biggest fear about the Bing/Yahoo collaboration was the effect on the Yahoo API. Yahoo is the last of the major search engines with a decent API: allowing the inclusion of additional operators to collect data about links to a specific web page or web site. This information is very important for understanding the perception of an organisation and its competitors, as well as allowing wider experimentation with the search process. With Bing turning off their link operators I feared that Yahoo would also lose its capabilities.

Rather than a sudden replacement of all the Yahoo search products on a single day, I was assured that Bing would only be replacing Yahoo products where it showed it was capable of doing so, so the changeover with the API wouldn’t come about at the expense of the current users.

Also, the potential of programs like Yahoo BOSS for innovation in search are increasingly recognised by people in Bing. Not only does it offer the opportunity for a range of experimentation not possible in a single organisation, but at relatively little cost to Bing it could a huge amount of good will among technology community. Jordi assured me that he would try to push for a better Bing API.

Conclusions

Overall I was more positive about the state of search at the end of the meeting than I was at the beginning, especially regarding the future of search APIs. At the moment search is very much stuck in the world of web 1.0, failing to tap into the true potential of the wisdom of the crowd, but I have hopes it could offer more in the future.

The future of Yahoo BOSS seems unlcear, but I think the writing is on the wall. At the recent Bing roundtable I discussed the lack of operators with the general manager of Bing’s Search Technology Centre Europe. As he explained in a follow up email:

…the Bing API is designed to help people build a customer-facing search solution and to foster an ecosystem of sites and application that use Bing as their content service to augment the information they have with the corpus of knowledge in our indexes.

Basically Bing isn’t looking to be an innovative company using the wisdom of the crowd, they are looking to be a powerful company  who will tell the crowd how it can us its data.

Whilst they recognise “…we may lose some customers like yourself with the decisions we’ve made”, with the likely loss of Yahoo BOSS, there are few places for customers like myself to go. Nonetheless it is time to start investigating some of the alternatives that are available.

In addition to the ‘link’ command available through Google API and Exalead (which finds links to a specific page rather than a whole domain) you can also search for URL citations. URL citations are mentions of a hyperlink in a web document whether they are linked or not, and searching for a domain name in Google also returns those sites which link to a page in the domain.

For example:

link:blog.webometrics.org.uk

Just returns 93 hits. The majority of which are selflinking pages, which are hard to get rid of as Google doesn’t seem to like the link operator and the site operator being used in the same query.

For instance link:blog.webometrics.org.uk -site:blog.webometrics.org.uk returns 2,710 hits, despite including an additional exclusive clause, because it stops treating link:blog.webometrics.org.uk as a single command and instead searches for the terms link blog webometrics org uk.

In comparison:

“blog.webometrics.org.uk” -site:blog.webometrics.org.uk

Whilst Google’s claim of 189 hits is an over-estimate, it nonetheless returns 95 pages, none of which are self-linking pages, and many of which link to specific pages or even photographs on my site. Without URL citations I would probably have never known that there was a picture of me on a site called “Enjoy Satan“!

Whilst URL citations are a partial solution, if Yahoo loses its operators, it is an opportunity for a smaller search engine to start making an impact. I think those worried about the loss of BOSS would be better petitioning an alternative search engine to provide a similar service rather than bothering trying to persuade Microsoft to keep it open. Yahoo BOSS is unlikely to fit with the Bing vision.