How the search engines sold out

The web needs honest search engines, but search results increasingly depend on who has paid what. Is public regulation on the way?

In Cyberspace, if the search engines can't find you, then effectively you don't exist. Of course it's possible to guess some URLs (web addresses), but in the main, people find web sites by submitting queries to search engines.

These are basically powerful computers which create a map of the Web by trawling web-sites, indexing web-pages according to keywords and then assembling huge databases that link page content to keywords and thence to URLs. When you submit a query by typing keywords into a search engine, the software looks up its database and then returns a list of URLs linked to those keywords, ranked according to some criteria.

From a user's perspective, the key factors are how comprehensive is the search engine's database and how relevant are its rankings to the inquirer's interests. Ranking is important because even a relatively specialised query can turn up thousands of potentially relevant links, and few users will look beyond the first few screenfuls. From a commercial point of view, therefore, what matters is not just that one's site is indexed, but that it comes up in that vital first screen. And that is where things begin to get interesting.

For search engines are not charities. They are run by companies which seek to make a profit. Because they have always been the busiest sites on the Web, entrepreneurs were prepared to invest in search engines at the outset in much the same spirit that Victorian speculators bought land around railheads. It would surely be possible somehow to extract some revenue from all those people passing through - to convert those 'eyeballs' into an income stream.

But this cash-conversion process has proved as elusive as cold fusion, and search engines inevitably turned to more tangible ways of earning revenue. One is advertising - charging people to put their brand-names on the Web portal's screen. But this strategy bombed because Web users loathe advertising which consumes their bandwidth, and in any event advertising dried up after the collapse of the dot-com bubble.

Which left only one possibility - charging website owners a fee in return for boosting their rankings in search results. This enables companies to ensure higher ranking for their sites than they might objectively deserve. The practice of charging for ranking rapidly became widespread (with the honourable exception of Google), and as a result millions of naive inquirers are nowadays directed every day to sites which have paid for premium placement.

Does this matter? Yes. Firstly, it introduces an imperfection into the frictionless marketplace that the Web could become. In principle, the Web makes it possible for customers to shop around for the best supplier; but it can only deliver on that promise is the search process is objective. Secondly, paying for placement distorts the Web as a medium for the unfettered dissemination of ideas: if powerful institutions (corporations, governments) can influence the outcome of web searches, then they can effectively ensure that some voices are rarely heard.

It's vital to the public interest, therefore, that search engines should be honest. But that raises a thorny policy issue. Should they become public utilities, paid for by public authorities? Or should they remain as private companies operating under public regulation? If the latter, who should be the regulator? In this context, it's significant that last week the US Federal Trade Commission wrote to several search engines that charge for prominent placement - notably AltaVista, LookSmart and AOL Search - demanding that they should make that practice clearer to Web users. The Commission said that prime space can confuse Web users who are looking for the best response to their search, rather than ads for sites that paid up front. This is the first step on what could be a very interesting journey. Stay tuned.