There's a new buzz about the way Internet watchers are trying tounderstand what's happening on the web - a sort of generalised hunt forthe next web 2.0. Tim O'Reilly---the one who Trade Marked web 2.0 --- hasbeen posting about the links between financial markets andweb services like Google, Wikipedia, etc. He is particularly intriguedby the parallel between market makers trading on their own accounts(possibly in conflict with their clients), and Google entering contentprovision, in services like YouTube and Blogger. CoryDoctorow and Jim Wales are writing about the possibility ofopen source, transparent search --- the moment and the reasonfor the community to take the power back from Google. Indeed, "Jimbo" has announced the launch of WikiSearch with the hope that "Transparency" and social aggregation will yield better results than Google.

All this is exciting stuff --- we are getting to the point at which wehave digested how social information can be aggregated by networkedcomputers, and we are wondering what comes next.

What I haven't seen fully laid out is the analogy between Web 2.0services and economic mechanisms. Having this analogy clear is usefulbecause it allows us to ask how all the results known about economicmechanisms translate to Web mechanisms. By and large, I think it alsoshows that Web 2.0 represents the naif phase of web service development--- akin to economists' modeling of perfect competition. The reality isclearly some way from that, but the lessons from mechanism design arenot all encouraging: from the point of view of quality of results, thebest from the wisdom of the crowds is behind us.

Google:Attention Auction

Let's start with the PageRankalgorithm. This mechanism "auctions" attention (the screenposition in a search result) and is paid for in links. At awine auction, lots are ordered from most to least valuable. Value ismeasured by bidders' willingness to pay. In a "Google search auction"web sites are ordered from most relevant to least, where relevance ismeasured by the number of quality-weighted links pointing to a page. IfI want to get "openDemocracy.net" to number 1 slot on a "Democracy"search, I need to make sure that no one has better quality-weightedlinks relating Democracy to the domain "openDemocracy.net".Google is auctioning slots in the results pages. On the lefthand side are slots auctioned for links; in the right hand side, theyare auctioned for money.

So? What of this parallel?

First, to Cory's point about search not being neutral: the design ofthe algorithm selects what information is returned, and what meaning isgiven to "relevance". This is generally the case with all auction-likemechanisms. In the wine auction, you will end up allocating wine todifferent bidders depending on whether you use an ascending ordescending auction; and open out-cry auction or a sealed-bid auction.There is a sort of "gold standard" --- Cory's notion of the "neutralsearch" --- which in the auction literature is called the "efficient"outcome: the one that allocates each good to the bidder that trulyvalues it most. ("Relevance" is a bit trickier than efficiency becauseof the philosophical issues it raises, so I am not sure Cory's ideal ofneutrality exists for long). The auction literature suggests thatachieving efficiency is very hard and often requires unbelievablycontorted mechanisms.

The auction design literature tells us that whatever mechanism youadopt, bidders will modify their behaviour to do best for themselves.So, in a "first price" auction (one in which you pay what you announceas being prepared to pay --- as opposed to the E-Bay style second priceauction), you think hard about what the next person below you isprepared to pay and bid close to that rather than bidding your ownmaximum willingness to pay. The electricity markets that I worked on inthe 1990's were, would you believe it, mostly designed as first priceauctions! This started years of very profitable manipulation by allpower companies. Enron was particularly adept. PageRank manipulationhas also turned into an industry. In its simplest form, you buy awell-regarded web property and you then sell links from that propertyto other sites that are trying to rise in the ranks.

The mechanism literature is very keen on discovering implementablemechanisms that are non-manipulable, in the sense that it is ineveryone's selfish interest to reveal the true information about yourvaluation. In the PageRank analogy, this would be an algorithm thatwould lead you to create your content without regard for its impact onits Google position, but only with regard to your readers' bestinterests. So, for example, the simple Search Engine Optimisationadvice that all links should be made with descriptive, meaningfulterms, might lead one to make this sort of link in an article: "openDemcoracy'sDemocracy in Kenya coverage suggests that ..." instead of "PeterKimani suggests that ..." If I do the first rather than thesecond because that is what the SEO handbooks say will improvePageRank's recognition of openDemocracy's links to "Democracy" and"Kenya" ,I am gaming PageRank just as I am gaming the wine auction bysecond-guessing how cheaply I can let it go before losing it.

The essence of the efficient mechanism design results is that it isimportant to divorce what someone pays from the outcome of themechanism. So, the beauty of the E-Bay style "second price" auction isthat what I pay is determined by the bid of the next lowest person, notby my bid. It is quite easy to see that it doesn't (usually) make sensefor me to game the E-Bay system. (For the interested, thegeneralisation of the E-Bay auction to many goods --- which a Googlepage of results is, since it has many slots, is tricky. See Ausubel).

What does this mean for search? I've thought for a while that theequivalent would be for Google to give you not your own PageRank as ascore, but the PageRank of your next closest "competitor", or web site.You could then SEO all you like, it won't affect your PageRank, exceptin so far as it affects your closest competitors'. The trick in thisscheme will be implementing who your "nearest neighbour" is for any webpage.

PageRank is a market mechanism. Implementing it---like allmechanisms---requires endless fixing around corner cases. An intriguingexample is Google'strouble with Jewishness. One might think that longevity inthe market allows you to perfect the algorithm like no one else does,and so protects you from entry.

But if I were a Google shareholder, I would be worried by the analogybetween Google search and a market mechanism. As every web contentproducer adjusts to Google, its results become necessarily less andless compelling. The joy of Google past was to think hard about thesearch query and get a first screen result full of relevant but quirky,even obscure material. A Google result today is much less sensitive tothe driver, because every content maker is trying to "buy" space thatit can't pay for in genuine links. SEO will ossify Google and a bettersolution will wipe it out with the speed of an epidemic. The web hasbecome over-fitted to Google like a strain of wheat becomesover-designed to a specific ecology. The web is covered in contentstrategies over-designed to Google, and new mechanism willfind a source of meaningful, un-manipulated information---just as thehyper-link was before PageRank made it a gameable commodity.

Google will disappear much faster than Wikipedia, because Googleprovides a flow of services, while the Wikipedia mechanism has beenaccumulating an asset in its millions of pages. But Wikipedia is notout the woods yet. There is an auction analogy there too, from which Iforecast that Wikipedia will be gradually locked down,the process for editing more and more institutionalised. Moreof that in the next post.