Attempts to tap collective intelligence on the Web come in lots of different forms.

Tapping Collective Intelligence

James Surowiecki, The New Yorker http://www.newyorker.com

In November of 2000, NASA did something unusual: it asked amateurs to help it map the surface of Mars. The agency set up a Web site called Clickworkers, where people could take a short tutorial on how to classify Martian craters and then get to work clicking on photos of Mars. NASA then aggregated all those clicks to come up with a Martian-crater map. There were two very interesting things about the results. First, although there was no financial incentive to participate, more than a hundred thousand people took part in the study, generating more than 2.4 million clicks. Second, and even more striking, the collective product of all those amateur clickers was very good—as a report put it, their “automatically computed consensus” was “virtually indistinguishable from the inputs of a geologist with years of experience in identifying Mars craters.”

At the time, Clickworkers may have looked like just an oddity. But it demonstrated, relatively early, that one of the Web’s most intriguing, and potentially most important, characteristics is its ability to harness the collective intelligence of large groups of people in order to solve problems. It’s easy to see how the Web has transformed the way we shop, the way we consume media, and the way we communicate with one another. But it’s also begun to change the way we make decisions and even forecast the future.

Attempts to tap collective intelligence on the Web come in lots of different forms. In some cases, it happens without anyone really knowing: it emerges, in effect, from choices people make for themselves. The most obvious example of this is Google’s search engine, whose PageRank algorithm relies, to a large extent, on aggregating the links from one Web site to another. When people add links, or sometimes even when they click on them, they don’t know that they’re making Google’s search engine a little smarter, but that’s the by-product of their actions. Similarly, people use Delicious to bookmark pages (and to categorize them with labels) for their own purposes. But, when you aggregate all those bookmarks and labels, it turns out that Delicious users are collectively producing a surprisingly useful categorization scheme for the Web.

In other cases, of course, the attempt to tap people’s collective judgment is more obvious. Think of Wikipedia, or of ranking systems at places like Yelp and TripAdvisor, which let you use the collective judgment of travellers and diners as a guide to where to eat and where to stay. Prediction markets, like the late, lamented Intrade, let people all over the world bet on the outcomes of things like Presidential elections, often with surprisingly accurate results. Companies like Threadless now rely on the collective input of users for decisions about what products to make; Kickstarter lets people collectively decide which projects get funded. Of course, we’ve always had surveys and customer research. But in the past these tools were always limited by the perception (correct or not) that it was too difficult and too expensive to let too many people have a say. The advent of the Web has demolished that argument.

This doesn’t mean that the process lacks hurdles. Systems can be gamed—think of how much money has gone into search-engine optimization. And the anonymity of the Web, which is sometimes one of its strengths, also makes it possible for people and companies to post false reviews and rankings. More important, perhaps, is the problem of feedback loops. The wisdom of crowds works best when the people in those crowds are thinking relatively independently. But the Web sometimes creates echo chambers of opinion, and encourages people to pay too much attention to what everyone else is saying. The much discussed problems that Reddit encountered during the hunt for the Boston Marathon bombers, for instance, showed how social influence on the Web can actually make groups dumber rather than smarter.

Nonetheless, the Web has some crucial characteristics that make it an ideal vehicle for using collective intelligence to solve complex problems. Its openness and scale make it possible to aggregate the knowledge of people of diverse backgrounds and opinions, which is crucial for collective knowledge. Because you can, on the Web, easily ask thousands of people to weigh in on a subject, it’s more difficult for a few bad apples to corrupt the whole bunch. Perhaps most important, the Web makes it easier to aggregate information from non-obvious sources. One of the key ideas behind the wisdom of crowds is that useful insights and information can come from anywhere, even from places you wouldn’t at first think to look. The trouble had always been that finding useful information was difficult. The Web makes it more likely that the people who have such knowledge will find their way to you. That’s why smart organizations are now using the Web to harvest their employees’ knowledge, much of which had traditionally gone to waste. And the truth of collective intelligence on the Web is that its potential is still greater than its reality. What we see in this field in the next twenty years, not just from places like NASA but also from organizations that haven’t even been born yet, should go far beyond what we’ve seen in the past twenty.