Social scientists have always been fascinated by crowds. From guessing the weight of a cow to identifying which company built the faulty part in the space shuttle Challenger disaster, the many have often been able to outguess the expert few. Crowd wisdom is often cited as the justification for the idea of efficient asset markets -- many investors, each weighing in with their buying and selling decisions, should combine to produce the optimal forecast of what a stock or a bond is really worth. Or so the story goes.

But there’s danger in relying on crowds to make decisions. Under certain circumstances, group wisdom can break down and become madness. A classic example of this is when Reddit users tried to identify the Boston Marathon bomber, and ended up accusing the wrong guy. Many believe that asset-market bubbles are also examples of crowds gone mad.

Why are crowds sometimes wise and sometimes mad? Social scientists already have a rough idea of the general answer to that question. Crowd wisdom works because people’s mistakes are haphazard and uncorrelated.

Everyone’s guess is a combination of signal and noise -- we have some idea of the real weight of a cow, or the real value of a stock, but we also have our own wrong ideas and preconceptions and irrationalities. But because my errors aren’t the same as yours, when you and I combine our guesses, the true knowledge shines through while the random errors tend to cancel out.

But when the people in a crowd communicate, their mistakes are no longer uncorrelated. When one person’s misjudgments influence another person’s thinking, the errors can snowball and wreck the whole forecast. Any number of studies confirms the general principle -- once people start talking and arguing and persuading each other, crowds turn into herds, and the magic disappears.

Why do people influence each other? There could be all sorts of reasons, both rational and irrational. People might simply have an instinct to copy other people’s actions, or take their word as gospel -- the old saw of “if you read it, it must be true.” Economists have built elaborate models of how rational herd behavior might cause bubbles and crashes. Alternatively, copying what other people do might be perfectly rational in many situations -- if you see everyone in a cafe suddenly run for the exits, it might be a good idea to follow as quickly as possible.

The difficulty comes in applying this insight to real-world problems. In real-life situations such as investing, it’s a certainty that most people have received some kind of information from others -- stock tips, Bloomberg News articles, investing advice, TV shows, etc. The question is how much people actually heed others’ opinions, as opposed to simply taking in factual information.

A team of researchers from Massachusetts Institute of Technology’s Sloan Neuroeconomics Lab may have found a new way to identify herd behavior before it strikes. In their paper, researchers Drazen Prelec, H. Sebastian Seung, and John McCoy ask forecasters a new and unusual question. In addition to simply asking people for their guesses, they also ask what people think others will guess.

If herd behavior is present, some people will know it, and will be contrarians -- they’ll guess something different from what they think other people will say. Prelec et al. find that the forecasts that receive the most contrarian support -- the guesses that people pick even though they think others will guess differently -- tend to be the right ones. They find that these forecasts, which they label the “surprisingly popular” options, tend to outperform standard crowd averages in a number of applications, with error rates more than 20 percent lower.

Herd behavior isn’t the only reason this method might work. Another possibility is the Dunning-Kruger effect -- the fact that ignorant people also tend to be ignorant of their own ignorance, while knowledgeable people know they’re better informed. This is closer to the explanation that Prelec et al. give for their result. But since herd behavior is the best-known force that breaks down the wisdom of crowds, it seems likely to me that any method that improves so much on traditional crowd-based forecasting does so by partially counteracting herd behavior’s effect.

In any case, this method obviously has some very important potential applications for finance. Hedge funds or other investors could poll investors, or their own analysts, using Prelec et al.’s method, and potentially beat the market. The Federal Reserve could use large groups of forecasters to identify when asset bubbles were happening, and try to pop the bubbles with interest rate hikes or other policy measures. And the government might loosen restrictions on short sellers, who tend to be contrarians.

In the search for the ultimate forecast, the wisdom of crowds might turn out to be very good, but not quite the best.

By Noah Smith

Noah Smith is a Bloomberg View columnist. – Ed.

Bloomberg