James Surowiecki writes the financial column in The New Yorker, so even before his book The Wisdom of Crowds
came out I knew he did good research, and wrote well. But the book
exceeded my high expectations: It is the best-written book I've read in
years -- clear, clever, accessible, and jammed full of brilliant,
original thinking and compelling, supporting stories and reference. The
only disappointment is that, following an annoying recent tradition in
non-fiction, the book has no index. If you haven't read it already, put it on your list, and on your gift list -- I can't imagine anyone not learning something important from it, and it's fun to read.
The book begins with a taxonomy of three types of problems that individuals and groups try to solve:
- Cognition problems: Problems with one definitive answer
that we try to accurately assess after considering available and
missing information (e.g. what's a stock worth, who will win an
election, or what caused a disaster),
- Coordination problems: Where an optimal combined solution
is needed for a problem that affects a whole group, and where this
optimal solution is usually sought by having each individual act in
personal self-interest (e.g. finding buyers and sellers for products,
or determining the best route to work in traffic), and
- Cooperation problems: Where an optimal combined solution is
needed for a problem that
affects a whole group, and where this optimal solution usually depends
on individuals trusting each other and acting fairly and in what they
perceive to be collective self-interest rather than just their own (e.g. how to deal with pollution, devise a tax system, or remunerate employees).
One of Surowiecki's principal arguments is that all three types of
problems are best solved by canvassing groups (the larger the better)
of reasonably informed and engaged people. The group's answer, he
shows, is almost invariably better than any expert's answer, even
better than the best answer of the experts in the group. The group's
answer is the collective answer,
a term Surowiecki prefers to 'average' or 'consensus' answer, which
aren't always the same thing. And the superiority of the collective
answer depends importantly on the group's members having three
qualities:
- Intellectual diversity: Different opinions and perspectives
(unlike most management teams and boards, who tend to select others who
think the same way they do),
- Independence: Freedom from the tendency to want to agree automatically with what one or more other group members says, and
- Decentralization with Aggregation: Individual access to
different, specialized knowledge, and a mechanism for effectively
sharing that knowledge with the rest of the group.
Much of the book describes the three types of problems, with copious
examples (some of them quite entertaining), justifying the three
conditions for collective wisdom, and drawing some remarkable
inferences about what all this means to some important decision-making
processes in our modern world. Surowiecki argues that too much communication can actually render a group dysfunctional, victims of 'analysis paralysis'. Likewise, he demonstrates, too much
consensus or compromise produces weak, suboptimal solutions, since
opposing views, which must have had some basis for belief, are not
adequately aired and weighed.
Surowiecki drops his first bombshell early: "There is no evidence that
one can become expert in something as broad as decision-making, policy,
or strategy...or perhaps even management. ... Large groups of diverse
individuals will make more intelligent decisions than even the most
skilled decision-maker." The implication of this is that business
executives, expert consultants, investment analysts, egomaniacal
doctors and heads of state are not competent to make important
decisions related to cognitive, coordination or cooperation problems,
and should always defer to
the collective wisdom of large diverse groups when such problems arise.
In fact, the implication is that these individuals (who Surowiecki
argues are grossly overpaid) have far too much power in our society,
and may be doing us far more harm than good. Instinctively we have
always known that more egalitarian social structures produce better
answers; now we have some empirical evidence to support this 'uncommon'
wisdom. Our individual perception and intuition has been denigrated in
the civilized world to the point where the personal judgements of those
in high-ranking positions in our hierarchies unreasonably carry more
weight than group judgement. Get a second opinion, and a third and a
fourth... and you'll get a better answer and make a more intelligent
decision. These overpaid individuals would be much better employed
using their expertise to implement the group's decision instead of making it for them.
The book describes at length the phenomenon of groupthink
and how it biases groups' decisions and gives collective wisdom a bad
name. In fact there are four phenomena at work: The tendency of groups
to excessively rationalize away minority views as improbable, the
shyness of individuals to voice the first opposing view in the face of
an apparent consensus, the tendency to accept consensus of a small
number as inherent 'proof' of that consensus' validity, and the
bandwagon tendency of groups to be infected by what Gladwell in The Tipping Point
called an 'epidemic'. These are all subtly different phenomena, and
they're natural behaviours, but they're irrational, and have led to
great skepticism about collective wisdom. These phenomena show up in
wild swings in popular opinion, in the inexplicable and transient
popularity of crazes, in market 'bubbles', in mob violence, in our
willingness to let one person dominate the discussion and bully us into
accepting his view (even if he's not the most senior or eloquent person
in the group), and in our ability to be brainwashed and engage in
barbaric and irrational behaviours. No wonder, then, that so many view
the wisdom of crowds as an oxymoron, and are so easily seduced by
leaders and experts even when their records of decision-making have
been deplorable.
I recently attended a conference that had great promise, but which in
my view wasn't anywhere near living up to that promise. I spoke to a
few attendees individually and was astonished when they nodded their
heads in strong agreement after I said how disappointed I was. They
encouraged me to speak up, to rescue the event. But as a junior
attendee, I rationalized that it wasn't my place to hijack the meeting,
and said nothing. Truth is, I simply lacked the courage to do so.
Perhaps because he's an American, living in the land where selfishness
is seen as a virtue and 'natural' behaviour, Surowiecki agonizes at
some length over what he sees as the 'irrationality' of subordinating
personal self-interest to collective self-interest in Cooperation
Problem situations. But while he argues that this is a learned
behaviour, I would argue that it's an instinctive one. It has its
roots, I believe, in the subordination of the individual to the local
community, something innate in all animal species and evident in tribal
human communities as well. It's in our DNA. The reason it appears
irrational or counter-intuitive is that we've lost this sense of
community, and instead we have nuclear families and huge impersonal
political entities. Neither of these are communities, or even
fundamentally democratic structures as true communities are. But we've
got this sense that 'community comes first' in our genes, and it's not
surprising at all to me that we exhibit that instinctive altruism, even
with strangers: Giving to charity, caring about what's happening in
Sudan, taking 'irrational' responsibility. Our instincts often quietly
trump our reason.
This book forces you to rethink almost everything you believe. Here are
a few of its implications for the things I particularly care about:
- How to Save the World: Despite
Margaret Mead's famous saying about the impact of a few caring people,
maybe instead of my proposed 'think tank' of experts working on
solutions to our intractable problems, we need to ask a lot more people
for their take on what the coming scenarios might be, and how to avoid
the worst of them. At the very least, the think-tank cannot be
self-selecting, and its membership will need some careful and objective
crafting to ensure it meets Surowiecki's three qualifications.
- Innovation: My article
earlier this week reflects an amendment to my earlier thinking on
optimal innovation processes, lessening reliance on a few
unrepresentative 'pathfinder' customers and drawing more expansively on
collective wisdom, both to find and qualify innovative ideas.
- The Blogosphere:
Shirky's Law tells us that we tend to read a few, early-launched blogs
on any subject, and largely ignore the late-comers. We might be better
informed, and in a better position to make judgements and decisions on
these subjects, if we instead diversified our reading, selecting which
bloggers to read largely at random (Technorati has a tool
that allows you to find every blog that has written about a subject, by
typing in keywords). And perhaps Google, by pointing us to the most
popular sites on a subject rather than listing them randomly, may be
doing us an intellectual disservice, reinforcing conventional wisdom
instead of helping us find collective wisdom. That pretty well rules
out the blogosphere and its corners as true communities. But maybe it
means the fact my blogroll is so large and unwieldy, and the fact I
don't read any one blog with great regularity, is a good thing.
- Liberal vs. Conservative:
The polarization we are seeing in America may be more illusion than
fact. Perhaps what is happening is that a minority of both liberals and
conservatives are becoming (thanks to the Internet etc.) more informed,
and that is entrenching their beliefs, but the vast majority of people
are largely uninformed and are drifting with the tide, subject to all
the vagaries of Groupthink. That suggests that, despite the pollsters'
claims that very few US voters are undecided, a very large number may
still be up for grabs, and could swing en masse, perhaps
irrationally. It also suggests that the decision may not be as
cut-and-dried as we would like to think. The fact that there is no
collective wisdom on which is the better candidate may indicate that
the election result isn't as important as we all seem to believe.
That's a scary thought, to both sides. In the recent Canadian election,
the collective wisdom of voters was 'we don't trust any of you, so
we're going to elect a minority government to keep you all in check'.
And they did just that, despite enormous pressure to vote
'strategically'.
- Expertise Finders:
Obviously, if you buy Surowiecki's argument that experts aren't worth
their price tag, you have to wonder whether technology that will help
us find experts is really all that important, and whether perhaps we
should instead be looking at ways to create and tap into large, diverse
impromptu groups that can better help us decide what's really going to
happen in our businesses and what to do about it.
- Knowledge Management: The
wisdom of crowds belies the assessment that 'best practices' will or
can be identified and volunteered by individuals in the organization.
And when he points out the importance of 'Decentralization with
Aggregation' as a condition of high quality group judgement, Surowiecki
puts his finger on the greatest challenge of KM: How to aggregate each
individual's knowledge so that it can be effectively shared with the
entire organization. He even suggests some answers to this challenge,
which imply that neither 'filtered' knowledge (edited and culled by an
expert group) nor 'unfiltered' knowledge (simply collected with no
editing, rating or comment) is the optimal answer.
Ultimately, though, it's the well-picked and cleverly-told illustrations and examples, not the theory, that makes The Wisdom of Crowds
such a joy to read. When he's describing the ability of a group to
accurately pick the number of jellybeans in a jar, or explaining the
instinctive coordination 'rules' that allow huge flocks of birds to
sweep through the sky as if they were one huge organism, or describing
examples of how large groups solve problems brilliantly even when they
are not individually or collectively aware of the knowledge they are
bringing to bear in doing so, or explaining why we begrudgingly pay
taxes until we think the number of cheaters is too high, or why
television shows are so awful and why traffic jams occur and how they
might be eliminated without reducing the number of cars on the road,
Surowiecki is at his best. Buy it, read it more than once, cherish it,
think about all it means. This is a profoundly important book.
|
MADE IN CANADA
trust your
instincts 
