The Wisdom of Niches: Why Experts Still Matter

There is a war against expertise on the web and in the business press. We are beginning to believe that crowds are the answer to our problems, and that experts will be confined to stodgy old companies that don’t “get it.” I confess, I too was taken with the “wisdom of crowds” when I first encountered the concept. The democracy, the egalitarianism, the anti-establishment aspects… it’s a very sexy idea isn’t it? After a lot of thought I have decided that I’m switching sides. Crowds have their place, but experts live in niches, and for businesses that is where the real value lies.

The Problems With Crowds
James Suroweicki released Wisdom of Crowds in mid 2004. The main argument of the book is that under certain conditions, crowds can be very wise – even wiser than experts. One of his major claims comes from the show “Who Wants to Be a Millionaire.”

Everything we think we know about intelligence suggests that the smart individual would offer the most help. And, in fact, the “experts” did okay, offering the right answer-under pressure-almost 65 percent of the time. But they paled in comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

Back up a second. Have you ever seen that show? I wouldn’t say the “experts” really qualify as experts. What happens is that contestants get a question about ancient Egypt, they know that their cousin Jim is a Civil War buff, so they extrapolate that and say “Jim is good at history” and they call him. Of course he doesn’t know the answer because he is not really an expert in this subject, so all he can do is guess. Unless the contestant has a history professor on his list, what else could he do? Thus the expert success rate here is artificially low. What about the crowd? Well in all the shows I ever watched, the crowd was never polled on any of the final questions, which were the hardest. Most contestants used the crowd when there was an easy question that they thought the crowd would know.

Crowds are good at determining what is popular, which is really a tautology since popularity = what the crowd likes. Crowds are also good at guessing things that have a high randomness component, but that is about all. If crowds are so smart, why were there books written long ago about their madness? It’s because when it comes to actual knowledge, they are usually wrong.

Think about this – the crowd would say that evolution is false. Does that matter? No, because most of the crowd doesn’t know the difference between a genotype and a phenotype. If you don’t know basic biology then your opinion on evolution is irrelevant. Outside of the most basic knowledge, experts still matter. Niche groups know important things that crowds never will.

Warren Buffett has said that conventional wisdom is often long on convention and short on wisdom. Crowds fail in so many areas because they are conventional. Their ideas, by definition, represent conventional wisdom. Crowds don’t notice counterintuitive things. Crowds don’t turn off their cognitive biases. That is what being an expert is all about. Experts study something until they can look past the conventional wisdom of the subject.

Experts, Entrepreneurs, and Niches
In business, crowds are the current buzz. Reddit, Digg, and 70 social bookmarking sites are all about crowds and what they like. Entrepreneurs need to ignore it. Why? The opportunities are in expertise. The opportunities are in niches.

But that doesn’t make sense. We want to sell popular things, right? We want to sell things to the crowd. Let me ask you this – how many popular things started off as niche products? In 1995 I was in college working at Radio Shack. We sold big bulky cellular phones. We got great commission on them, so we really pushed them. Do you know how many people told me they would never have a need for one? Most people that came into the store said that. Now cell phones are ubiquitous.

Entrepreneurs need to realize that niche knowledge adds more value than crowd knowledge. Niche knowledge solves difficult problems. Crowd knowledge tells you about trends, but trends are fleeting. Niches are less about trends and more about vision – they are about what is possible, not what is popular. Experts studying something in depth can see opportunities that crowds will never see. All kinds of companies that are currently in niche industries like areas of nanotechnology, AI, neuroscience, biotech, etc. will be mainstream Fortune 1000 companies some day.

Aren’t Experts Often Wrong?
This is a complicated question, made more so by books and articles that discuss the failure of experts. I think you have to look at how we define expertise.

Let’s face it, some people are smarter than others. Smart people, while not always correct, are correct more often than other people. But the fact that experts are frequently wrong is a symptom of the stupidity of crowds. The problem is that our supposed experts aren’t always the smart people or the people that have studied an issue in depth. The “experts” are considered experts because they are popular, because they tell us what we want to hear, or because they are self-promoters and we have fallen prey to their tactics. Does anyone really believe Donald Trump is a business guru? If I was running a company I would put a nobody (at least, in most circles people don’t know him) like Kevin Rollins at the head of my company way before I would consider Trump to run it. There are plenty of superstars that never get their due because they don’t run their mouths about how great they are. There are plenty of experts that never become popular gurus.

The other problem with “experts” is that predicting future events is totally different from diagnosing problems. Crowds may be better at predicting changes in interest rates because we really don’t understand at a deep level everything that can affect interest rate changes. But what about diagnosing computer issues? In that arena, and many others like it, the experts will always outperform the crowd.

The point of all this is that it seems that everybody and their brother that is on the web these days wants to be an entrepreneur, and most of them want to build some new social app that harnesses the power of wisdom of crowds. Stop. There are still plenty of difficult problems in all kinds of fields that need to be solved, and those solutions will provide a better basis for a new startup than offering some software package that already exists but adding AJAX. Use your time and brain power to harness what the niche knows, and bring that to the crowd.

Why We Hate Niches
Why don’t we try to become experts in more areas? Why do we prefer breadth to depth? I think there are two reasons. First of all, breadth is easier than depth. Secondly, breadth impresses people because we can throw out this or that latest news fact and people find it interesting. We like to be liked, and breadth helps (and some breadth of knowledge is important). The truth though is that breadth helps you sound smart, but depth helps you be smart. And I think we are sacrificing breadth for depth these days.

Scott Berkun recently wrote about attention, and why we try so hard to be information rich even though so much of what we consume has no intellectual payoff. The thought processes in your brain depend on two things, inputs and structure. Scouring the web all day reading a million blog posts changes your inputs. Mastering an idea and achieving a deeper level of understanding about something changes your structure. Structural changes will lead to the revolutionary ideas that become great companies.

Niche knowledge adds value. If you want to be an entrepreneur, focus on a niche and you will find lots of interesting problems there to solve. Remember that sometimes the best way to go is away from the crowd.

Note: Don’t confuse peer production with wisdom of crowds. I’m still a big fan of peer production when incentives are properly aligned. Also note that I’m not totally against the idea of crowds. It has a place. The problem is that it is a small place and we are treating it like the future of business. It isn’t.

  • Great thinking Rob… even if it was a painful journey for you to reach this conclusion.

    But one advantage of the “wisdom of crowds fad” is more a open feedback system for decision makers.

    Most business decisions are made in a closed system. That system has (over time) removed dissenting voices and disconfirming evidence. Now when you remove dissent and disconfirming evidence you become less smart. And over time the lack of dissent and disconfirming evidence will actually make you stupid. That’s a reason why closed groups (groupthink) can cause disasters.

    Now as you write crowds include a lot of ignorant and crazy people with worthless input. But they also include smart dissenters and people with disconfirming evidence who have been removed from giving feedback. By opening things up those voices can get heard and decision makers are forced to stop being so stupid.

    A great case in point is made by the slow and painful (yet ultimately positive) changes you are seeing in the news business. They are getting feedback from peopel normally closed out of the reporting business. And the quality of their decisions I believe is (slowly yes!) improving.

  • Rob

    Exactly, which is why I didn’t address issues like internal decision markets at companies. My complaint isn’t that crowds are useless, it’s that people are starting to imply that crowds are better than experts across the board.

    The Business Experiment was a perfect example. The crowd was dead on with the broad idea of where there was a market need, but could not solve all the various implementation problems associated with it. For instance, the legal issues needed one smart legal person, not a crowd.

    Entrepreneurs need to figure out how to use the crowd for only the things crowds are good for, and not to ignore the value of expertise. Probably the most important thing is for them to understand the differences with each.

    Wisdom of crowds is becoming the panacea, when we need to remember it is just a tool in the kit.

  • J

    Don’t discount breadth entirely. One advantage it can confer is helping you know what you don’t know. Also, I agree with you’re criticisms of crowds for the most part, but what about a crowd of experts? Wouldn’t their input have value, particularly in a situation where a business is run by a niche expert with little or no operational knowledge?

  • Brilliant essay, thanks Rob.

    I think that there isn’t an experts vs. crowds issue at all as long as we use each for its intended purpose …

    Experts should never be asked for their opinions, only for their expertise.

    Crowds should never be asked for expertise, only for opinions.

    What do I mean?

    All problems can be solved by either a) knowing their outcome or b) guessing their outcome. “Expert” & “expertise” are derived from the same word used for “experience” – i.e. some first-hand know-how earned through “being there”. Problems that can be solved by drawing on “history” should always be put to the expert who has previously solved the mots similar problems.

    Forward-looking problems can seldom be solved through expertise. We actually set experts up to fail when we let them become pundits.

    When developing opinions about the future that have no precident, it’s obvious that a large group consensud is the best solution we have.

  • Rob

    I agree that breadth has value. It helps spot related opportunities. It is also important in building what Charlie Munger calls a lattice of knowledge. The problem is that I see people going broader and shallower all the time.

    Great points.

  • Exactly “Wisdom of crowds is becoming the panacea, when we need to remember it is just a tool in the kit.”

    And it’s not a new tool either. Leadership (and that means commercial leadership too) has been defined as “finding a parade and getting in front” for centuries.

  • You’re right that there seems to be a fad going on here. But I’d observe that to the extent that there’s a general reaction against “experts” in our society, it’s in part because too many disciplines are making spurious claims of expertise. The knowledge one gains in ed school, for example, is *not* expertise in the same sense that chemistry is expertise. The knowledge that one gains in journalism school is not expertise in the same sense that skill at graphics arts is expertise.

  • You compare the knowledge of a single expert to that of the crowd. And wisdom, a trait of an individual, to something epherial – the sort of knowledge that we might value emergent in or from a crowd or group. Valuing this last is almost new, much as the democracy political structure. If not wisdom, what is it? Trend spotting? Statistical market opinion?

    And a second comment: We do not commonly consider experts to have wisdom either. Experts have and provide educated, in-depth knowledge in niche areas. Wisdom is something else again.

  • J

    I should clarify – breadth enhances your ability to know THAT you don’t know, not WHAT you don’t know. Bad wording on my part there.

  • Mark makes a great point.

    My favorite quote about business decisions is that “you need to be widely informed and need to think straight. There are no exotic models or magic formulas you can depend on.”

    But in my experience no one person can be informed widely enough nor can they consistantly think straight.

    Therefore great business decisions takes a “team” with the right people all willing to communicate, coordinate, and collaborate. It’s kind of a wisdom of crowds thing without “the ignorant and the crazy.”

  • Rob

    I disagree with that. Fermat’s Last Theorem was proven by a mathematician who worked on the problem for years, not a lawyer with a little interest in math.

    On the flip side, reading a puff piece in the NYTimes about chaos theory doesn’t mean I really understand it.

    Granted, we can find reveresed examples. Much of Darwin’s thought was influenced by Thomas Malthus. New ideas in A.I. come from all over the place, including fields as broad as biology, neuroscience, computer science, electrical engineering, robotics, linguistics, philosophy and psychology.

    I agree that horizontal connections are important. They are the basis of creativity. But there is difference between knowing about something and understanding something.

    It seems to me (and this is admittedly, purely anecdotal) that people tend to skim lots of sources and then mention them in passing as if they know what they are talking about. They are trying to sound smart.

    On the flip side, I think there are lots of people in this world that never dig in and understand anything – not one single idea – on a decent level of depth. I equate it to reading music. At first you sit and think about the notes. Then you start to recognize some patterns. A ‘C’ maps directly to a finger position instead of having that intermediate ‘oh that’s a C’ step. Then one day you don’t even think about what chord it is, you just play. I think people can do this in business too. You start to *feel* things, not in an emotional sense but in a know-it-like-the-back-of-your-hand sense.

    I’m not disagreeing that breadth is important. I just think most people fall into your description of “throwing out half understood terms.”

    What prompted the post is that people no longer seem to see the value of digging deep into an area of knowledge, and I think that still has tremendous value. Depth is being ignored in exchange for ever shallower breadth.

  • Sayo

    I think that that book should be used only for adults not freshmen that ARE FOCED TO READ IT ITS BORING SORRY BUT ITS THE TRUTH

  • They are the basis of creativity. But there is difference between knowing about something and understanding something. Host Decker Marketing goes for the straightfoward approach and kindly includes my post about comparing Google to a couple of new niche search. Just exactly what I wanted to say.

  • i think an expert’s opinion is welcomed but let’s face the fact that crowds have been and will always influence the power of the leader(s).