The Wisdom of Niches: Why Experts Still Matter

There is a war against expertise on the web and in the business press. We are beginning to believe that crowds are the answer to our problems, and that experts will be confined to stodgy old companies that don’t “get it.” I confess, I too was taken with the “wisdom of crowds” when I first encountered the concept. The democracy, the egalitarianism, the anti-establishment aspects… it’s a very sexy idea isn’t it? After a lot of thought I have decided that I’m switching sides. Crowds have their place, but experts live in niches, and for businesses that is where the real value lies.

The Problems With Crowds
James Suroweicki released Wisdom of Crowds in mid 2004. The main argument of the book is that under certain conditions, crowds can be very wise – even wiser than experts. One of his major claims comes from the show “Who Wants to Be a Millionaire.”

Everything we think we know about intelligence suggests that the smart individual would offer the most help. And, in fact, the “experts” did okay, offering the right answer-under pressure-almost 65 percent of the time. But they paled in comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

Back up a second. Have you ever seen that show? I wouldn’t say the “experts” really qualify as experts. What happens is that contestants get a question about ancient Egypt, they know that their cousin Jim is a Civil War buff, so they extrapolate that and say “Jim is good at history” and they call him. Of course he doesn’t know the answer because he is not really an expert in this subject, so all he can do is guess. Unless the contestant has a history professor on his list, what else could he do? Thus the expert success rate here is artificially low. What about the crowd? Well in all the shows I ever watched, the crowd was never polled on any of the final questions, which were the hardest. Most contestants used the crowd when there was an easy question that they thought the crowd would know.

Crowds are good at determining what is popular, which is really a tautology since popularity = what the crowd likes. Crowds are also good at guessing things that have a high randomness component, but that is about all. If crowds are so smart, why were there books written long ago about their madness? It’s because when it comes to actual knowledge, they are usually wrong.

Think about this – the crowd would say that evolution is false. Does that matter? No, because most of the crowd doesn’t know the difference between a genotype and a phenotype. If you don’t know basic biology then your opinion on evolution is irrelevant. Outside of the most basic knowledge, experts still matter. Niche groups know important things that crowds never will.

Warren Buffett has said that conventional wisdom is often long on convention and short on wisdom. Crowds fail in so many areas because they are conventional. Their ideas, by definition, represent conventional wisdom. Crowds don’t notice counterintuitive things. Crowds don’t turn off their cognitive biases. That is what being an expert is all about. Experts study something until they can look past the conventional wisdom of the subject.

Experts, Entrepreneurs, and Niches
In business, crowds are the current buzz. Reddit, Digg, and 70 social bookmarking sites are all about crowds and what they like. Entrepreneurs need to ignore it. Why? The opportunities are in expertise. The opportunities are in niches.

But that doesn’t make sense. We want to sell popular things, right? We want to sell things to the crowd. Let me ask you this – how many popular things started off as niche products? In 1995 I was in college working at Radio Shack. We sold big bulky cellular phones. We got great commission on them, so we really pushed them. Do you know how many people told me they would never have a need for one? Most people that came into the store said that. Now cell phones are ubiquitous.

Entrepreneurs need to realize that niche knowledge adds more value than crowd knowledge. Niche knowledge solves difficult problems. Crowd knowledge tells you about trends, but trends are fleeting. Niches are less about trends and more about vision – they are about what is possible, not what is popular. Experts studying something in depth can see opportunities that crowds will never see. All kinds of companies that are currently in niche industries like areas of nanotechnology, AI, neuroscience, biotech, etc. will be mainstream Fortune 1000 companies some day.

Aren’t Experts Often Wrong?
This is a complicated question, made more so by books and articles that discuss the failure of experts. I think you have to look at how we define expertise.

Let’s face it, some people are smarter than others. Smart people, while not always correct, are correct more often than other people. But the fact that experts are frequently wrong is a symptom of the stupidity of crowds. The problem is that our supposed experts aren’t always the smart people or the people that have studied an issue in depth. The “experts” are considered experts because they are popular, because they tell us what we want to hear, or because they are self-promoters and we have fallen prey to their tactics. Does anyone really believe Donald Trump is a business guru? If I was running a company I would put a nobody (at least, in most circles people don’t know him) like Kevin Rollins at the head of my company way before I would consider Trump to run it. There are plenty of superstars that never get their due because they don’t run their mouths about how great they are. There are plenty of experts that never become popular gurus.

The other problem with “experts” is that predicting future events is totally different from diagnosing problems. Crowds may be better at predicting changes in interest rates because we really don’t understand at a deep level everything that can affect interest rate changes. But what about diagnosing computer issues? In that arena, and many others like it, the experts will always outperform the crowd.

The point of all this is that it seems that everybody and their brother that is on the web these days wants to be an entrepreneur, and most of them want to build some new social app that harnesses the power of wisdom of crowds. Stop. There are still plenty of difficult problems in all kinds of fields that need to be solved, and those solutions will provide a better basis for a new startup than offering some software package that already exists but adding AJAX. Use your time and brain power to harness what the niche knows, and bring that to the crowd.

Why We Hate Niches
Why don’t we try to become experts in more areas? Why do we prefer breadth to depth? I think there are two reasons. First of all, breadth is easier than depth. Secondly, breadth impresses people because we can throw out this or that latest news fact and people find it interesting. We like to be liked, and breadth helps (and some breadth of knowledge is important). The truth though is that breadth helps you sound smart, but depth helps you be smart. And I think we are sacrificing breadth for depth these days.

Scott Berkun recently wrote about attention, and why we try so hard to be information rich even though so much of what we consume has no intellectual payoff. The thought processes in your brain depend on two things, inputs and structure. Scouring the web all day reading a million blog posts changes your inputs. Mastering an idea and achieving a deeper level of understanding about something changes your structure. Structural changes will lead to the revolutionary ideas that become great companies.

Niche knowledge adds value. If you want to be an entrepreneur, focus on a niche and you will find lots of interesting problems there to solve. Remember that sometimes the best way to go is away from the crowd.

Note: Don’t confuse peer production with wisdom of crowds. I’m still a big fan of peer production when incentives are properly aligned. Also note that I’m not totally against the idea of crowds. It has a place. The problem is that it is a small place and we are treating it like the future of business. It isn’t.