The Business Experiment: What I Learned About the Limits of Peer Production, Wisdom of Crowds, and Other Web2.0 Philosophies

Now Reading
The Business Experiment: What I Learned About the Limits of Peer Production, Wisdom of Crowds, and Other Web2.0 Philosophies

  • Facebook
  • Twitter
  • LinkedIn

I have been promising a handful of people since March that I would write this post. Finally, after months of letting brew in the back of my mind, I think I am ready. Over the last 4 years I have taken part in numerous projects of various sizes that fall into a peer production or wisdom of crowds model of some sort. These projects have had various levels of success and with all the hype surrounding these ideas, I feel the need to voice some caution based on what I have learned. As regular readers know, I don't have very high hopes for "Web2.0." I think it is a bubble, and outside of bringing us some new technologies like AJAX, I suspect 9 out of 10 companies will fail because they do not have an underlying business model that is economically sound.

This post is divided into two parts. The first part explains the projects, how they worked, and provides some tips on why things happened and what I learned. The second part discusses the major lessons I pulled out of these experiences, and how you can apply them to make sure you use crowd behavior, etc. in the right way. If you find part 1 a bit boring, please scroll down to part 2.

Part 1 – The Three Main Projects

1. The Carnival of the Capitalists
The Carnival of the Capitalsts was something that Jay and I modeled on the Carnival of the Vanities but we gave it the niche focus of business and economics. It was one of the first niche carnivals to arise, and has been one of the longest running blog carnivals.

The CotC has been popular in the business blososphere because of the way that work is distributed. Bloggers sign up to be a host, and that week they compile and present the entries on their blog. It takes a half day, sometimes a lot more, depending on the number of entries and the quality of presentation.

Later in this post I will talk about the importance of proper incentives when it comes to group work like a blog carnival. I think one reason the CotC has been running for almost three years is because the incentives are properly aligned for everyone involved. Hosting is a lot of work, but bloggers host only once or twice a year, and as a reward, they get lots of extra traffic as everyone links to the carnival. Bloggers submit posts every week because post submission is easy, and it guarantees you a small but important amount of new exposure and traffic.

2. Morespace

Morespace began when Todd Sattersten proposed the idea that lots of business bloggers had just as much interesting stuff to say as the business gurus. Todd wondered what would happen if bloggers were give "more space" to write, mainly, 5,000-10,000 words in a book chapter instead of the usual couple hundred that make up a blog post.

Todd put up the seed money for the project. The writers worked together, for free, with the idea that any profits would be shared among us all. I think the book turned out pretty good. With 9 different topics from 9 different authors, you are bound to find at least an essay or two that you like. But business books don't sell well in general, and despite great reviews and some good blog promotion, Morespace only sold a few hundred copies.

Even though this was a peer production project, it was ultimately successful because it did have a leader driving it to completion. One of the lessons I learned as I contrasted MoreSpace to TBE (which is discussed later) is the importance of having someone in charge. Otherwise, people are much less likely to deliver on their part of the work.

The incentives were well aligned for Morespace because there was nothing to lose but time and most of us weren't published authors. So even if the book didn't sell, we had the chance to say we had written part of a book. Thus even though the project never had large scale sales success, the authors all received a benefit simply from participation.

3. The Business Experiment
The Business Experiment began as an idea in February 2005, after I read James Suroweicki's "Wisdom of Crowds." I wondered if a crowd could make better strategic decisions than a handful of upper level managers, and decided it could be tested (sort of). In June 2005 I launched the site, which quickly grew to over 500 members. My initial plan was to let the crowd choose a business, and have them vote on all the major strategic decisions of the company. It made some nice waves around the blogosphere.

Early on their were problems. First of all, the legal structure involved in giving equity to what eventually became 900 people was expensive because it required us to file a private placement offering. The legal bills involved were mounting. I naively assumed that someone with more money than sense would think it was a cool project, and would throw 25K at it just to see what would happen. That money could have been used to outsource work, take care of legal costs, and would have helped test the wisdom of crowds concept in a much better way.

For the first few weeks, people submitted business ideas to the forums. The ideas were discussed and debated, and we began a tournament style vote off. We voted for the best idea out of a group of five, with the winners advancing to the next round to be paired against other winning ideas. Eventually, the winner (Askspace, a small business wisdom-of-crowds consulting website) was chosen. From there, the project quickly deteriorated.

We didn't receive any money, which means that the members were going to have to do the work, not just vote. This began a debate about how much equity to give for certain types of work, who would be in charge, who/how is work verfied for completeness and accuracy, etc. You can see one of my first "lessons learned" posts here.

In addition to the legal problems, participation in the site dropped off dramatically after the final business idea was chosen. Approximately 1/3 of the members never logged in again. They did not like the winning idea and therefore had no interest in continuing with the project. And in retrospect, this was one of my biggest mistakes. If I had set the idea up front instead of letting everyone vote, I would have attracted the right people – people that liked that particular idea. Instead I attracted people that primarily liked to discuss ideas, and hoped their idea would be picked.

The project lacked leadership, and I was very reluctant to make it "Rob's project," so I sought to have a TBE member step up to lead. Sean Clauson, a young database programmer from Minneapolis, answered the call. Sean built a leadership team and really drove the project forward.

Over a 9 month stretch, lots of smart successful people worked on TBE or participated in the project in some way, shape, or form. In March of 2006 though, we came to realize that the way we operated was no different from most other companies. Decisions were made at high levels because there wasn't time to put everything up to a vote to the crowd. In addition, the crowd sometimes voted in wasy that gave us conflicting direction. Difficult decisions were never popular. At that point, the executive team realized how difficult it would be to pull the whole thing off, and I made the case for why the project should be discontinued. Still, in the spirit of the project, I let the crowd vote. It was fairly close, but they voted to shut down the project.

I learned a ton of stuff from TBE, and met some fantastic people. The executive team that ran TBE was sharp, and really gave it a great shot. And let me just say that Carolyn Burke is an absolute superstar in every sense of the word. Everyone on the team was smart and dedicated, but Carolyn was unlike anyone I have ever worked with.

What I Learned From All of This

The wisdom of crowds is huge right now. I frequently hear about this or that company (usually a "Web 2.0" company) that is going to tap into the wisdom of crowds for this or that, or start a user driven site for ______ (fill in the blank with whatever). Old businesses are dead, it's all about embracing edge competencies and network effects, etc. etc. Sorry, but I'm calling bullshit. I don't buy it. Business is still business. It is still about doing something that is valuable to the customer. Usually, this involves making things easier, which is the exact opposite of Web 2.0. Sure, Web 2.0 companies make things easier for some hard core web users, but most of the world – the people that just use the web to search for information and check email – aren't any better off. People will talk about Web2.0 being all about the "long tail," but I think Web2.0 itself is a long tail. Maybe I'm just bitter because TBE wasn't the hit that I hoped it would be. Or maybe I'm smart and have learned a lesson that most people haven't. It really doesn't matter because 90% of you are committed to one view or the other and are whether you agree or disagree with me won't be affected by reading this post. For those of you that do want to give this some real thought, here are the things you should consider when looking at these Web2.0 philosophies.

1. Incentives Matter
This cannot be overstated. Most people don't do random altruistic things just to make the world a better place. People need incentives to act, and they are different for everyone. Some people need money, some need power, some need popularity, some need to express themselves, some need to satisfy their curiosity… the list goes on and on. One problem with TBE was that incentives were intangible, long-term, and likely to never materialize. That isn't a good way to get people involved. People used to be impressed that the executive team at TBE put so much time into the project because it seemed to provide no return. But I got to know those people well, and the incentive for them was knowledge, satisfaction of their curiosity, and the desire to succeed. I think they will all say that they learned a great deal from the experience.

The most important piece of advice I can give you about this is to make your site/product/software useful even if you only have one user. People keep trying to create sites where users can share all this stuff, but unless you build a critical mass, the site isn't very useful. Something like, on the other hand, is useful even without the user base. The incentive to use it is to manage your bookmarks in an effective way.

2. Think Marginal
Economists often talk about marginality, which means "the value of the next unit." When establishing incentives for group participation, you have to look at the marginal value of the incentive, not the absolute value. Five dollars is a lot to someone that makes $20 a day, but to someone that makes $1000 a day, $5 is largely irrelevant. TBE members were a successful bunch, which I think may have worked against getting things done. Successful people that have enough money, contacts, ideas, etc. are most likely lacking in time, so the value of sacrificing one hour is very high.

It is easy to fall into the mind trap of thinking "surely people will spend just 30 minutes a week to do…" but 30 minutes a week is a lot, and your incentive may not make it worth the time for people that are already successful. Your site may add value to what someone is doing, but it has to have a high marginal value to get used. You can't bet your business on the fact that your product or service is valuable to some random user in a perfect world.

Blogging is a great example. Bloggers think you should read their blog because it is good, but that is irrelevant. There are lots of good blogs. I don't have time to read them all and neither do you. If I'm reading two blogs a day, the attention cost to add a third one is pretty low, so I don't need a high incentive to add your site. If I'm reading 150 blogs a day, you better be far above anything else I am reading because I need a high incentive to make it worth my time.

Take the example of hosting the Carnival of the Capitalists. If you do it on rare occasion, it is worth the time for the extra traffic bump. Once your traffic gets to a certain level, several hundred unique visitors doesn't make a difference and thus extremely popular blogs are, on average, less likely to host.

So don't ask yourself if your product/service provides value, ask if it provides enough value to be worth the time of busy consumers.

3. Crowds aren't smart – except under a few specific conditions.
Wisdom of crowds fails in many cases because the members making the decision don't understand the context surrounding it. You shouldn't rely on a crowd for advice in your industry, because the crowd doesn't know anything about your industry.

The wisdom of crowds is a byproduct of other processes, not a process in and of itself. People trying to figure out how to harness the wisdom of crowds are already thinking about it the wrong way. It cannot be your end goal. You have to establish a process that allows people to make their own individual decisions and reap the rewards and consequences of their decisions individually. Then, and only then, can you look to see if you can tap into the collective wisdom for some extra benefit.

Wisdom of crowds is really a misnomer. It should be referred to as "the aggregated wisdom of lots of individual decision makers." Crowds tend to feed of themselves and they get stupid as a result. Unless members of the crowd have an incentive to act honestly and seriously, and aren't influenced by other crowd members, the results of the crowd will likely be wrong.

4. Peer production has extremely limited uses do the necessity of understanding the context associated with work.
The plan for TBE was to build the business by "microchunking" all the work to the members. My theory was that people would self select by picking microchunks that they were interested in and could handle reasonably well. But a microchunk isn't really just a simple task – it comes with a history. Much thought and time and action has been put into whatever it is to get it to the current state. An understanding of that history is necessary if you are going to move the task forward, even if the work itself (the microchunk) only takes a few minutes.

For instance, Morespace authors each got to pick a topic, which made the project work. If we each had to write a chapter that had to fit in with everything everyone else wrote, it would have complicated the project significantly because we would have needed to understand the broader context of the book.

5. Crowds don't pick difficult solutions.
What I mean is that crowds often do what is popular, fun, exciting, novel, etc. Crowds don't make decisions that say "let's bite the bullet and endure this temporary pain so that we can reap the long-term benefit." If you don't believe me, look at our political system. Decisions are focused on election cycle time periods, not the long-term health of the country.

Some great ideas were thrown out at the beginning of TBE, like starting basic businesses that were easy to understand, but the crowd didn't like those ideas. The crowd did not want to do something boring, they wanted to do something cool and cutting edge.

6. Groups need leaders because they need direction.
If everybody has an equal voice, it can lead to lots of talking in circles and very little productive decision making. Occasionally, the CotC has an issue that Jay and I have to address. Todd Sattersten had to put his foot down several times when Morespace got off track. Sean Clauson had to reign in TBE members that wanted to go all over the place. The self directed nature I've hoped to see in large groups has never materialized.

I admit that back when TBE first began, and I had never heard the term Web 2.0, I thought we were on to something. Altruism, sharing, group decision making – it's sexy. It's alluring. Egalitarianism will make you popular. Talking about the genius of the average Joe will make the average Joe like you quite a bit. And therein lies the real problem.

Human beings are full of cognitive biases, driven by emotional decision making, often irrational, unanalytical, and flat out weird. Group decision making minimalizes the impact of the rare individual that can overcome human cognitive faults.

We talk about the excellence of Wikipedia or Linux or some other group contribution project, but these things are driven forward by a handful of excellent individuals, not the crowd at large. Group contribution simply serves as a means to help identify those individuals. Look at Sourceforge. Ninety percent of the projects never become real software. So much for the power of groups.

I know this post has been all over the place, and I am sure that many of you disagree with much of it, but my experiences have led me to these conclusions, and I felt it was important to finally share them. If you don't find them valuable then keep in mind that you didn't pay anything to read this. If you have specific questions about something, you might have better luck sending me email (rob-at-businesspundit-dot-com) so I can file it away for a response. I'm soon boarding a plane for 25 hours of travel and may not be checking in on the comments very much with the limited web connectivity that will be available.

Leave a response
  • July 26, 2006 at 11:29 am

    Thanks for putting your thoughts in writing Rob. As a outside participant in TBE and a reader of Morespace I agree some with what you conclude but also have a lot of different thoughts and different memories of the “siginificant” incidents. (The waste of time spent on a logo before TBE had a product that could attract flies was one.)

    But when you write that “groups need leaders because they need direction” you should add that groups need leaders who “know” the right direction or at least have the discipline to quickly land on the “right” direction.

  • July 26, 2006 at 2:43 pm

    I think there is an important point here about motivation. Is it intrinsic or extrinsic?

    More Space was a combination. People wanted to be involved, but they also thought there might be a payoff for them (however samll).

    TBE depended on intrinsic and the extrinsic motivations weren’t strong enough (I am definitely speaking of myself in this instance).

    COTC I think is about extrinsic. It is about more people reading the stuff someone wrote.

  • Anthony D.
    July 28, 2006 at 9:44 am

    You may have only learned one thing: crowds cannot be controlled. People need to be able to create their own path and follow their own leads. Creating a single project with a large crowd is a disaster waiting to happen – especially when most of the individuals want it to be their way… their project. I would follow to see how they fare with the wisdom of crowds. I think they understand the fundamentals more than yourself and the leaders of TBE did. The wisdom of crowds is not flawed, it is just harder to execute than you may imagine.

  • Ignat
    July 31, 2006 at 12:50 pm

    Great conclusions, Rob. Thanks for sharing! I liked Carolyn Burke’s work too very much. She was awesome. And thanks for the link with Carolyn’s picture. I always wanted to know how she looks like, but was afraid to ask. :-) Thanks!

  • ken
    July 31, 2006 at 1:40 pm

    Absolutely wisdom of crows is B.S.

    The world has always been and will progress because of elites, whatever the shape and form of those elites (even in democracies) for the very simple reason that the average I.Q. in the West is 100. Starting at 110 you get into the trully intelligent band.

    Besides, by definition, a minority is right and the majority is wrong. Perhaps this also explains why Western democracies are grinding to a halt. Too many special interests to accomodate.

    my two cents from someone who seldom posts comments :)

  • ken
    July 31, 2006 at 1:43 pm

    Absolutely wisdom of crows is B.S.

    The world has always and will progress because of elites, whatever the shape and form of those elites (even in democracies) for the very simple reason that the average I.Q. in the West is 100 (90% of the population). Starting at 110 you get into the trully intelligent band. That 10% creates the system/establishment.

    Besides, by definition, the minority is right and the majority is wrong. Perhaps this also explains why Western democracies are grinding to a halt. Too many special interests to accomodate.

    my two cents from someone who seldom posts comments :)

  • August 3, 2006 at 11:45 am

    Hey all – My experience with TBE and AskSpace was invaluable. For me, it was a personal success, working with a distributed and self-selecting team, and in a volunteer capacity. Lots of good lessons. I followed up that success by immediately volunteering for a non-profit, which immediately signed a contratc with me. Here’s my most recent picture as Communications Director of the Singularity Institute for Artifical Intelligence:

    So Rob, Sean, Ignat, Jonathan, et al – Thanks!

    ps.. Here’s a cool new toy at linkedin:

  • August 8, 2006 at 8:08 am

    interesting insights. The wisdom of crowds do not really exist, better talk of the Wisdom of wise crowds. Furthermore i agree on you that collective wisdom often leads to collective stupidity when there is no expert input. See more on this in two recent posts on my blog:

  • August 27, 2007 at 4:05 pm

    I am a member of a new start up that is a co-operative site. This article is a great study and will be forwarding this to all our members. Currently the site is not considered live and we are in development. We are looking for a Director of IT…please email if anyone is interested…

  • August 19, 2008 at 3:19 pm

    Thank you, Rob, for my part, too. It’s very valuable that people take the time to share this kind of experiences. I’m doing a pretty TBEish project in It’s also a site where people discuss business ideas. Difference is that all the ideas are socially relevant and people are also invited to an event which enables them make their own favorite ideas happen.
    Now I learned new risks I didn’t think about before, e.g. the site attracting just people who like to discuss ideas, not to execute them, or busy people not contributing because of not enough incentive. Now starting to think about how to avoid those..!

  • March 29, 2009 at 5:26 pm

    Great insights and just so you know, some of us former TBE’ers are still around. Keep in touch.

  • Rich
    July 1, 2009 at 2:27 am

    You’re dead right that crowds can be stupid. No further proof needed than the date of the economy, and the “failure” of the efficient market. In the boom, everyone in the markets believed
    – house prices could continue rising indefinitely
    – cheap credit was indefinitely sustainable
    – financial innovation had brought about an end to boom’n’bust

    The investor community put its money where its mouth was, and lost $$$$$ – a collectively wise crowd? I think not.

  • November 3, 2010 at 5:47 pm

    I find this post fascinating, because we are trying to do something similar with Bettermeans, but I’m hoping that we have made some incremental changes that will allow a team just like the one you are describing to work better. In our webapp that we are launching next week, we are excited about transparency and allowing the crowd to have a voice, but what is different is that ultimately the team members (elected in by the group) are the ones that make the decision.

    I would be fascinated to speak with you more about this topic and some of the things you learned and what you think would have made it better.

Leave a Response