One thing that justifies this much attention on Collins’ study is that the quantity of available data is so large that you can ask more probing statistical questions. The conventional stat. significance threshold in social sciences is 5%, which would indeed yield lots of “significant results” that are completely meaningless. If you wanted to use the methodology for Buffett, though, you could easily go with .001, or even lower, just because the data are so numerous: Maybe he will flunk that test — I haven’t run the numbers. But if he does pass it, you can have very high confidence that selection bias is NOT the cause. (You might even be able to run a full Bayesian analysis, to assess the conditional probability that he is extraordinary. But that’s getting a bit arcane here.)

I still stand by my earlier comment, though, that this flawed study — statistics or no statistics — is more useful than a consultant’s hunch. For one thing, we can dissect the study as we are doing here, and appreciate its limitations as well as its virtues.

For another, I believe that GtG and BtL have a salutary effect on business discourse. The conclusions are often contrary to conventional business thinking. But unlike a lot of the latter, they are backed up by SOME real data, not merely antisocial excuses masquerading as “hard-headed business logic”.

Overall, methodological critiques are a good thing. If the first thing you think when you see statistics or any research is “how did they screw this up?” — you’re in good company.

On the other hand, tossing out a half-baked list of potential problems with a book addressing a comprehensive research program is capricious at best. The methodology is based in behavioral science and if only platitudes are gleaned from it, that appeared to me intentional. The book is supposed to be describing the similarities between a subset of companies that experienced unprecedented success.

No kidding, the companies regressed toward the mean over time. J. MacAuslen was right to think of probability, although his plan to test whether a company performed better than random would produce consistent, unhelpful, significant results.

Here’s the ACTUAL problem with the book (confirmed by another commenter’s 2007 comparison of the G2G companies’ S&P): Regression toward the mean (and here’s an excellent student-written explanation of the phenomenon : http://www.socialresearchmethods.net/tutorial/Cheng/lcheng.htm)

Other critiques of G2G sound too much like old standby criticism to drag out in a grad school response paper… “sample was drawn from college students only” or “results may vary by gender/race/location/day/time/etc”

I personally liked the add-on monograph, Good to Great and the Social Sectors – Why Business Thinking is Not the Answer. The case studies were selected as examples, but did not emerge from a major data based study like in G2G. If we’re questioning the validity of his selection criteria, Good to Great and the Social Sectors dodges that bullet altogether. I *strongly* recommend starting with the 35page 2005 follow-up. There’s just enough information about the main concepts, like hedgehog, that you may want to buy G2G to elaborate. But, if you’re unimpressed, you saved yourself a lot of fluff reading and still retained the main concepts (with examples that AREN’T of failing business!)

Jim’s methodology could be used to find that companies with red stars in their logos do better (or blue bars or gold stripes – you’d need to actually do the research to find out which) but this wouldn’t be a good reason to switch your corporate logo. Likewise, “people-first” might make sense sometimes and not at other times (just to take one example). Thus, business leaders who follow his advice might end up, as you say, focusing on things that don’t matter in search of a silver bullet.

The findings have not been replicated (i.e. proven) and though many have intuitive appeal some have been at least partially refuted (take a look at the link I submitted above). I can’t see how your analogy with the marathon runner fits – it’s more like if most successful marathon runners drink soy milk then aspiring marathon runners should too – it may or may not be a good idea. And in business, focusing on something that may or may not add value is rarely a great idea. For this reason, I’ve not heard of a whole lot of “smart business people [raving] about this book.”

To be a world class marathon runner you have to eat healthy, run every day, have a great coach, and put your heart into running. That doesn’t mean that by finding someone who follows these things and is not a great marathon runner that the principles are useless. Conversely it doesn’t mean that by finding an exceptional person who is a great mararthon runner and doesn’t follow these principles that you would advise others to negelect the fundamentals.

From a scientific point of the argument pointed out here does put holes in his book. But for anyone who has run a business or been deeply involved they understand that Jim’s principles are right on and they serve as a reminder that we always need to focus on the fundamentals. We don’t need it to be proven with the scientific method to understand that… we just like knowing that Jim followed a reasonable methodology based in science.

Yes, survivorship bias is alive and well and living in *parts* of GtG. (Reading “Outliers” now. Haven’t come to that section, but I’m looking forward to it.) FWIW, Berkshire Hathaway was excluded from GtG on what I consider fluke grounds (calendar date), and it might confirm Collin’s finding, not undercut it.

As I see it, the problem with looking for the groundbreaking companies (MS, Nokia, Apple) as models is that the ground is constantly breaking *everywhere* — but only exposing extraordinarily fertile soil in a few places that don’t announce themselves at the time. (Why didn’t videophones take off to dominate the world in 1955? or jet packs in 1965? or Segways in 2000?)

You might get more from Collin’s first book, BtL. At least in there you can see businesses that have been around for decades (my favorite, and Bill Hewlett’s(!): 3M) AND “took risks, saw market gaps, and went for it.”