Businomics Blog

Firefox Gets the Trial and Error Economy

The folks at Mozilla who put out the Firefox browser (which is great) understand the Trial and Error economy.  They recently wrote up a description of an experiment they did in marketing their download web site. (Hat tip to Steve Levitt at Freakonomics.)

Here's the test: if you go to a search engine to download the Firefox browser, you get the official website as the number one result from most of the popular search engines.  So should they pay Google or Yahoo for a pay-per-click ad on those search results when they are already number one?  It turns out that they get a little more traffic when they do.

One lesson from this, though, is that the majority of the traffic that comes from pay-per-click advertising would have come anyway; a majority but not all of the traffic.  So your cost per customer is not really what you are paying per click; it's your total costs of the clicks divided by the customers who would not have come to you without the pay per click.

In one experiment, it appears that about seven out of every ten people who clicked the paid advertisement would have clicked the regular search result (called "organic search") if the paid ad had not been present.  So all of your click payments only brought you those three out of ten who otherwise would not have come to you.

It takes a lot of work to get a good experiment, and you often have to keep running more experiments to get the results fine tuned.  But you can learn far more, and boost your profit tremendously, by regularly experimenting in your business.

eBay's Culture of Analytics: We All Need to Get on Board

Fast Company's latest article about eBay describes how the new chief technology officer created a "culture of analytics."  The old approach had been to set up a new feature or interface and see if it works.  If not, roll it back, returning to the old way.

The new approach was started by Matt Carey, the CTO.  eBay would show a new feature to a sample of users, about one to two percent of the total users.  They didn't know they were guinea pigs.  But their behavior on the site was tracked, in terms of how long they stayed on the site, how many pages they visited, how much business they did, etc.  Now one of the top developers says: "In a Darwinian sense, to be a survivor something has to keep producing."

The remarkable thing about this story is . . . that it's remarkable.  This should be cut and dried, basic business these days.  The data needed to evaluate an experiment at an on-line business are all available, if you just set up the systems to collect and analyze it.  Let's get on with it, guys.

(By the way, I've been watching what you click on to get to my page, and I've reacted.  Traffic is up.  Thanks.)

Business Goals vs. Strategic Intuition

Read the first two paragraphs of this Wall Street Journal book review:

Set big goals. Do whatever it takes to reach them. These muscular sentences form the core of commencement addresses, business-advice books, political movements and even the United Nations approach to global poverty. In "Strategic Intuition," a concise and entertaining treatise on human achievement, William Duggan says that such pronouncements are not only banal but wrong.

Mr. Duggan, who teaches strategy at Columbia Business School, argues that the commonplace formula has it backward. Instead of setting goals first, he says, it is better to watch for opportunities with large payoffs at low costs and only then set your goals. That is what innovators throughout history have done, as Mr. Duggan shows in a deliriously fast-paced tour of history.

(The full review is available here; subscription required.)

This message strikes home in two ways.  First, I've been blogging about the trial and error economy, in which companies adjust their actions as they get feedback on what's working and what's not.  That seems to be the essence of Duggan's book, which I have not yet read. (A couple of my key posts about the Trial and Error economy are here and here.

The second way this strikes home is that I've been working on a new business venture.  I'll post more about it here as it gets started, but it's a web-based information service.  As we talk to angel investors, they want to see our plans and goals.  Fine, I understand that.  But I have a forecast that three years into the project, at least half of our content will be driven by customer reaction rather than our own plans.

You could well ask if we should be starting a business if we don't know what to provide our customers.  But as we look around, hardly anybody is really doing a good job at monitoring customers to see what they want.  Here's an example:  suppose a supermarket manager noticed that customers were spending a lot of time in one aisle looking for something, and that half of those customers left the aisle without having placed anything in their shopping carts.  A smart manager would figure out that they are looking for something that isn't there.  Even the ones who buy something from that aisle may have been settling for second best; their time in the aisle indicates that they were looking for something rather than just grabbing their old-time favorite.

It's hard for the old-time supermarket manager to spot this behavior, identify the specific section of the aisle where this is happening, and bring in additional offerings to tempt the shoppers.  But it's pretty easy to do that on the web.  The biggest surprise is how little it's being done right now.

For the business leader or investor, I recommend: think less about plans and goals, and think more about how to identify opportunities and quickly exploit them.

Gum Marketing We Can All Chew On

Ben Pratt over at Market-Based Management Institute has a great post on his blog about selling chewing gum in Poland. Here's an excerpt:

He said that several years ago the Polish division of Wrigley ... had somewhere on the order of 70% market share of chewing gum sales in that country, almost none of it was the sugar-free variety.

Part of their sales and operations planning process was to document assumptions and revisit them at each planning cycle.  One of the assumptions they documented was that as people grew older their preference would shift from sugar-based products to sugar-free products, and demographics suggested that the population was aging.  At some point in the future they decided they'd have to change their mix of sugar / sugar-free products, but that time would be many years in the future.

Charles Koch's Science of Success quotes George Will as saying, "the future has a way of showing up unannounced."  And it seems to show up a lot faster than we expect.

Wrigley began seeing small but meaningful deviations to their expected sugar-free gum sales numbers.  Because they were revisiting their assumptions each month, they got curious and began testing whether the sugar-to-sugar-free preference shift was really as far out as it they'd assumed.  The result of their analysis caused them to recognize an opportunity: people in Poland were looking for sugar-free products, but not many companies were supplying them.

This information caused them to re-think their strategy and they immediately shifted their product mix almost entirely to sugar-free gum.  They captured 90%-plus of the market.

I'm guessing that without a formal sales forecast as a benchmark, the company might not have caught the shift in trend.  A minor product grows a little faster than expected; ho hum.  But the trial-and-error economy demands that you watch for the little things that are growing faster than expected.  Then be ready to nurture them into really big things.

How Quants Go Wrong

Here's a great quote from a recent Wall Street Journal article:

"Events that models only predicted would happen once in 10,000 years happened every day for three days."  Matthew Rothman, Ph.D., Lehman Brother Holdings

I hate to take pot shots at analytical guys--I'm one myself.  After all, you don't go to graduate school in economics just to pick up chicks.  (Well, that's one reason among many.)

Let's ask, how could they be so wrong?  Simple.  They were not remembering their assumptions.

Remember the normal curve (also called the bell shaped curve)?  It looks something like this (courtesy of Wikipedia):

Suppose we have a bunch of data, like on daily prices.  (More likely, on the change in the logarithm of the price.)  We can look at the data and see if it fits the normal curve.  But wait.  We don't have many observations out at the tails of the curve.  Like things that should occur once every 10,000 years.  We may not have any observations, or we may have one.  In any event, we can't really tell if the tails of the distribution fit the normal curve or not.  So we look at the great mass of data in the middle, say that it looks like it fits the normal curve, then assume that the tails of the normal curve also fit the data.

But they don't.  We've known for years that financial transactions are "leptokurtic," which is a great way of saying they have fat tails.  In other words, there are more extreme values than the normal curve would predict.  I assume that Dr. Rothman knew that, but what does one do with the knowledge?  Assume some other distribution, similar to the normal curve, but with fat tails.  Unfortunately, it's very hard to know just how fat the tails really are.  We can collect many, many years of data and still not know very much about the far tails of the distribution.

Great example:  In 1997, my local newspaper reported that the Tualatin River (near my home) had surpassed the hundred year flood mark, but "it wasn't as high as last year's flood."

What's the moral of this story?  We have to be very cautious about extreme events.  We don't really have a clue how frequent they are.  That's true of financial markets, but also true of physical sciences.  If an engineer tells you the boiler has one chance in 10,000 of blowing up, I'll bet that the engineer does not really know the true odds--he's just extrapolating on the assumption that the distribution is nice and neat.

What to do about it?  Don't place too much confidence in hedging strategies.  Instead, plan for bad events, and be sure that you know how to react when they happen.

Employee Performance: Measuring to Reward Success

Labor markets are tight.  It's important to retain your best employees.  But  who are the best employees?

That question is tough.  A recent article in Business 2.0 magazine helps employers get a handle on employee performance.  I liked the article from its opening line:

"Let's be honest:  Most performance reviews are a joke."

Couldn't have said it better myself, but then the article describes some online services that help bosses improve their measurement of employee performance.

Imagine this:  you allocated you bonus pool so that your best employees were very motivated to stay.  You wouldn't even have to get rid of the worst employees to see improved results overall.  Just retention of the top performers and normal attrition of everyone else will give your organization a shot in the arm.

In a tough, competitive economy, you need to measure performance as well as it can possible be measured.

Advice for Entrepreneurs (which Corporate Executives Can Use)

Guy Kawasaki moderated a panel at the Churchill Club in Silicon Valley:  No Plan, No Capital, No Model ... No Problem.  The program is well worth watching not only for start-up entrepreneurs, but also for seasoned corporate executives.  One key point is the need for speed and flexibility in start-ups, which is how start-ups clean the clocks of major companies.  One key takeaway showing their understanding of the Trial and Error Economy:  "Iteration is the key to success."

Incidentally, Kawasaki is a great example of a panel moderator; he knew good questions to ask, gently nudged them back on track when they got into too much detail, and kept the audience chuckling.

Jeffrey Pfeffer Understands the Trial and Error Economy

Jeffrey Pfeffer is the Stanford business school professor who write a monthly column for Business 2.0.  He has a new book out.  In conjunction with that Guy Kawasaki asked him 16 questions about management in a blog post entitled "10 Questions with Jeffry Pfeffer."  (Guy is pretty smart, but possibly challenged by counting past 10.)  Pfeffer's first answer shows that he understands the essence of the Trial and Error Economy:

Question: Why do companies do stupid things?

Answer: First, they ignore feedback effects.

On the subject of feedback, I have a short article about a cool example of feedback. (Originally published in The Freeman: Ideas on Liberty.

Auto Dealers Best Practices--Lessons for All Businesses

All business leaders should scan the horizon outside of their own industries for ideas to improve their business.  A recent McKinsey & Co. study, How to Build Top-Performing Auto Dealerships, offers insights that everyone can ponder.  McKinsey consultants found three key factors that differentiate top-performing dealership.  First, top dealerships had lower employee turnover.  (We've talked about this many times in the Businomics blog; review my 7 Steps to Better Employee Retention.)  This isn't just luck; the report cites several different management practices that improve retention.  Second, top dealerships apply various customer retention strategies, such as mailing out service reminders.  Third, top dealers are more likely to be affiliated with manufacturers that provide strong coaching of dealers.

So what if you're not a car dealer?  Here are some ideas to consider:  reduce employee turnover, improve customer retention, and work with vendors who can help you learn the best practices in your industry.

But don't just follow these practices by rote.  They may not be the top priorities in your industry.  But if one of these ideas strikes you a worthwhile, give it a trial.  Do a test of specific practices, and look at results with respect to specific goals.  That's the trial and error economy.

Does the CEO Hear the Truth?

I just started reading Competing on Analytics, because analysis of data is a huge competitive advantage in the Trial and Error Economy.

The preface, by Harrah's CEO Gary Loveman (who has a Ph.D. in economics!) has a telling point:

"... all organizations seek to please the leader, so there is constant pressure to give my otherwise lame and ill-considered view far more gravitas than they deserve."

First, let's praise Gary for recognizing the problem.  Many CEOs say that their people will speak up to them--but they only promote people who never speak up.

Second, let's help the business leader deal with this problem.  Scientists with a hypothesis are trained to ask, "How could this hypothesis be disproved?"  The CEO with an idea should ask what might disprove the idea.  Then the boss should ask for data.

As an example, I posted a while back about Starbucks CEO Howard Schultz's thoughts about his company's strategy.  This gist of his thoughts, apparently, were that Starbucks had to return to its roots regarding roasting beans on premises and letting customers see the baristas work.  The danger to Starbucks is that the vice presidents begin a "back to our roots" campaign without offering the CEO any pushback.  Schultz could ask, "How would I know if I'm wrong?"  Simple answer: we test the approach on a random sample of stores and see how they do relative to a control group.  Then a staff member doesn't have to say, "Howard, you're wrong."  The staffer simply says, "Here are the results from the test that you requested."

A third approach surprised me a while back.  A corporate CEO asked me to come up with some ideas for implementing a particular strategy.  After nosing around the company, I suggested that this strategy was not a good idea.  The CEO asked me to present my findings to the board.  Turns out the CEO wanted me to validate his own disbelief in the strategy; so he gave me the assignment just to see if I would push back.  The use of consultants isn't perfect here--we have an incentive to tell the client what he wants to hear, just as employees do.  However, if we take the risk and lose, we've lost one client, worth maybe 5% to 20% of annual revenue.  If an employee loses a job, that's 100% of revenue.

The important takeaway message for CEOs:  give some thought to how you can get employees to tell you when you're wrong.