Why I'm just not buying the obsession with One Key Metric

By Sarah Lacy , written on December 27, 2012

From The News Desk

Since the Web has existed, three things have happened over and over again: Industry king-makers and wannabes have obsessed about a metric. The obsession gets pushed too far as the metric is gamed. And then the industry rejects the metric.

Hits, page views, uniques, time spent, Diggs, Likes, RTs, comments, downloads -- there are plenty of examples. They can all be gamed in varying ways. They can be bought and goosed.

But many remain industry standards despite the criticism that inflation has ruined them, because it's what investors and advertisers reward, as much as they may claim to hate them. It's fashionable to decry hollow vanity metrics, but my guess is most startups still lead with them because whoever they're trying to get money (or press) from at that moment still wants something tangible they can compare to other things they're seeing.

The new sexiness in the Valley is to have One Key Metric. There are good things about this movement. It stresses things like engagement and cohort analysis and improvement over time, not absolute numbers. But the problem with vanity numbers wasn't just what we were measuring. It was a false sense of security that pushing up any single metric would magically result in a great business. It's gaming a metric as the goal in and of itself, not the result of building a great company. As long as startups are run by people with a hacker mentality any slavish focus on any metric will lead to abuse and distraction.

The advantage is supposedly focus: One thing you're driving at, a number you'd bet your business on. We're driving at quality and consistency. When we do that, all kinds of numbers jump. But there is no single number that directly correlates to it.

Why can't creating a great product be enough of a North Star? Why in our boil-people-down-to-a-Klout-score world does that have to be a single number? Even the king of OKM, Mixpanel's Suhail Doshi struggled to come up with one that worked for PandoDaily when Richard Nieva wrote about them a week ago. He offered "comments," but look at your average YouTube or Yahoo Finance thread and tell me that's an indication of a great business. Comments matter, indeed. But it's not a number of comments. It's the quality of them. Again: Something that cannot be boiled down to trackable stat.

Maybe we're alone as a weird, content-oriented startup operating amid a tech world. Maybe this isn't a common problem other entrepreneurs face. But I doubt it.

I find it ironic that VCs refuse to give OKM by which they would govern their own investments. Over the last few months as I've had broad conversations about what would help a startup avoid being "crunched" as it seeks to raise a Series A or B, almost no one has given a single answer. Instead they always cite the always vague "traction." When pressed Mike Maples of Floodgate simply said he'd know it when he sees it. I'm not picking on Maples. I absolutely agree that's how good VCs should invest.

The absolute metric the matters for VCs are their returns to investors. But few good VCs would tell you that he or she makes an investment in a seed round with that eventual outcome solidly in mind. Returns are a result of a job well done, not the north star that drives day-to-day decision making.

So why do they expect something so different out of their portfolio companies? In my case, I know a good article when I see it too.

Even in an industry as obsessed with metrics as baseball, stats were a red herring. Moneyball was about a blind adherence to the wrong stats, leading to wildly over valued and under valued players. And any baseball manager -- or even fan -- will tell you that outs and wins and losses aren't all even. A series of games that a struggling team loses, may still portend that their luck is turning around if the scores are narrowing or the loss was the result of something like an umpire's bad call. That's why scouts are sent to games.

I first got in a debate about this with Twitter's CEO Dick Costolo. I insisted my company didn't have One Key Metric. He insisted I had to have one. It went on for a while, but we left it unresolved.

Part of my reluctance is seeing what did work in old media -- where such page-viewed and time spent metrics just weren't available to obsess over -- and part of it is my experience in new media where they were.

Page views are clearly bad single barometer for any news organization that wants to prioritize quality. Uniques are important to watch, but if your goal is to reach an influential audience deeply, it's not really something that will tell you how you are performing.

At TechCrunch, staffers were given bonuses based on Techmeme headlines instead. The intention as a OKM was great and incredibly forward thinking for an early blog. It was meant as a barometer for TechCrunch's ability to break stories, because that was the focus. In theory whoever broke a story would get the headline. There wasn't necessarily a better way to judge this.

But like any metric, it can be gamed. In reality, it became a silly game of trying to rewrite a press release five minutes faster with a more direct lead and a brain-dead basic news headline, because those were the things the Techmeme "algorithm" valued and the easiest way to consistently get the most headlines. Post-AOL acquisition that was thrown out the window in favor of page views. Not a big surprise, as AOL can easily game those by flooding dumb traffic from other verticals.

Clearly, in this case the OKM, while flawed, was way better than a lame vanity metric a big public company could goose, understand, and sell like page views. But that flaw is still important to recognize. Writers, like hackers, are wiley. Give them a way to check off that they've succeeded today, and they'll do it at the expense of absolute quality all the time, particularly as your organization grows. It doesn't matter what it is.

There's something deep in our psychology about the fear of the unknown and in the context of entrepreneurship, it's particularly acute. People paychecks, investors' money, your reputation are all riding on things you cannot control. There's comfort in a number you can track minute-by-minute to tell you things are going well. That the sacrifice and gut-level decisions you made were warranted and right.

But life and business is messy. Economies and financial stocks have always traded based on perception, rumor, narrative, buzz and lots of intangibles. They're annoying and hard to quantify. But that doesn't mean they don't exist or matter.

Whenever a CEO relies too heavily on a dashboard or a number, I'm reminded of Tom Siebel bragging that his software could see around corners, just before his business shocked Wall Street by falling off a cliff. Sometimes the best way to see around corners is to go outside and look around the corner.

Likewise, the best way to know how your business is doing is look at a variety of metrics with one eye, while looking at real world factors too. Talking to customers, 1:1s with employees, surveys, revenues, measuring social media engagement and sentiment. Listening to your gut as a CEO. The more we drive towards One Key Metric, the more I'm convinced we need more data points than ever, and most of those don't come from an analytics panel.

[Image courtesy musicisentropy]