The New War Between People and Machines

By Sarah Lacy , written on September 10, 2012

From The News Desk

So far in the evolution of the Web we've seen a tug of war between whether we need people to find and accomplish what we want online, or whether we only need machines.

In the early days of directories, people were prized. Then came Google and with it a slavish love of algorithms. With the dawn of social networks, the benefits of humans came roaring back into vogue but with a new twist: Let's not pay them! Rather than having a staff of people to find or create awesome stuff for you, Digg, YouTube, Wikipedia, Yelp and other Web 2.0 pioneers would get the users to do all the grunt work and reward them with special titles and badges and the lofty group title of "community". Even Google's search results-- the ultimate bastion of the algorithm-- are no longer sacred in a social media era.

There has been a somewhat parallel trend when it comes to ecommerce. The first wave were essentially catalog businesses on the Web, which still needed a lot of humans to help. Then there were the ambitious Webvans of the world with outrageously mechanized warehouses. "It's like the Jetsons in here!" a Webvan executive told me back in the day. Sadly, the Jetsons was no more practical in 1999 than it looked on TV. The futuristic automated Webvan warehouses weren't remotely sustainable and worked so poorly that drivers sometimes had to stop at Safeway and buy groceries to make a delivery. Tony Hsieh and Zappos brought back the value of people with his Vegas-based call centers, staffed with people who will spend all day on the phone with you if you like.

Oddly enough the tug of war is now pulling in two directions. But both have the same cause. Both are a reaction to the gargantuan size of the Web and the massive amounts of comments, Tweets, Likes, photos, articles and things for sale that are flying through cyber space everyday, all clamoring for your attention or cash.

There's a desperate grappling to make sense of it all, tailor it all, and serve up the thing most likely to result in revenue. Rather than the industry at once turning to either machines or people as the answer, the industry is pulling in two directions at the same time. Interestingly enough, both approaches may be overhyped, and both may still be incomplete without the other.

On one hand you have a movement towards curation, which Erin wrote an in-depth post about this morning. Sites like Jetsetter, Fab, NastyGal and many smaller ones Erin mentioned, have found that no computer program can determine quality nor can the user generated content masses, so why try? What the Web misses most today is the human touch, these sites argue. This trend is heaviest in ecommerce -- witness the curated everything in a box/celebrity stylist trend.

Part of this is a reaction to horrendous aggregators like -- the machines' attempt to make sense of a limitless long tail of online inventory. Some of those comparison shopping engines had decent exits, somehow. But I never once found anything I was looking for using them. The only exception were travel search engines like Kayak, because flights are essentially interchangeable commodity listings. There's no judgement call there. Finding, say, a great black dress on, however, was a laughable exercise.

It's not a surprise that a lot of the leaders when it comes to curation are entrepreneurs from the New York and Los Angeles ecosystems. From Hollywood to old media, these are economies made up of people who prize gatekeepers, editorial, and professionals doing professional work. The power of the smart people.

In the Valley, meanwhile, the buzzwords du jour is "Big Data". This is the flip-side of making sense of the long tail of everything online -- just build more powerful systems to comb through it all. The inspiration here is frequently Amazon's killer recommendation engines. But for every Amazon, there are plenty of examples of automated recommendation engines not working. Like, say, Netflix recommending "White Chicks" after I'd just watched "Iron Man."

Nearly anything raising money in the Valley now is using big data as its opener. But while companies as diverse as Uber and Groupon certainly have a lot of data they can theoretically use to make their systems better, actually doing that is a massive challenge. (One that Uber excels at; Groupon...not so much.) It's the power of the machines.

Ironically just as Erin was publishing her post on curation today, the New York Times published an ode to big data. (Although the Times post utterly missed that not all of the Web is pulling in this direction.)

Most of the big stories in the tech world today tie back to one of these two opposing trends -- even all the recent pressure on PayPal. Forget PayPal's fees, which people have bitched about for years. The biggest gripe that forces people to actually switch? PayPal's automated fraud detection that keeps cutting off organizations' funds at crucial times. Not surprisingly an organization like Braintree has made inroads trumpeting a reliance on real people, including having real people to pick up the phone in case of a glitch.

A handful of companies are using both strategies, for example, Quora. The question and answer site has avoided going the way of Yahoo Answers or Formspring, because of its community guidelines and high quality answers. The quality of answers is determined by a mix of technology and curation.

But at their hearts most companies hew to one approach or another. Consider even our PandoTicker. We have found that editorial employees do a quicker and more comprehensive job of curating a day's news relevant to startups than any machine or algorithm could. But it helps that we're trying to do something very specific that has inherent limitations. We are not trying to aggregate millions of posts per day. The entire point is condensing what's out there for time-pressed entrepreneurs and investors. Where curation breaks down, as Erin wrote, is when it tries to scale. Frequently some help from machines is the answer.

The question is whether the reverse will prove true, whether big data companies find they still need human judgement. My hunch is that we'll see this less. People who value humans still generally value machines for what they are good at (unless they are Amish). But in my experience, people who slavishly see algorithms as the answer always think you just need a better algorithm. Witness: Klout's attempts to put your "influence" into a crunchable number. The recent relaunch changed a lot of what critics hated about Klout, like the gap between you standing on Twitter and actual real world influence. But it didn't add human judgement into the equation, it just added in things like a LinkedIn profile or a Wikipedia page to the algorithm.

What's interesting about the two opposing trends playing out at once is what it says about how Web innovation has evolved and substantially broadened. It's not that we're any less lemmings than we've been in previous waves -- witness the Pinterest-ification of the Web for strong evidence to the contrary. It's just that the game has gotten so big with such a diverse set of players. You don't have to be remotely technical to build what passes for a "tech" company anymore. Likewise, there's no longer one main hub of innovation -- that means we have a few significant tribes of lemmings, rather than one herd.

And this is ultimately better for the audience. There are a billion people online now seeking to do everything from shop to entertain themselves to find out information and other things that Hamish writes about late on a Friday night. The idea that machines can help all of them is as absurd as the idea that there's a human being sitting somewhere tailoring an online experience for each of them.

Hopefully the fact that we have the two trends pulling at once will yield to a more common sense approach: That any modern Web company really needs both.

{Image courtesy x-ray delta one]