VCs have poured funds into Silicon Valley enterprise IT gear startups for years, knowing that data center technology is decades old and ripe for an overhaul.  As a former Intel executive, I watched these revolutions come and go.

However this time, with Big Data, it’s different.

Big Data has taken a sledgehammer to budgets and performance expectations, and these old architectures just can’t cope. IDC estimates that the total digital data produced in 2010 reached one zettabyte, and continues to more than double every two years.  Over 90 percent of this information will be unstructured — videos, MP3 files, and social media applications, which is rich information, but challenging to store reliably.

The situation will get worse, as:

  • Public cloud-hosted data now constitutes between 10 and 20 percent of all annually generated digital content and enterprise users will expect the same reliability, reporting and compliance support they have come to know from on-premises data centers.
  • New disk drive density technologies such as HAMR (Heat Assisted Magnetic Recording) will soon push disk drive capacities WAY beyond what we have today. The day of the 60TB hard drive is on the horizon .
  • Managing data centers at this level of scale will be intractable — for example, rebuilding a corrupted 1PB file system. Since rebuilding a single 2TB drive can require a day or more with traditional RAID technologies, rebuilding a PB file system could easily require weeks to months of time.

CIOs know this. That’s why they’re bypassing their reticence about buying from the little guys, who were traditionally viewed as too risky, and making choices that were unheard of a year ago and frankly very exciting. They know that without the dramatically different paradigms from these startups, data centers simply cost too much — and block their organizations from staying competitive.

In the data storage sector, for example, movie studios, oil and gas ventures, satellite images, video security and life sciences organizations — organizations with a preponderance of Big Unstructured Data — are following the lead of Google, Facebook, Dropbox, and Amazon S3 and adopting object storage. Object storage systems eliminate one of the fundamental components of storage systems to date — the file system — represent a genuine paradigm shift that is built for cloud deployment and, if done correctly, can scale nearly limitlessly at low cost.

Where is this headed? In the coming 18-36 months a number of key technologies will replace older infrastructure in in enterprise data centers:

  • Flash media will permeate even enterprise-class IT gear as it drops in cost. Transactional data will move onto flash drives, and away from fast/expensive hard drives.
  • Big unstructured data and archives of transactional data will move to newer paradigms, because the cost is so compelling.
  • Infrastructure, code and data will become ever more intertwined, with less need for the physical and far more of the virtual.

Certainly traditional glass house data centers won’t go away in the Big Data ecosystem, whether systems reside in an on-premise design or at an outsourced service provider. But Silicon Valley startups that can show CIOs how adopting these new data center technologies will allow them to out-innovate their competitors will win.

[Illustration by Hallie Bateman]