It’s shocking, I know, but the storage industry has more juice than Ryan Braun leaving a Biogenesis clinic.

Approximately two years after Fusion-io’s IPO, yet another in a lengthening list of flash storage start-ups, Violin Memory, has also filed for an IPO. What on earth is going on?

The underappreciated technology ecosystem behind flash storage, the memory technology capable of turning a smartphone into a bottomless media repository, or a USB thumb drive into your digital life-on-a-stick is hurtling its diminutive self toward a gargantuan market opportunity, fueled by a massive, continuous infusion of venture capital and sparked by innovation rooted in physics and chemistry, the literal kind.

IDC predicts that we will be generating 40 zettabytes annually by 2020. (Trust me, that’s big. A single zettabyte is 1,000,000,000,000,000,000,000 bytes, or 1 billion terabytes.) Sensors, surveillance video, photos, medical devices, financial market data, social and location data, and other creators of structured and unstructured information are demanding more capacity and real-time performance, and they’re getting them.

Flash storage is racing quickly to meet the inevitable need. Flash chips will get smaller, store more data more quickly and for less money, and just as the economics and physics hit a wall newer methods will be ready for commercialization. At least, that’s what’s happened throughout history.

What’s more, flash technology has already started to invade the data center, either augmenting disk, or supplanting it, perhaps completely some day soon. The technology’s small footprint is increasingly vital in today’s jam-packed data centers, especially the ones building so-called hyperscale infrastructures.

VCs are raining money on enterprise flash like Drake in a strip club. Dell Ventures put $51.6 million into Skyera earlier this year, Samsung Ventures poured $31 million of Series C financing into SolidFire in July, Meritech Capital Partners closed a $35 million C round for Tegile Systems just this month, and In-Q-Tel, which seeks technology investment opportunities for the U.S. intelligence community, funded Pure Storage in May.

And that just scratches the surface.

Flash is a non-volatile stored charge memory technology. DRAM, which has been the cache in front of disk storage for the past 20 years or so, is extremely fast, but volatile, thus better as a means of temporary storage. Flash technology fits somewhere in between: it’s not quite as fast, but the capacity is higher and volatility is lower, making it the natural choice for more permanently storing important data that needs to be accessed constantly and quickly.

Today’s flash memory technology, known as NAND (stands for Negated AND or NOT AND, a boolean operator that provides the logic for storing data), will reach its evolutionary peak in the next few years, when flash chip lithographies get too small to be reliable. That inevitability has sparked schemes to extend the life of NAND, to evolve its architecture, and to replace it completely with other potentially more inventive storage schemes.

Take Skyera, whose CEO and co-founder, Radoslav Danilla, started SandForce, a flash controller company, and before that he was a chip and graphic processor architect at NVIDIA. Skyera’s just-announced skyEagle is a small form-factor (1u) storage appliance that holds a whopping 500 TB flash, delivered at $1.99/GB, which falls closer to the $1.50/GB of traditional disk technology than the $15 – $20/GB of most enterprise-grade flash systems. When Skyera ships skyEagle next year it will provide 20x higher density, at least 6x lower cost, and 5x higher performance than any other manufacturer, Danilla boasts. It is, to be sure, impressive.

Skyera accomplishes this by using consumer-grade flash, which has significantly less reliability (by a factor of 10) than most enterprise-grade flash, but makes up for it with a variety of inventive workarounds. One example: Skyera’s system uses voltage shaping to reduce the voltage applied to the chips, thereby reducing damage and extending the life, and thus reliability of the technology.

Companies like Samsung, one of a small handful of NAND chip manufacturers, are already at work on 3D V-NAND technology, a scheme to stack more NAND cells vertically (the V in V-NAND) to increase capacity. Samsung’s chips also uses a different design to capture and store a charge, and promises better performance and a 40 percent power consumption improvement.

Others are moving in different directions, like using phase change memory, which lowers and raises resistance through crystalization, and spin transfer torque techniques known as Magnetic RAM (MRAM) or resistive RAM (ReRAM). Some of those technologies are starting to see the light of day, says Howard Marks, founder and chief scientist at analyst firm DeepStorage. The first PCM and ReRAM chips are less dense than the current generation of flash, so they are going toward special purpose applications.

Crossbar is one of the more interesting companies betting on ReRAM. The company’s CEO George Minassian says one of ReRAM’s benefits is simplicity, because most chip fabs have the readily-available material to make them, and the architecture promises more upside, including greater reliability at smaller chip lithographies. Whereas traditional flash memory uses electrons to trap a charge, ReRAM uses metal ions. In the case of Crossbar, that metal is actually silver.

Howard Marks says that enterprise flash deployment options — in an external storage array, or in the server, via the PCIe bus — are still being debated. A host of vendors sell hybrid storage arrays, with a mix of flash and disk, and a growing number have begun selling all-flash arrays, especially in environments where the latency of fetching data from a disk drive will impact customers.

Fast-growing Tegile Systems makes hybrid storage arrays using Western Digital’s Hitchachi Global Data Systems drives and enterprise-grade flash, and off-the-shelf Intel-based controllers. In other words, there’s nothing very fancy going on here. The key element is the ability to integrate the different media (disk and flash) in order to optimize for performance and cost. Tegile marketing VP Rob Commins says that his company’s arrays can be configured with 5 percent flash storage, 10 percent or 20 percent, and the company also offers an all-flash array.

On the PCIe side, Fusion-io has been one of the industry’s most visible companies. It went public in June 2011, landed customers like Apple, Facebook, and HP, boasts Apple co-founder Steve Wozniak as its chief scientist, and made key acquisitions like NexGen Storage and IO Turbine. But Fusion-io has been in the news lately because the company’s revenue growth has stalled, which fueled a shake-up in top management, namely the ouster of founders Rick White and David Flynn, who are now starting up Primary Data, a company that seems aimed at the opportunity to virtualize these pools of storage. Shane Robison, formerly HP’s chief strategy officer, is at the helm of Fusion-io now.

For both Tegile and Fusion-io, much of the magic lies in software, where data movement among disk and flash is managed based on the data’s value. Take virtual desktops, which commonly boot up at approximately the same time each morning, accessing the same boot images from a server — a problem flash systems are well positioned to tackle. Tegile and Fusion-io both announced new technology at VMworld this week to optimize such VMware-oriented environments.

For example, Fusion-io’s ioVDI includes a feature called write vectoring, which takes some of the temporary data created by virtual desktops and puts it on the server’s flash memory, reducing workload on traditional back-end storage systems, according to Gary Orenstein, Fusion-io’s senior vice president of products. The Fusion-io software also accommodate environments that use VMware’s VMotion, which lets IT managers move workloads without disruption. ioVDI can also dynamically allocate flash, essentially allowing organizations to virtualize pools of flash memory across servers.

There’s also new promise beyond the traditional array and PCIe methods of flash deployment. SMART Storage Systems, in partnership with Diablo Technologies, announced that it would put flash on the memory bus, where applications could get the benefits of flash (non volatile, but still very fast and high capacity), but with the access latency of main memory DRAM.

Applications like SAP’s HANA, which uses DRAM for its in-memory database architecture (providing near real-time access to complex data), could be a fit for flash on the memory bus, essentially ushering in more cost-effective implementations of HANA. Marks says that blade servers, where space is at a premium, would also be a logical target for this technology.

The next step, Marks says, would be “to modify applications like HANA and underlying operating systems and hypervisors to recognize that all memory is not the same.” That is, to ensure that more important data gets stored on more expensive, faster technology, like flash.

Skyera’s Danilla dismisses flash in the server memory channel, saying that while it sounds sexy and will definitely reduce data transfer time to the CPU, that is only a minor performance bottleneck.

And what better barometer of an industry’s ascent is there than when competitors get into a cat fight?

[Image Credit: alikins on Flickr]