photo by Michael Miller

HPC Storage Scheme Targets Wall Street

by Alan R. Earls

May 15, 2013 - More data combined with higher processing speeds calls for special measures, and not only in supercomputing.

It's hardly news that Wall Street traders value speed, that they go to great extremes and great expense to shave microseconds off the time required to complete transactions. But traders also work to grind through masses of historical data in what amount to large-scale, "what-if" simulations with requirements, data-wise, that are much like those found in the scientific world.

And so, Wall Street companies are looking for help in the high-performance computing (HPC) marketplace, traditionally focused on gigantic simulation problems where results mattered more than absolute speed. Many simulations take hours, or even days, to finish.

One company scrambling to take advantage of this trend is Terascala, a Boston-based maker of acceleration products that have found good uptake with the HPC crowd. This company says that the typical network attached storage (NAS) infrastructure cannot deliver the IO throughput required for scientific and engineering tasks, particularly when those tasks involve extremely large data sets. Cycles get spent moving data in and out of memory instead of processing it.

To fix that, Terascala has come up with what amounts to a high-speed bridge, leveraging standard commodity storage with specialized software to drive data transfer rates to as high as 20GBps. The company's storage appliance integrates with storage controllers from companies such as Dell and EMC, using parallel processing and object storage techniques to boost IO performance. At the heart of the system is the Lustre file system, originally developed by Sun Microsystems, which harnesses distributed systems working in parallel.

Lately, Terascale has been selling its storage system not only to those working in science and engineering but also to financial services companies working to speed the execution of market trading transactions.

Enterprise Strategy Group analyst Steve Duplessie profiled one of the company's financial sector customers, called Tradeworx, which operates an equity market neutral hedge fund and a high-frequency proprietary trading business. (Tradeworx also licenses its trading platform through an affiliate, Thesys Technologies.) Tradeworx was having trouble scaling its storage environment to make the most of its multi-year trove of market data and ended up selecting Terascala's technology.

The results were that the company moved 40TB of data to a new NetApp storage system, boosted by TeraScala, and claims it can now sustain 5GBps throughput.

Sweet spot In a recent blogpost, Kevin Murphy at Terascala explained that his company's "sweet spot" is where "Big Compute meets Big Data and the goal of the output is simulation, analysis, or modeling." He says that traditionally, scale-out NAS solutions have been used to address these challenges, but they don't deliver the IO throughput needed for ever-larger data sets and performance demands. It is, in fact, a bottleneck.

While the technology provided by Terascale, and the problems that Tradeworx cares about, may not be mainstream, the trends in both cases are merely magnifications of what all large enterprises are facing in varying degrees. Data volumes are rising and performance requirements are growing more extreme.

Thus, the example of Thesys and Tradeworx may be a harbinger of things to come -- fast processors (Intel Xeon); faster networks; a fast, parallel processing system -- and, on the horizon, even more use of solid state drives (SSDs).

Wall Street is, after all, home to a high-tech arms race. And you can bet someone, somewhere, is praying for a physics breakthrough that can boost the speed of light, too.