Flash is Disrupting the Benchmark Game too

reindeer raceA couple days ago, Lou Lydiksen from Pure Storage posted a blog titled “What’s wrong with using 100% non-reducible data?” The post was better than the title suggests because it exposes a problem storage buyers have today, which is validating the performance of flash storage systems using benchmarks developed for disk-based systems.

Independent storage performance TALK-NERDY-TO-MEbenchmarks have been nerdy infotainment for many years due to the work that often goes into getting the best possible results. Storage systems are usually configured and tuned a certain way to get the best results. (An example of a benchmark that appears to break this rule is the recent SPC benchmark from Kaminario). There is nothing wrong with optimizing application performance, even if that application is a benchmark, but it obviously makes sense to use benchmarks that model realistic, production workloads.

New architecturesThe larger question is what happens when new technologies with new architectures come to market?  For example, the SPC1 benchmark for transaction processing accommodates flash storage technology, but it does not allow the use of compression and deduplication. That’s a big deal because both are important cost-saving features of enterprise flash systems that most customers want but have no way to predict if, or how, they will impact performance. The lack of a feature-comprehensive benchmark does not appear to be slowing the growth of flash storage systems, but that is beside the point – customers would benefit. Other realistic tests would be running the benchmark in a virtual environment alongside simulated background processes and pinning the benchmark application in flash while background processes run in disk or some combination of disk and+flash.

The SPC has indicated that they are sorting some of this out and are developing a way to include compression (first) and deduplication (later) rainbow in skittlesin their benchmarks. I don’t know where they are with respect to virtualization and hybrid designs but in my opinion, storage benchmarking is suddenly a green-field opportunity again – brought about by the rise of enterprise flash storage systems. Will we see enterprising analyst/entrepreneurs rise up to grab the brass ring and establish clear thought leadership amidst the chaos? Are there any takers?