It seems every time that DCIG releases a Buyer’s Guide (and DCIG has released two this month) a few detractors emerge, hide behind fake (and sometimes not so fake) identities in the ether and proclaim, “The results in this Buyer’s Guide are not valid! No testing was done on any of these products!”
While I typically do not know who these individuals are, their criticism on the surface seems to have merit. After all, testing is the ultimate barometer of a legitimate review, right? But before anyone dismisses a Buyer’s Guide because no testing was done on any of the products included in the Buyer’s Guide, consider that product testing has its own set of issues.
First, in case anyone is wondering if DCIG formally tests the products listed in its Buyer’s Guide, the simple answer is, “No, it does not.” The reasons behind that decision are both simple and practical.
The intent of each DCIG Buyer’s Guide is to provide a snapshot of products in the market place at a particular point and time for specific products. In that vein, DCIG has released numerous Buyer’s Guides on topics such as Midrange Arrays, Virtual Server Backup Software, Small Enterprise Storage Arrays and two more this month, one on Midrange Deduplication Appliances and another on Oracle Backup Software Solutions.
However in order to cover the products included in each of these Buyer’s Guides so the Buyer’s Guide is still relevant when it is released, there is no way DCIG can do any meaningful testing of all of these products. So whether it is the smallest of these Buyer’s Guides (the DCIG 2011 Oracle Backup Software Solutions Buyer’s Guide has only 12 products) or the largest (the DCIG 2010 Midrange Array Buyer’s Guide listed 71 products) the time, cost and effort it would take to properly test all of these products would be astronomical.
But even if DCIG did test all of these products and wrote up the results, who wants to pay to access out-of-date information much less take the time to read it? I mean, if DCIG really did test all 71 midrange arrays listed in the DCIG 2010 Midrange Array Buyer’s Guide, it would probably be right about now that DCIG would be finishing that testing and starting the write-up. In the meantime, nearly every midrange array vendor has released a new, updated model of its midrange array making whatever findings that were published interesting but largely irrelevant.
But let’s just for a moment assume that DCIG did test all of these products and that DCIG could get it done in a 3 – 6 month window and publish all of the results. Here are the challenges DCIG would face:
- Establishing performance standards. “Opinions are like performance standards. Everyone has one.” You and I both know that is not how the expression goes but ask any vendor if its system has the best performance and they will always swear it does. Ironically, that statement may even be true depending on what conditions its system is tested under.
But therein lays the rub. What conditions are the “right” conditions under which to test the performance of a system? And if the system is not tested under those conditions, are the performance results valid? But maybe more importantly, are the conditions under which the performance test conducted applicable to your environment? If not, the results of the test may not be applicable to you so what good is the performance test?
- Creating a standard testing environment. To say that today’s enterprise environments are complex is probably the understatement of the year. So how do you create a standardized testing environment that meets these criteria? And if you settle for something less than 10 different operating systems, dozens of applications, and hundreds of servers (virtual and physical) – have you really created a valid test environment?
- Getting all of the equipment in place. But let’s just assume for a moment in this magical test bed that I am creating that both performance standards and an application, server, network, and storage environment are agreed upon. We are all set, right? Not even close.
In the case of midrange arrays, DCIG would still have to get millions of dollars of equipment shipped to the testing facility, insured while DCIG is testing them and then shipped back. This is a huge logistical operation even assuming all of the vendors would agree to provide equipment at no cost. This does not even get into issues as to how the equipment should be configured and what to do if it is configured incorrectly. In the case of trying to configure and set up 71 midrange arrays and then run them though a battery of tests, I would safely say that it would take two years to complete if it ever got done at all.
- Setting up the equipment and running the tests correctly. But since we are already living in fantasy land, let’s make one final assumption. Let’s assume that we now have all of the equipment in place and it is ready to test. Now the next problem surfaces – has the equipment really been configured correctly and do I know how to operate it. Odds are, probably not. Or if I am testing the twelfth product and have learned how to do some tasks that I did not know how to do when I got the first unit in, does that mean I go back and re-test the first unit? I mean, the assumption always is that the tester knows what they are doing. But what if they don’t?
I hope you get the point. DCIG does not test the products in its Buyer’s Guides not because we don’t want to. Trust me, we know this would add further validity to these Buyer’s Guides and make them even more compelling than they already are. But running such tests is neither practical nor cost-effective nor are we even convinced that testing would significantly change our conclusions or recommendations.
So, yes, there are naysayers about the DCIG Buyer’s Guides and we appreciate the criticism as we are always looking to improve these Buyer’s Guide and make them more authoritative. But testing all of the products included in a Buyer’s Guide is not a logical next step as a means to improve them.
We are convinced based upon the tens of thousands of downloads that have occurred of these Buyer’s Guides and the overwhelmingly positive feedback that DCIG has received that these Buyer’s Guides are helping users find groups of products that meet their specific needs. We are then simply leaving up to individual users to develop a method of testing these products that is the most appropriate for their environment and in line with their specific requirements.