The Socha-Gelbmann Electronic Discovery survey will continue but its rankings are gone. The announcement that was recently posted on Socha Consulting’s blog that it is killing its ranking that made up a component of its annual survey since its inception caused a small stir among the vendors covered in the report. However apparently very few vendors shed any tears over Socha-Gelbmann’s decision to drop these rankings from its annual report, even among those it benefited.
Vendors do not dispute that a need for such rankings by a third party analyst firm exists. Yet how the Socha-Gelbmann survey arrived at the numbers in its rankings and the conclusions it drew from them were coming under increasing criticism by the vendors and end-users it was supposed to serve. Even vendors like Kazeon Systems, which for the most part received favorable ratings, felt that a change was in order.
The crux of the criticism about the rankings in the Socha-Gelbmann report stems from two principle areas. The first is the types of the companies covered in this report. The rankings included data from over 150 organizations. Of these, 107 of these organizations were third party service providers and another 29 were law firms. It was the minority of end-user corporations covered in the report (only 19) that was causing some heartburn among vendors.
The problem with polling and relying so heavily upon data provided by service providers and law firms is that they need different features in an electronic discovery and/or information management software tool than what end-user corporations may need. Part of this has to do with the business model of service providers and law firms. They only focus upon a few processes within the broader Electronic Discovery Reference Model (EDRM) such as Processing, Review and Analysis. So it only makes sense they will use software tools that specifically focus on these processes which will result in more favorable rankings of this software but ends up skewing the rankings overall.
Conversely end-user companies may need a software product that covers the entire spectrum of EDRM, not just a subset of it. Information Management, Preservation, Production, Processing and Presentation processes may all carry more weight among end-user corporations since they use the tool as part of their broader, ongoing corporate data management practices. Yet if only 19 end-user corporations are included in the survey, tools that are specifically designed to serve them are not as likely to come out as well in the rankings as the tools preferred by service providers and law firms who use the software in a very different manner.
The other criticism of Socha-Gelbmann’s rankings is less about the quality of the report and more around the growth of the industry. When electronic discovery and information management was still a nascent industry (such as in 2003 when Socha-Gelbmann put out its first report), two analysts could satisfactorily analyze the vendors and products in the space and generate a thorough report. Now conservative estimates put the number of products at over 100 with tens of thousands of companies of all types using this software. In this respect the industry has outgrown the size of the analyst firm trying to cover it.
Objective rankings of vendors are something users need as much as vendors so users can make appropriate technology choices and vendors know in which areas they need to improve. But when the sources of the data used in a report start to skew the outcome of the report, it is time to call into question both the report and those producing the report. To Socha-Gelbmann’s credit, it recognized some of these factors and is re-analyzing the best way to produce their annual report going forward. In a forthcoming blog entry, I’ll share some thoughts as to how Socha-Gelbmann may want to refocus some of the report going forward to more appropriately reflect these different market segments.