Lies, damn lies and benchmarks

The always-excellent CMS Watch has an interesting article on benchmarking portals.

The author, Janus Boye, makes the valid point that benchmarking has to be seen in the context of what the portal is intended to do. This might seem an obvious point but it underlines how slippery even empirical data can be in the wrong hands.

We’re all familiar with the phrase about lies, damn lies and statistics (although nobody is quite sure who coined it) but I’ve always preferred the less popular one about those who use statistics the way a a drunkard uses a lamp post -– more for support than illumination.

In many fields of computing, benchmarking is broken. In server microprocessors, companies like Intel, AMD, IBM and Sun can make merry with numbers but they don’t mean much, at least not without a ton of accompanying explanation. The fault is not so much with the design of the simulations themselves or their accuracy, but with the multiple, and often mutually contradictory, nature of the metrics being applied. For example, in servers, some buyers will want raw performance, some energy efficiency, and an increasing number will want the perfect virtualisation host.

Too often, benchmarks are cited to impress even though the numbers are meaningless. As my old history teacher used to say, if you can’t blind them with science, baffle them with bullshit. However, used properly, benchmarks offer short cut to finding valid, independent insights.

When benchmarking an ECM, CMS or other system, the temptation will always be to impress the person who is asking. If the FD wants to see ROI, the person who bought the system will try to find a way to show ROI.

One thing that would help buyers is a little help from the vendors. Assistance in measurement tools based on specific criteria would help buyers offer up proof points for not just ROI but usability, service improvements and other factors.

So far, that assistance has been limited. I wonder if some vendors are a little coy of providing tools that might produce answers they don’t like.