The Simpson's Paradox -- CMS Matrix Revisited

I've recently run up against Simpson's paradox at work and thought I'd look at some CMS stats through that lens. Its been awhile since I dug in at CMS Matrix, so let's begin there.

If you click on their internal link to the detailed statistics (thank you CMS Matrix), you'll be able to dig down into the specific ratings reported on each category from Security to Built-In Applications. With a little help of cutting, pasting, and rearranging in a simple database, one can generate a table such as the following for any CMS of the 517 or so that they have listed. (Is everyone building CMSs these days?!)

The count is just the number of responses that make up the rating. Remember, this is a self-selected, self-reporting rating system. Don't fall in love with the numbers.

Category	Drupal	Plone
Security	6.68	6.70
Ease of Use	6.20	6.73
Management	6.49	6.62
Flexibility	6.76	7.03
Commerce	5.67	5.09
Sys Req	6.70	5.77
Support	6.37	6.25
Performance	6.43	6.00
Interop	6.27	6.50
Apps	6.40	6.51
Sum	13.38	12.47
Count	215	101
Rating	4/10	6/10

At first glance one would look at the sums (Drupal = 13.38; Plone 12.47) and conclude that Drupal is the winner. But Simpson's Paradox tells us to look into the underlying data and make sure that there aren't confounding parameters. In this case, there are--the categories themselves.

If we look at the individual categories, we see that Plone outranks Drupal in 6 of 10 categories. By this reckoning, one could say that Plone is the superior CMS.

But again, to harp on my theme of requirements-driven decision making, one should consider "what you need your CMS to do" in your particular and probably unique circumstances.

Are security and flexibility your top concerns? They are at the national laboratory where I work. Commerce and system requirements rank very low in our situation--we don't "sell" anything and we have complete control over our servers--we can create any required system environment. Interoperability is a big one for us, since we develop on Windows testbeds and deploy on UNIX machines. Plone makes perfect sense.

If you are weak on Python, heavy on PHP, and can't get your ISP to install a Zope instance, you will likely find Drupal the preferred product. Of course, I'd argue that your PHP programmers could learn Python in a flash and its a far better language in terms of code maintenance and scalability, but that's a discussion for the Pythonistas to hold.

Bottom line: Requirements, requirements, requirements!