Meaningful Stats?

Dylan Jay recently asked on the Plone Evangelism msg board, "Are there any stats about Plone's success that are really meaningful?" My first thought was to reply over there. But since the question is truly existential from a statistician's point of view as well as being of general interest, I thought I'd post a reply out in the wider blogosphere.

Dylan had also queried in his post by wondering if there's a way of determining amalgamated Plone revenue? His reasoning went that as a community, are there numbers that can compare apples with apples when looking across open-source and commercial products?

The difficulty here is that any single measure we pick will be integrating across multiple dimensions: large customer companies vs small, community sites vs web publishing, dot com's vs dot org's, and so on. In this blog I've explored (and continue to explore) various surrogates for the health of Plone and related CMS's.

Whether its PageRank at Amazon, BuiltWith percentages, Google trends, security vulnerabilities, Plone.net sites, Technorati posts, CMS Matrix features, or other statistical tidbit, these all fall short. Which is why I've always advocated a requirements-based decision process for determining if Plone is right for a particular situation. But that doesn't stop me from questing for the Holy Grail of web statistics--a simple way to measure effectiveness.

I'm still of the opinion that an aggregated set of metrics will be useful in this regard. See for example, one of my earliest posts where I was trying to fill in the gaps in a table where widespread adoption of Plone was based on acceptance and visibility. One of the attributes of acceptance was the economic health of 3rd party companies, something we still don't have a solid handle on (sorry Dylan).

Meanwhile, there are new "sentiment analysis" tools coming out as per this morning's NY Times. Of the free services, things look pretty rough. For example, Twitrratr lists as a negative post this tweat from our very Plone-positive friends in Pennsylvania, "xml is the wrong language for confi..." This confuses a post's source (planet Plone) with the keyword "wrong" and misses the subject of the negative feelings, "XML." The algorithms still need work, but here in the interest of fairness is a rundown.

	Postive	Neutral	Negative
Twitrratr	9.09%	87.66%	3.25%
Twendz	30%	59%	11%
Tweetfeel	0%	n/a	100%

Clearly Twitrratr is displaying more precision than their algorithm merits and Tweetfeel (with n = 2) is missing most of the traffic of interest. Twendz does give a few details on how they are dynamically processing up to the latest 70 tweets. I'll keep an eye on these and related products to see how the technology is maturing and how it can help the Plone community improve itself.

Finally, in closing today I thought I'd finish up not with more statistics, but with a couple anecdotal items recently seen on Twitter:

Collaboration via Sharepoint is like kissing a warthog in August.

and

Trying to create a good site with MS SharePoint... cumbersome... I rather prefer Plone! #microsoft #plone.

Anecdotal evidence by definition isn't statistically significant and can be easily dismissed by those who need numbers to back up their decisions, but it occurs to me that I should be cataloging these sorts of statements. In time, I suspect we'll have quite a corpus because Plone does good stuff.

Next week: The diffusion of innovation and why people often standardize on sub-optimal solutions. Black swans, path dependence, network externalities, and the lack of information on the relative merits of systems.

Developer Community Growth

Tom (twrichar) asked the other day, "does anyone have any statistics on how the Plone DEVELOPER community may be growing, hopefully over a (recent), multiyear span?" There is one recent answer to this quesion: Chris Calloway's code swarms. They are a terrific way to visualize movement in the community. The code_swarm project page at Google explains what the video is displaying.

But if someone is interested in numbers, here are two that I've got. In October through November, 2007 I counted 45 participants in the Core Developers forum on Nabble. Repeating the exercise today for the interval 23 Oct. through 23 Nov. 2008, we find 75 core developers who have actively posted. That's a 60% increase in a year.

Other stats that I've been watching include the number of sites listed on Plone.net, which is now up to 1420.

The graph shows a very linear trend. At this rate we'll break the 2000 sites barrier before the end of 2009.

The fact that we have linear growth belies some statements I've seen by observers of the CMS scene who think that Plone has passed its peak and may be becoming a niche player (sorry, can't recall the exact citation).

Another metric of interest, also from Plone.net, is that the number of providers listed there has increased from 225 last March to 287 today. That's a 28% increase in 6.5 months.

I've also been tracking Plone releases. I've plotted them such that, for example, version 3.1.4 is placed at 3.14. That means major releases have ten times the weight of minor releases and bug fixes have ten times less weight than minor releases. My guess is that minor releases should have more weight.

At any rate the graph at left shows the zigs and zags of releases, where there is on-going support for one version (for example, 2.5) while another major release takes off (3.x).

The regression line has a slope of 0.001, which means there is 1/1000th of a major release per day. Taking the inverse, we have 1000 days (2.7 years) per major release.

This picture is complicated by the fact that 2.5 was clearly a major release. If we elevate 2.5 to major release status, it basically doubles the slope, changing release frequency to 1.4 years per major release.

Regression Coefficients

I shared the graph of Plone site development at Sandia with a coworker and she asked what the numbers in the upper right-hand corner meant. So I figured that might be a common question and it would be worthwhile to discuss it here in today's posting.

The R-squared is the regression coefficient, which describes how much of the variability (the "wiggliness" among the data points) is explained by the linear equation. In this case, the coefficient is over 98%, meaning that the line is significant (>95%) but not highly significant (>99%). 98% of the "noise" in the data can be explained by the linear trend.

The linear equation y = 0.0269 x - 1031.5 simply defines the predicted best-fit straight line as having a slope of 0.0269 sites/day (about 1 site every 37 days). There is a non-zero x-intercept (where the line hits the x-axis, that is, has a y-value of 0) because we started our data with the first Plone site on day zero. That means that zero Plone sites is somewhere to the left of the origin.