"Count what is countable. Measure what is measureable. What is not measureable, make measureable." -- Galileo

Sunday, September 30, 2007

Here are some interesting tidbits and links:
Also, here is the link to an Excel version of my matrix of effectiveness measures.

I'll be looking for more up-to-date material in the days and weeks ahead, but its clear that people have been mulling over the problem since the late 90s. I'd like to hear from those attending the Naples Convention on what they learn about the issue.

Wednesday, September 26, 2007

A few good URLs

A recent review of CMS (CMS Watch) showed Plone doing well in a wide variety of categories: standards, access control, i18N, aggregation, repository services, user-generated content, micro-applications, active user groups, and good value.

Another place to watch is the CMS Matrix, which is an online feature comparison tool. Of course, feature comparisons do not in and of themselves make a winning product--matching requirements with features and your IT capabilities is the ticket.

Tuesday, September 25, 2007

More Measures of Effectiveness

Here's the rest of the Plone Effectiveness Measurement table, starting with attributes and drilling down to measures, including a column of notes about where we might be able to gather the necessary inputs.

Attribute Measure Notes
Release Size Major, minor, RC, beta, alpha Sourceforge history
Release Frequency Date
Bugs Subsequent bug reports See FLOSS Study of Available Tools
Core mailing lists Activity per unit time
Support mailing lists Activity per unit time
Security vulnerabilities Number of known vulnerabilities MITRE CVE list of known vulnerabilities
Size of CDT Number of core developers
Involvement of CDT Length of service; role Questionnaire
New features Major and minor features included per release http://plone.org/products/plone/roadmap
Downloads Downloads per platform http://sourceforge.net/project/stats/?group_id=47214&ugn=plone
Installations Number of installations ?
Defectors Number of installations removed Uninstall feedback to Plone.org
Economic health of third-party companies Metrics from Plone Foundation, Enfold, One/NW, etc. Direct requests?
Technical reviews Number and 'rating' from published reviews Data mining?
High-profile installations Number and 'quality' of high-profile users http://plone.org/about/sites
Users of Plone portals Number of registered members of Plone portals Sampling?
Visitors to Plone portals Number of visitors to Plone portals Sampling?


Tomorrow I'll post a listing of URLs for various reviews and comparisons of Plone and other CMS.

Monday, September 24, 2007

Plone Metrics Goes Online

OK, so I couldn't talk management into Naples. That means I'll miss this year's World Plone Conference. And its an important one: Plone 3.0 is the current stable release and I need to spin up on all the new capabilities.

Best I can do for the Plone community is add my voice from afar. I've been corresponding intermittantly with Paul Everitt, this year's chair of the Plone Foundation, about metrics for Plone. One idea that's gotten a little traction is a method by Richard Bullocks from, of all things, the Air War College. He starts with a desired end-state and drills down to objectives, their values, and measurable attributes. See his dissertation for the full 188-page blow-by-blow description.

For Plone, I've come up with the following.

Plone End-State Objective Value Attribute
Stable yet evolving Software Releases Release Size
Release Frequency
Software Quality Bugs
Core mailing lists
Support mailing lists
Security vulnerabilities
Core Development Team Size of CDT
Involvement of CDT
Features New features
Widely adopted Acceptance Downloads
Installations
Defectors
Economic health of third-party companies
Technical reviews
Visibility High-profile installations
Users of Plone portals
Visitors to Plone portals


In a future posting, I'll add measures, sources of these metrics, and examine some of the ways we might obtain a valid and useful measure of Plone progress towards a successful end-state. Please comment freely on what you see as strengths and weaknesses of my methodology, not to mention on Plone itself.