Here are some interesting tidbits and links:

• Nucleus Research 2003 whitepaper "ROI Comparison Report: Portals"
• "Recommendations for Benchmarking Web Site Usage among Academic Libraries" by Christy Hightower, Julie Sih, and Adam Tilghman (1998 but still interesting)
• "Content Management Problems and Open Source Solutions" by Seth Gottlieb (2005)
  
• "Defining Open Source Software Project Success" by Kevin Crowston, Hala Annabi, and James Howison (2003)

Also, here is the link to an Excel version of my matrix of effectiveness measures.

I'll be looking for more up-to-date material in the days and weeks ahead, but its clear that people have been mulling over the problem since the late 90s. I'd like to hear from those attending the Naples Convention on what they learn about the issue.

A few good URLs

A recent review of CMS (CMS Watch) showed Plone doing well in a wide variety of categories: standards, access control, i18N, aggregation, repository services, user-generated content, micro-applications, active user groups, and good value.

Another place to watch is the CMS Matrix, which is an online feature comparison tool. Of course, feature comparisons do not in and of themselves make a winning product--matching requirements with features and your IT capabilities is the ticket.

More Measures of Effectiveness

Here's the rest of the Plone Effectiveness Measurement table, starting with attributes and drilling down to measures, including a column of notes about where we might be able to gather the necessary inputs.

Attribute	Measure	Notes
Release Size	Major, minor, RC, beta, alpha	Sourceforge history
Release Frequency	Date
Bugs	Subsequent bug reports	See FLOSS Study of Available Tools
Core mailing lists	Activity per unit time
Support mailing lists	Activity per unit time
Security vulnerabilities	Number of known vulnerabilities	MITRE CVE list of known vulnerabilities
Size of CDT	Number of core developers
Involvement of CDT	Length of service; role	Questionnaire
New features	Major and minor features included per release	http://plone.org/products/plone/roadmap
Downloads	Downloads per platform	http://sourceforge.net/project/stats/?group_id=47214&ugn=plone
Installations	Number of installations	?
Defectors	Number of installations removed	Uninstall feedback to Plone.org
Economic health of third-party companies	Metrics from Plone Foundation, Enfold, One/NW, etc.	Direct requests?
Technical reviews	Number and 'rating' from published reviews	Data mining?
High-profile installations	Number and 'quality' of high-profile users	http://plone.org/about/sites
Users of Plone portals	Number of registered members of Plone portals	Sampling?
Visitors to Plone portals	Number of visitors to Plone portals	Sampling?

Tomorrow I'll post a listing of URLs for various reviews and comparisons of Plone and other CMS.

Plone Metrics Goes Online

OK, so I couldn't talk management into Naples. That means I'll miss this year's World Plone Conference. And its an important one: Plone 3.0 is the current stable release and I need to spin up on all the new capabilities.

Best I can do for the Plone community is add my voice from afar. I've been corresponding intermittantly with Paul Everitt, this year's chair of the Plone Foundation, about metrics for Plone. One idea that's gotten a little traction is a method by Richard Bullocks from, of all things, the Air War College. He starts with a desired end-state and drills down to objectives, their values, and measurable attributes. See his dissertation for the full 188-page blow-by-blow description.

For Plone, I've come up with the following.

Plone End-State	Objective	Value	Attribute
	Stable yet evolving	Software Releases	Release Size
			Release Frequency
		Software Quality	Bugs
			Core mailing lists
			Support mailing lists
			Security vulnerabilities
		Core Development Team	Size of CDT
			Involvement of CDT
		Features	New features
	Widely adopted	Acceptance	Downloads
			Installations
			Defectors
			Economic health of third-party companies
			Technical reviews
		Visibility	High-profile installations
			Users of Plone portals
			Visitors to Plone portals

In a future posting, I'll add measures, sources of these metrics, and examine some of the ways we might obtain a valid and useful measure of Plone progress towards a successful end-state. Please comment freely on what you see as strengths and weaknesses of my methodology, not to mention on Plone itself.