"Count what is countable. Measure what is measureable. What is not measureable, make measureable." -- Galileo

Sunday, November 25, 2007

Measures of Effectiveness

Back in September I introduced Bullock's dissertation on measures of effectiveness. In it he argues that for an entity to reach a desired end-state, one must identify objectives that are described by values. These values will have attributes, which are in turn measurable.

I did not have to identify the exact Plone end-state, choosing instead to focus on two objectives: widely adopted and stable yet evolving. I believe these two objectives form a reasonable desired state for Plone, although the term "end-state" seems a little too final for something that we all hope is on-going.

The problem that Bullock recognized is that objectives and values are quite likely something that can not be directly observed. We only have hard measurements for attributes of values. That's where his use of a Kalman filter is novel. A Kalman filter essentially uses observed attributes to predict unobservable values. Its a useful way of pooling apples and oranges.

Here are my slightly revised attributes, measurements as of 24 Nov. 2007, and the units.

Attribute Observed Actual Measure

Release Frequency 0.94 Years/major release
Bugs 704 Active tickets
Core mailing lists 1.875 Msgs/day
Support mailing lists 17.5 Msgs/day
Security vulnerabilities 4 Recent but resolved
Size of CDT 48 Oct & Nov 2007 from Core Dev forum
Involvement of CDT
New features 95% Percentage of CMS Matrix features
Economic health of third-party companies
Technical reviews 8.7 InfoWorld
High-profile installations 950 Plone.net/sites
Users of Plone portals
Visitors to Plone portals

As you can see, I still have some attributes to collect. As a proxy for development, use, and usage, I may look at measures of activity on Openia and Objectis. More to come.

No comments: