Trends from Builtwith

I just found BuiltWith Trends, which has some interesting CMS data among other things. My understanding is that they sample a large (but unspecified) number of domains and determine the web technology used. Looks like they're only publishing data beginning late last November.

The graph here definitely puts the Google Trends stats in perspective. I've extracted the individual data for a couple CMS and compiled them into one chart. It turns out that Joomla and Plone each account for a tiny fraction (<0.02%), whereas Google Trends shows Joomla far outstripping Drupal. Speaking of Drupal, they just recently scrabbled above 1.0% after bottoming out at 0.70% last Christmas.

At the bottom of each technology page is a pie chart showing how the last survey slices the entire domain, in this case, CMS. Drupal's 1.35% is 38.5% of all CMSs surveyed. Joomla comes in with 5.13% and Plone has 1.71%. In terms of ranking, Drupal is on top, Joomla is fifth, and Plone ties for eighth.

BuiltWith conveniently lists top sites using a given technology, although I'm not sure how they determine this. Plone's top sites are the CIA, Discover Magazine, ACM, and Connexions. Typically BuiltWith lists a maximum of 20 top sites, so I'm not sure how they missed Oxfam America, NASA Science, and a couple thousand others.

Just to give you an idea of the other's top sites, Drupal's top 4 sites are BrightCove, Us Magazine, iVillage, and NW Source. Joomla's are The Hill, RCN, SpellingCity, and everythingiCafe.

Clicking on a listed site will take you to a summary page that displays all the technologies that BuiltWith was able to extract from the domain in question. Very interesting stuff.

Like all web stats, they are to be used cautiously, all the more so when an explicit methodology is not stated. None-the-less, I'll definitely be following BuiltWith to see how things track over time. There's considerable noise in the data and one can't yet tell a trend from seasonal noise or something associated with a new version rollout.

All that said, this may be as close to market share as we're likely to get in the near future. The numbers don't segment the marketplace, so we still don't know if Plone is killing Drupal or vice versa in the government, education, and not for profit areas. Quite frankly the big surprise for me was that the total usage percent for CMS is only 3.67% of all the sites sampled by BuiltWith. Looks like global domination is still a ways off.

Usability of CMS Home Sites

Color in the News

An interesting article on color and cognition turned up in Science today and its making its way into the mainstream media. The long and the short of it is that the color blue unconsciously enhances the performance of creative tasks. The color red unconsciously enhances the performance of detail-oriented tasks.

I think the take home lesson in terms of home site usability is that, all other things being equal, one should emphasize blue on those pages whose purpose is to elicit a positive decision about a CMS product. I believe that evaluating software and making a go/no-go decision is a creative task. Detail-oriented tasks (which would involve red pages) are in-depth analysis and feature comparison, more like reading and digesting CMS Matrix results and such.

Usability, Part 3

I'm continuing with my review of CMS and usability, triggered by my elderly parent's difficulties with accessing information on the web. I certainly can't survey hundreds of various CMS implementations and I don't have a handy panel of users to conduct timed usability tests, so I'm looking at something a little less ambitious (and a good bit less rigorous). I have scored each of the parent sites of a few top-flight CMSs against Jakob Neilsen's "Top Ten Mistakes in Web Design."

For the moment, I've limited my study to plone.org, drupal.org, joomla.org, and dotnetnuke.com. The results are displayed in two tables, again due to my CSS not displaying wide tables. Italics are simply to visually separate rows. The snippets of text are to help readers link a particular score with a discussion item in the paragraphs below.

The results have been enlightening and they mirror my earlier heuristic look-and-feel analysis. In the tables below, I've scored a point for the presence of each of Neilsen's top ten mistakes. Half points were given where the flaw was not deemed egregious or of minor impact. I urge readers to go to AlertBox and get the full description of Jakob's top ten mistakes.

Mistake	Plone		Drupal
Bad search
PDF files for online reading
Not changing the color of visited links			1	#0062A0 and #027AC6
Non-scannable text	0.5	No "F pattern"	1	Walls of text everywhere
Fixed font size
Page titles with low search engine visibility	1	"How to"	1	"Join"
Anything that looks like an advertisement			1	True advertisements
Violating design conventions
Opening new browser windows
Not answering users questions	1	"Market-ese"	1	Reverse chronological listing on homepage
Total	2.5		5

Plone. The homepage is mercifully uncluttered, but doesn't display information in the highly scannable "F-shaped" pattern that eye-tracking studies have shown is common when users are looking for something. Page titles in the documentation could be better and its often difficult to find a specific article unless you remember the exact wording. Try to "lock down your site" so people can't "join as users?" You have to search for "Close your site." Plone also has far too many items that start with the phrase "How to...." And finally items in the marketing box at the bottom of the page use catchy phrases that don't necessarily fit with the questions or concepts that a prospective new user might have. None-the-less, Plone.org had only 2.5 of Neilsen's baddies.

Drupal. Meanwhile, the heaviest hitter among CMS has a much weaker home site. Visited links barely change color, large walls of unscannable text are everywhere, and true advertisements are prominent. While it has an obvious simple search box, trying to "lock down your site" so people can't "join as users" results instead in hundreds of items about using "join" in queries. The lengthy reverse chronological listing of news items only serves to obscure important, basic information that first-time site visitors might be looking for. Drupal comes in with a disappointing 5 out of 10.

Mistake	Joomla		DotNetNuke
Bad search	1	No search box
PDF files for online reading
Not changing the color of visited links			1	No change at all
Non-scannable text	0.5	Mini-walls of text	1	Icons scattered; menu bar; "toolbar"
Fixed font size
Page titles with low search engine visibility	1	No "How to"	1	No "How to"
Anything that looks like an advertisement	1	"Contribute" icon	1	True advertisements
Violating design conventions	1	No "Home"	1	Pulldowns interfere with toolbar icons; no "Home"
Opening new browser windows
Not answering users questions	1	"What is?"	1	Questionable relevance; Where is the documentation? "Downloadable files"
Total	5.5		6

Joomla! This was the only home site that lacked an obvious simple search feature. When I finally found a search box one level down, its advanced search was useless to an outsider. How would I know to search a "chunk talk," a "transwiki," or a "JDOC?" Its first page text areas were reasonably deployed, although not in an "F-pattern," but each was a monolithic mini wall of text. The look-and-feel of the documentation area was completely different from other pages. The "Contribute" link looked like a pitch for money, and their was no explicit "Home" link on many pages nor easy to recognize bread crumbs. It took some getting used to the non-standard nav menu with its confusing "Main page" and "Main" links that went to different places. Finding "How to" documents was frustrating. Perhaps worst of all, there are things like the "Getting Started" link, which takes you to a "What is" section where the download link is buried below many column-inches of scrolling text. Joomla's site was beautiful and balanced but inconsistent not very usable with a score of 5.5.

DotNetNuke. This site had a number of Neilsen's sins. Visited links didn't change even slightly, icons were scattered across the page in both a panel of images and in a toolbar-like menu. The text menu bar expanded down and interfered with seeing the icons below and was awkward to clear. Finding "How to" information or any documentation at all was difficult. Why would I look under "Downloadable files" to find documentation? When I did search for items, the returned set was of questionable relevance. My question about "locking down a site" returned results that were absolutely off mark. The presence of true advertisements detracted from the main purpose of the site, unless the main purpose is to generate revenue for sponsors. DotNetNuke came in at the bottom of our four sites with 6 critical mistakes.

Summary. Clearly and not unexpectedly, all four CMSs have some weaknesses. Surprisingly, Plone comes out ahead of the pack while the other three had 5 to 6 of Neilsen's top ten usability mistakes.

Drupal -- Plone -- DotNetNuke

I've been playing around with a logo for the blog and came up with this. It also might be useful for the Plone Metrics and Statistics group over at OpenPlans. It does occur to me that its a repurposing in part of the Plone logo, so I should check with the Plone Foundation about the proper way to handle this.

Today's topic is the Packt Publishing Awards now that the overall winners have been announced. Something doesn't seem quite right. The logic of the matter goes like this:

1. In the non-PHP category Plone was first, dotCMS was second, and DotNetNuke was third. Thus, Plone ranks higher than DotNetNuke.
2. In the PHP category, its Drupal > Joomla! > CMS Made Simple.
   
3. In the overall category, it turned out Drupal > Joomla! > DotNetNuke.

Hmm? Recall that Plone was a finalist in the overal category. Since from (1) above Plone ranked higher than DotNetNuke, one could infer that it should be somewhere to the left of DotNetNuke in (3). The question becomes, where did Plone end up in the overall category and why was it not ranked higher than DotNetNuke? Or put another way, why didn't DotNetNuke beat Plone for the non-PHP title?

Curious.

From the inconsistency in the results, we can't compare Plone with Drupal and Joomla!. How can Plone be ranked simultaneously both higher and lower than DotNetNuke?

The answer may lie in the fact that the Packt voting process separates the votes for each category. I would suggest that DotNetNuke fans voted well in the overall category and then forgot to punch in also for the non-PHP category. Does this mean DotNetNuke voters haven't noticed that their favorite CMS isn't PHP based? Or perhaps Plonistas only voted in the non-PHP category and didn't go to the overall section.

But then there's another factor at work here: Packt judges used an undisclosed method in addition to the popular vote to rank the contenders. I have no idea how number of Packt titles (Drupal 9, Joomla 9, DotNetNuke 3, Plone 3), sales figures, and so on might influence the judges. One might expect that the Packt Awards are foremost meant to benefit Packt by selling more books. In the absence of a public, objective methodology and with the inconsistent ranking of Plone and DotNetNuke, one can only guess.

Whatever the actual phenomenon, it points out that one can take these kinds of popularity contests only so far. Yes, we can now market Plone as the top non-PHP CMS for 2008 and that has a certain cachet. And I'll admit I was thrilled to see Tuesday's result roll in.

But deep down, it isn't popularity that should guide your software choices. True, highly popular systems will likely have more consultants, a larger professional base, and possibly a longer useful life. But that doesn't guarantee that the popular use-case is going to match your use-case. Other requirements need to be factored in--your internal support capabilities, deployment platform, user environment, security, overall goals, and many other needs.

In my case, our predisposition for Python and the need for a secure system were the key factors that drove us to Plone. The National Vulnerability Database lists ten records for Plone over the last three years. Drupal shows 158 records; Joomla! 265. Or put another way, "...it's striking that three [Drupal, Joomla!, Wordpress] of the Top Ten contenders on IBM's security worry-list have PHP in common. You can read whatever you want to into that, I suppose." (Kas Thomas, 8/2008)

For me, that's the end of the voting.

Visualizing CMS

I'm still working on quantifying this, but here are some screen captures of website visualizations from http://www.aharef.info/static/htmlgraph/. Fun stuff. The colors are coded as follows:

blue: for links (the A tag)
red: for tables (TABLE, TR and TD tags)
green: for the DIV tag
violet: for images (the IMG tag)
yellow: for forms (FORM, INPUT, TEXTAREA, SELECT and OPTION tags)
orange: for linebreaks and blockquotes (BR, P, and BLOCKQUOTE tags)
black: the HTML tag, the root node
gray: all other tags

Finding the black root node in these small images can be tricky. Look for a starburst of grey nodes with no further daughter nodes and then go upstream.

From top to bottom they are:

• Drupal.org
• Plone.org
• Wordpress.org
• Joomla.org

CMS Blogs Stats

I was watching Darci Hanning's "Top Ten Ways to Get Involved with the Plone Community" and was pleased that first out of the box was "Blog about Plone." So I thought I'd look at some stats on CMS and blogs.

Technorati has Plone posts averaging a little under 20 per day for the last month with a total of 6866 posts in 134 blogs. By way of comparison, Drupal is listed with 12,853 posts in 2269 blogs. Joomla has 10,742 posts in 1389 blogs. SharePoint: 11,305 in 1451. Alfresco: 447 in 58. Openedit: 11 in 1. DotNet Nuke: 8 in 5.

What is one to make of this? Of course, first, its all just numbers. There could just as easily by 6800 outstanding posts for Plone and 12,000 dirges for Drupal or the other way around. But clearly the buzz is, whether good or bad, is more about Drupal than anyone else.

However, there appears to be more than just the usual continuum in these numbers. We've got four CMS that each are in the many thousands of blogs and three that are only in double digits or less.

Even so, there's yet another way to look at things: posts per blog. Take a look at this summary table sorted in descending order by posts/blog.

CMS	Posts	Blogs		Posts/blog
Plone	6866	134		51.2
Openedit	11	1		11.0
SharePoint	11305	1451		7.8
Joomla	10742	1389		7.7
Alfesco	447	58		7.7
Drupal	12853	2269		5.7
DotNet Nuke	8	5		1.6

Wow! Plone blows everything on the chart away, even the newcomer and statistical outlier Openedit barely makes it above 10 posts/blog. Darci is on to something here--Plone bloggers may not be as numerous as SharePoint's or Drupal's, but they are either very active or have been at it for a long time. (Case in point, Plone Metrics has had 70 posts since last October.)

BTW, with all this poking about in blogs, I did turn up some interesting reading. Limi's latest on simplifying Plone and the editing experience is a good place to start. Don't miss Christian Schlotz's posting on Data Portability.

I also stumbled upon an NGO on "Ask Metafilter" asking for a comparison of Plone and Drupal. Of the 15 comments, most were adamantly anti-Plone, but seemed to be ill-informed, based on incomplete or obsolete knowledge. I'll have to register at Metafilter ($5, bummer) and then post an up-to-date Plone advocacy comment over there.

The Weighting Game

Spent the morning doing laundry and looking back at the recent InfoWorld CMS ratings. These ratings illustrates a couple of dangers and a couple of best practices.

My first points have nothing to do with InfoWorld, but rather on what people do with review data.

• On the down side, the ratings were taken by Matt Asay and resulted in his statement "The winner? Alfresco, and by a significant margin (over Plone, Drupal, DotNetNuke)." "Significance" is not a term to be lightly tossed around when dealing with statistics. It is usually accompanied by a significance level (often 5%, but occasionally "highly significant" at 1%) and denotes a strict statistical formula for distinguishing hypotheses. The InfoWorld data is not designed to provide significance sensu strictu and one is left to imagine what the difference of 8.6 vs 9.2 means. (Even then its actually 9.15, but they rounded up.)
  
• Also, Matt forgot to mention that Joomla was scored in the survey and came in third.
  
• On the positive side, Matt does full disclosure--he's VP of Business Development for Americas for Alfresco.
  

However, InfoWorld doesn't get a free statistical ride today.

• On the positive side, they provide a link to a methodology page and make an effort to justify their results. This is all too rare and should be emulated.
  
• On the negative side, they don't explain their selection of categories, even though they state that some combination of the listed categories will be used. The absence of Availability is understandable (these are all open-source), but Performance, Reliability, Setup, and Support should have been addressed.
  
• Also, their methodology does not describe what is considered under the Feature category. Many would say that Interoperability and Setup are features. Curiously, only Alfresco warranted a 10 for Features and I seriously doubt that anyone, especially the developers at Alfresco, would claim that their feature set is 100% complete.
  
• They also never explain how they arrived at an 86-80-70-60-50 grading curve when one expects 90-80-70-etc. On top of that they use rating names to bin results, thus disguising the numerical results (still my favorite complaint against school grading of A-B-C-D-F).
  
• Finally, InfoWorld never explains the rationale behind their weighting of categories (25-25-15-15-10-10). If one doesn't weight scores (or uses a uniform 16.7% across 6 categories), Alfresco scores 9.0 and Plone comes in at 8.7, which puts them both in the InfoWorld "Excellent" bin. The weighting clearly doubles the Alfresco "gap," making it appear a clear leader, and moves Plone just barely into the "Very Good" rating.
  

Armed with this critic we can play all sorts of statistical games. (Remember that Mark Twain said that there are lies, damn lies, and statistics.)

• Using equal weights, applying a 90-80-70 scale, and giving Alfresco a more realistic 9.5 for Features, we find that they then are only "Very Good."
  
• Flipping the weights over (10-10-15-15-25-25) puts Alfresco and Plone in a dead heat (8.85 vs 8.75).

Another trick is to play with graphs. Here's the InfoWorld data done as a default Excel bar chart. Alfresco looks far out in front.

But here's a more accurate bar chart with a properly scaled y-axis. What is significant now?


I'd like to wrap up this long posting with a tip o' the hat to the comments by Amy and Bryan at the bottom of the CMS Report posting on the InfoWorld article. Amy from OpenSource.org correctly raises the point that some CMS reviews seem to take a perverse joy in pitting one open-source CMS against another. Bryan rejoins that reviews are both popular and useful as long as everyone stays well behaved.

Here on PloneMetrics, I am an unabashed Plonista. But also I am trying to look at the world with a little more rigor. Tomorrow I'll post the latest on my work to fill in the matrix ala Bullard's method. Stay tuna'd.