Measurement and Uncertainty

Today I came across Mike Cassady's piece entitled "Does Lack of Measurement Mean Uncertainty?" on a Sandia internal website (sorry, no public URL). He began by posing that logical positivism rests on an assumption that "to measure is to know; not to measure is to remain uncertain." He suggests that they rely on the fact that "We are comfortable using numbers as an assessment tool.... but we are suspicious of any assessment approach that doesn't rely on objective data computed numerically."

But then, unexpectedly for a place full of engineers and scientists, Mike goes on to say "If you can't measure it numerically, it's probably ten times more important than anything you CAN measure numerically." He goes on saying, "customer sets often live in the gray area of 'customer experience' rather than the firm scientific feather bed of 'measure is meaning' [and] that [programmers] are going to have to broaden their receptivity to qualitative and judgment-oriented measurements."

Hmm? Where does that leave us?

Let me make another observation about a phenomenon called an "information cascade." Here's the nitty gritty from Wikipedia:

Definition: An information cascade is a situation in which every subsequent actor, based on the observations of others, makes the same choice independent of his/her private signal.

Erroneous Mass Behavior: In an information cascades everyone is individually acting rationally. Still, even if all participants as a collective have overwhelming information in favor of the correct action, each and every participant may take the wrong action. The probability that everyone is taking the wrong action is less than 50%, but it is easy to construct examples in which everyone is wrong with 30-40% probability.

Fragility: A little bit of public information (or an unusual signal) can overturn a long-standing informational cascades. That is, even though a million people may have chosen one action, seemingly little information can induce the next million people to choose the opposite action. Fragility is an integral component of the Informational cascades theory!

So, what does this mean for society? Cascades predict that you can get massive social imitation, occasionally leading everyone (the "herd") to the incorrect choice. (Because everyone knows that there is very little information in a cascade, cascades are "fragile"; a little bit of new public information can make a big difference).

For someone trying to decide on the best CMS to implement, they might just look around and see what others (the "herd") have done and make a sub-optimal choice. From this I conclude that we most definitely need to temper judgement-oriented decisions with cold hard objective knowledge. At the very least, we need enough rigor to ensure that an information cascade doesn't occur.