December 12, 2006
For a client's web site I needed to calculate percentages and determine if the differences were statistically significant. In other words, if Group A answered yes 53% of the time and Group B answered 55% of the time, is that 2% difference likely to mean anything?
The first step is to compute the averages. That's easy enough.
Next come the standard deviations. Assuming the markup I'm about to write displays properly the calculation is:
σ² = i=1Σn (x - xi)² / (n - 1)
In English, and accounting for the fact that HTML can't quite show what I want to*, that line says that to get the variance (the square of the standard deviation) you:
- Find the difference between each measurement and the average
- Square each of those differences
- Add those all together
- Divide what you get by one less than the total numebr of measurements
If you want the standard deviation you just take the square root of that whole mess.
OK, now we've got the variance of each group. Now we need to figure out the standard error. This is a measurement that lets us compare two groups of different sizes.
σA-B = √ (σA² / nA) + (σB² / nB)
For each group, divide the variance by the number of measurements. Add those to numbers together, then take the square root of the result. We now have the standard error (though I may not have written it out in "standard" stats format).
And now we're almost home. Our standard error is kind of like a standard deviation for both of the populations combined. Now, if we assume the normal bell curve, we have the following rough estimates about a sample:
- About two thirds of the group is within one σ
- About 95% is within 2σ
- About 99% is within 3σ
So let's find the difference between the two averages and see how many standard errors apart they are:
Z = |xA - xB| / σA-B
That one's easy enough. Now, those numbers (1σ = 66%, etc.) aren't quite accurate. The 95% figure is actually at 1.96σ. So if the two averages are more than 1.96 standard errors apart there's a 95% chance we're dealing with two different results. In statistician-speak, there's a probability (p) of 0.05 (5%) that they're not different. As Z goes up, the p of them being statistically insignificant goes down.
In most cases, the 95% chance is good enough, so if the two populations have a Z ≥ 1.96, we can say that the difference is statistically significant. And that means that you can say Group A is doing better (or worse) than Group B, or (if Z < 1.96) the two groups are basically the same.
The last (and only) stats course I took was in 1997. And just about every stats page I found online wanted me to read five chapters of material just to get the math. After a few days of searching I finally found the equations (and simple explanations) that let me write the VBScript code to do the math I wrote above.
Or, I could have paid attention in that stats class junior year.
* Yes, I could use MathML. There are only two problems with that. 80% of the world can't see it, and it requires about 5 megs of XML to write "2 + 2 = 4". I'll just stick with HTML and style sheets and let whoever's reading this figure out that the "i=1", "Σ" and "n" are all supposed to be vertically aligned.
