The American Statistical Association released a statement yesterday on the (mis)use of p-values. It includes 6 principles on p-values which I think are worth repeating and spreading:
- P-values can indicate how incompatible the data are with a specified statistical model.
P-values do not measure the probability that the studied hypothesis is true, or theprobability that the data were produced by random chance alone.
Scientific conclusions and business or policy decisions should not be based only onwhether a p-value passes a specific threshold.
- Proper inference requires full reporting and transparency.
A p-value, or statistical significance, does not measure the size of an effect or theimportance of a result.
By itself, a p-value does not provide a good measure of evidence regarding a model orhypothesis.
Like Tim Haab (see his post and take on p-values, to which I agree, here) I am not much of a statistician or econometrician. I gladly leave that to others in our research group. However, I do teach a course in advanced quantitative methods in our M.Sc. program in cultural sociology, and that requires of course that I explain the p-value. If students of that class read this blog, I hope they recognize the above principles at least partly from my classes, even though I have not treated the principles separately and explicitly.
Overall, I do tend to try and make clear that:
- Statistical significance is not sociological (or economic, or biological, or …) significance
- The absence of proof is not a proof of absence.
- Correlation is not causation.
- The importance of full reporting and transparency.
Incidentally, the last principle is one of the main reasons why the course is taught in R; it makes the results more readily replicable, although it generates an additional complication:students have to learn how to program, in addition to learning about p-values.
*Yes, I know it’s a terrible pun, but I do hope that it helps boosting the effect size of the ASA-statement
Illustration credit: Tyler Vigen. http://tylervigen.com/spurious-correlations