Some Right Ways to Analyze (Psycho)Linguistic Data
Shravan Vasishth
Abstract
Much has been written on the abuse and misuse of statistical methods, including p values, statistical significance, and so forth. I present some of the best practices in statistics using a running example data analysis. Focusing primarily on frequentist and Bayesian linear mixed models, I illustrate some defensible ways in which statistical inference—specifically, hypothesis testing using Bayes factors versus estimation or uncertainty quantification—can be carried out. The key is to not overstate the evidence and to not expect too much from statistics. Along the way, I demonstrate some powerful ideas, including the use of simulation to understand the design properties of one's experiment before running it, visualization of data before carrying out a formal analysis, and simulation of data from the fitted model to understand the model's behavior.