What do (most of) our dispersion measures measure (most)? Dispersion?
Stefan Τh. Gries
Abstract
Abstract This paper discusses the degree to which most of the most widely-used measures of dispersion in corpus linguistics are not particularly valid in the sense of actually measuring dispersion rather than some amalgam of a lot of frequency and a little dispersion. The paper demonstrates these issues on the basis of data from a variety of corpora. I then outline how to design a dispersion measure that only measures dispersion and show that (i) it indeed measures information that is different from frequency in an intuitive way and (ii) has a higher degree of predictive power of lexical decision times from the MALD database than nearly all other measures in nearly all corpora tested.
Topics & Concepts
Dispersion (optics)Measure (data warehouse)Degree (music)Computer scienceVariety (cybernetics)Predictive powerStatisticsNatural language processingArtificial intelligenceMathematicsData miningPhysicsOpticsPhilosophyAcousticsEpistemologyNatural Language Processing TechniquesTopic ModelingAuthorship Attribution and Profiling