On hapax legomena and morphological productivity

Conference item

Quantifying and predicting morphological productivity is a long-standing challenge in corpus linguistics and psycholinguistics. The same challenge reappears in natural language processing in the context of handling words that were not seen in the training set (out-of-vocabulary, or OOV, words). Prior research showed that a good indicator of the productivity of a morpheme is the number of words involving it that occur exactly once (the hapax legomena). A technical connection was adduced between this result and Good-Turing smoothing, which assigns probability mass to unseen events on the basis of the simplifying assumption that word frequencies are stationary. In a large-scale study of 133 affixes in Wikipedia, we develop evidence that success in fact depends on tapping the frequency range in which the assumptions of Good-Turing are violated.

Authors: Pierrehumbert, JB; Granell, R

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for Computational Linguistics
• Host title: Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology
• Journal: SIGMORPHON 2018
• Pages: 125–130
• Publication date: 2018-10-01
• Acceptance date: 2018-08-19
• DOI: 10.18653/v1/P17
• ISBN: 9781948087766