Meaning in Hyperspace: Word Embeddings as Tools for Cultural Measurement

Abstract

Word embeddings are language models that represent words as positions in an abstract many-dimensional meaning space. Despite a growing range of applications demonstrating their utility for sociology, there is little conceptual clarity regarding what exactly embeddings measure and whether this matches what we need them to measure. Here, we fill this theoretical gap by clarifying how cultural meaning can be understood in spatial terms. We argue that embeddings operationalize context spaces, where words’ positions can reflect any regularity in usage. We then examine sociologists' embeddings-based measurements to argue that most sociologists are instead implicitly interested in capturing concept spaces, where positions strictly indicate meaningful conceptual features (e.g., femininity or status). Because meaningful features yield regularities in usage, context spaces can proxy for concept spaces. However, context spaces also reflect surface regularities in language—e.g., syntax, morphology, dialect, and phraseology—which are irrelevant to most sociological investigations and can bias cultural measurement. We draw on our framework to propose best practices for measuring meaning with embeddings.