Journal article

Visual and Affective Multimodal Models of Word Meaning in Language and Mind

Simon De Deyne, Danielle J Navarro, Guillem Collell, Andrew Perfors

COGNITIVE SCIENCE | WILEY | Published : 2021

Abstract

One of the main limitations of natural language-based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text corpora and those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and a..

View full abstract

Grants

Awarded by ARC


Funding Acknowledgements

We acknowledge the valuable contributions of Andrew Olney and two anonymous reviewers who reviewed a previous version of this manuscript and Meredith McKague for valuable input and discussion. Salary support for this research was provided to Simon De Deyne from ARC grants DE140101749 and DP150103280. Guillem Collell acknowledges the CHIST-ERA EU project MUSTER (http://www.chistera.eu/projects/muster).Data for the triads, together with the language and image embeddings, are available at https://simondedeyne.me/data/