Perceiving Talking Faces
Dominic W. Massaro,
UC Santa Cruz
4:00 pm Thu. Sept. 12 in 311 Psychology
Speech perception has been studied extensively in the last decades.
We have learned that people use many sources of information in
perceiving and understanding speech. This talk focuses on the
important contribution of visible information given in the talker's
face in face-to-face communication. Although the influence of
visible speech is substantial when auditory speech is degraded,
visible speech also contributes to performance even when paired
with intelligible speech sounds. Our studies use a synthetic
talking face to achieve control over the visible speech and to
study those visible aspects that are informative. Our talking head
can be heard, communicates paralinguistic as well as linguistic
information, and is controlled by a text-to-speech system. Using
this talking head, auditory and visible speech are manipulated
independently of one another to uncover the fundamental processes
involved in speech perception by ear and eye. Perceivers judge
audible, visible, and bimodal speech under a variety of
experimental conditions. The results from a plethora of studies
are consistent with the Fuzzy Logical Model of Perception (FLMP).
This model assumes three successive but overlapping stages of
processing: evaluation, integration, and decision. At evaluation,
each source of information provides independent and continuous
support for linguistic alternatives. The multiple sources are
combined at the integration stage, and an assessment is made
concerning the relative goodness of match of the relevant
alternatives. This assessment is then mapped into some type of
behavior such as a categorization or a rating judgment, using a
straightforward decision rule. The model makes the strong
prediction that the contribution of one source of information is
inversely related to its ambiguity. The FLMP outperforms other
models that assume categorical perception, selective processing of
one modality, auditory dominance, and averaging of the sources of
information. Finally, it is proposed that the FLMP describes a
universal law of behavior. Demonstrations of the talking head and
various psychological phenomena will be will be provided.