Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Oct 7:11:2242.
doi: 10.3389/fpsyg.2020.02242. eCollection 2020.

How Soundtracks Shape What We See: Analyzing the Influence of Music on Visual Scenes Through Self-Assessment, Eye Tracking, and Pupillometry

Alessandro Ansani  1   2 , Marco Marini  2   3 , Francesca D'Errico  4 , Isabella Poggi  1
Affiliations

How Soundtracks Shape What We See: Analyzing the Influence of Music on Visual Scenes Through Self-Assessment, Eye Tracking, and Pupillometry

Alessandro Ansani et al. Front Psychol. .

Abstract

This article presents two studies that deepen the theme of how soundtracks shape our interpretation of audiovisuals. Embracing a multivariate perspective, Study 1 (N = 118) demonstrated, through an online between-subjects experiment, that two different music scores (melancholic vs. anxious) deeply affected the interpretations of an unknown movie scene in terms of empathy felt toward the main character, impressions of his personality, plot anticipations, and perception of the environment of the scene. With the melancholic music, participants felt empathy toward the character, viewing him as more agreeable and introverted, more oriented to memories than to decisions, while perceiving the environment as cozier. An almost opposite pattern emerged with the anxious music. In Study 2 (N = 92), we replicated the experiment in our lab but with the addition of eye-tracking and pupillometric measurements. Results of Study 1 were largely replicated; moreover, we proved that the anxious score, by increasing the participants' vigilance and state of alert (wider pupil dilation), favored greater attention to minor details, as in the case of another character who was very hard to be noticed (more time spent on his figure). Results highlight the pervasive nature of the influence of music within the process of interpretation of visual scenes.

Keywords: audiovisual; empathy; environment perception; eye tracking; film music; interpretation; pupillometry; soundtrack.

PubMed Disclaimer

Figures

FIGURE 1
FIGURE 1
Illustration of three representative frames of the scene.
FIGURE 2
FIGURE 2
Empathy toward the character as a function of condition in Study 1 (violin plot). The boxplots within each violin represent interquartile ranges (IQRs). Red crosses indicate means, black horizontal lines indicate median, and black points are outliers. Participants in Evans condition showed significantly higher Empathy ratings.
FIGURE 3
FIGURE 3
Plot anticipations as a function of the condition. Participants in Evans condition imagined the character to be remembering something; on the contrary, Rachmaninov’s track led the participants to imagine that the character was taking a decision.
FIGURE 4
FIGURE 4
Empathy toward the character as a function of condition in Study 2 (violin plot). The boxplots within each violin represent interquartile ranges (IQRs). Red crosses indicate means, black horizontal lines indicate median, and black points are outliers. As in Study 1, participants in Evans condition showed significantly higher empathy ratings.
FIGURE 5
FIGURE 5
Plot anticipations as a function of the condition. Coherently with Study 1, the Evans’ track led the participants to imagine a remembering character; in contrast, half of the participants in the Rachmaninov’s group imagined a character on the point of taking a decision.
FIGURE 6
FIGURE 6
Percentage change in pupil diameter as a function of time (s) and condition. Throughout the whole scene, the pupils of the participants in the Evans group had significantly lower dilation as opposed to those in the other conditions.
FIGURE 7
FIGURE 7
Affective state attributed to the main character as a function of the time spent on his moving area of interest (mAOI). The boxplots represent interquartile ranges (IQRs). Black horizontal lines indicate median. As the participants watched the main character to a larger extent, with Rachmaninov’s track, they progressively decreased the affective state attributed to him; conversely, with Evans’ track, they progressively increased the affective state attributed to the main character.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Ansani A., D’Errico F., Poggi I. (2019). ‘You will be judged by the music I hear’: a study on the influence of music on moral judgement. Web Intellig. 1 53–62. 10.3233/WEB-190400 - DOI
    1. Asch S. E. (1946). Forming impressions of personality. J. Abnorm. Soc. Psychol. 41:258. - PubMed
    1. Au K., Chan F., Wang D., Vertinsky I. (2003). Mood in foreign exchange trading: Cognitive processes and performance. Organ. Behav. Hum. Decis. Process. 91 322–338. 10.1016/S0749-5978(02)00510-1 - DOI
    1. Auer K., Vitouch O., Koreimann S., Pesjak G., Leitner G., Hitz M. (2012). “When music drives vision: Influences of film music on viewers’ eye movements,” in Proceeding of the 12th International Conference on Music Perception and Cognition and the 8th Triennial Conference of the European Society for the Cognitive Sciences of Music, New York, NY.
    1. Batson C. D., O’Quin K., Fultz J., Vanderplas M., Isen A. M. (1983). Influence of self-reported distress and empathy on egoistic versus altruistic motivation to help. J. Person. Soc. Psychol. 45:706 10.1037/0022-3514.45.3.706 - DOI