Background
Studies have shown that music is an integral part of filmmaking and can have a profound effect on the audience. However, only until relatively recent times has more intensive research been conducted. One example of this was an experiment by Oliver Vitouch titled “When Your Ear Sets the Stage: Musical Context Effects in Film Perception” (summarized here). The Vitouch experiment demonstrated that the audience’s perception of a film and its further development could be systematically influenced by the music. The Vitouch experiment was the inspiration for this research project.
Aim
The purpose of this experiment was to investigate what influences the emotional perception of a video, but more so, whether the altering of audio or visual cues had a greater effect.
Method
Four different video clips were made from a sequence of color and grayscale pictures taken from the 2004 Tsunami and combined with two different soundtracks: (1) color with a Hip Hop track, (2) grayscale with a Hip Hop track, (3) color with a piano track and finally, (4) grayscale with a piano track.
The first song used was the instrumental version of “Locked Up” by Akon and the second song was an acoustic piano version of “Numb” by Linkin Park. Instrumentals were selected so that respondents would be less likely to recognize the songs. Sad songs were intentionally selected to enhance the sad emotions felt by viewing the images alone. Hip Hop is not normally associated with sadness however this song was appropriate for the images as the song was tense, dark and gloomy. The piano song was selected due to its slow tempo and high legato which made it much more different than the Hip Hop track.
Grayscale images typically evoke sadness because as opposed to vibrant colors, which are more intense, lively and exciting, grayscale images are more associated with somberness. Due to this, grayscale images should increase the feeling of sadness when looking at the pictures.
The four different video clips were presented to subjects in the following order: (1), (3), (4) and (2). This order was selected so that the purpose of the experiment would be less clear to the subjects. In order to increase the number of respondents, the videos were put on a website (http://the1truecoolguy.blogspot.com/2006/05/experiment-audio-video-and-emotions.html) with instructions on how to participate. A limitation of this was that the environment (lighting, distractions, sound levels, etc.) could not be controlled as if they were in a laboratory.
The subjects were asked to rate their sadness level for each video on a scale from 0 to 10 in an anonymous survey. The survey also included questions about the participant’s level of musical training, if the participant recognized the songs in the videos and a final open-ended comment field. A total of 62 people participated in the experiment.
Hypothesis
The primary hypothesis was that the video with the grayscale pictures together with the piano song (video 4), would be perceived as the saddest one of the four and therefore would score higher than the others on the sadness scale. The secondary hypothesis was that the video the color pictures accompanied by the Hip Hop track (video 1), would be perceived as the least sad one.
Results
Average Sadness Scores (in descending order)
5.27 *** Video 3 (Piano / Grayscale)
5.11 *** Video 2 (Piano / Color)
4.31 *** Video 4 (Hip Hop / Grayscale)
3.76 *** Video 1 (Hip Hop / Color)
Sadness Level Charts
(Click to Enlarge)
Conclusions
As shown in the graphs, most people felt “Sort of Sad” (scores of 5 or 6) and only 7 of the 62 respondents felt very high levels of sadness (scores of 9 or 10) when watching the videos.
We see that the third video, which was the slow piano song combined with the grayscale images was indeed perceived as the saddest. A respondent mentioned that due to the lack of other instruments in the piano track, those videos felt more “lonely” and thus were sadder. Although a few people commented that color made the pictures appear more ‘real’, the results show that the majority of them believed that grayscale felt sadder. It was also mentioned that grayscale allowed the viewer to focus more on the details of the images, which would make them sadder, rather than being distracted by the colors.
By contrast, the least sad was the complete opposite, which was the Hip Hop track combined with color imagery. This video did not score any 8, 9 or 10s at all. It’s worthy to note that both videos which used the Hip Hop track received 7 “Not at All Sad” ratings. This would indicate (as predicted) that Hip Hop typically doesn’t evoke sadness. Comments were that Hip Hop made the scenes appear more “cool” and “upbeat” rather than sad. On the other hand, many people mentioned that Hip Hop worked well with the grayscale images and perhaps that would explain why the final video (although still Hip Hop) achieved significantly higher sadness levels than its color counterpart.
As expected, the variation in music had a much stronger impact than the visual changes.
Future Research
As mentioned earlier, a major limitation of this experiment was the lack of a controlled environment. Another limitation was that although most respondents (27) didn’t recognize either song, many did and this certainly affected the results. Most of the respondents (39) were musically trained and perhaps they were aware of the emotional cues being used. Finally, the music selected
(Hip Hop vs. acoustic piano) was a drastic and obvious difference, which could be an issue.
A similar experiment could be conducted in a controlled environment, with musically untrained subjects, with music they wouldn’t recognize and songs that are not as different (e.g. 2 acoustic piano songs of different tempos/modes/volumes/timbres/etc.). It would also be interesting to record demographics of the subjects such as age and sex.
Another interesting point would be to ask the respondents about perceived vs. felt emotions (these can be 2 separate experiments). For instance, did the grayscale actually make them feel sad or did they perceive that they should feel sad because grayscale is traditionally used as a visual cue for sadness.
Overall, this experiment demonstrated that altering audio cues (the soundtrack) has much more impact on the audience than simple visual cues (color filters) if the same images are used. However, it would be interesting to try and further isolate (with more controlled experiments) which cues specifically affect our emotions when watching films.