Summary / Abstract

This project was completed as part of a research paper done during my Masters degree at University of Maryland (Go Retrievers!). It focuses on elicitation of emotions using auditory stimuli in a virtual reality setting. The task in the virtual reality (VR) setting is that of an archery game in a medium-high fidelity immersive forest environment. Through this research, my team's aim was to gain an understanding of the effect of auditory stimuli in VR on an individual’s emotional experience and attention as little is known about the role of auditory stimuli in evoking human emotions in VR settings. Various methods of analysis were deployed in both qualitative and quantitative methodologies, such as in-game video clip review with the participants, heart rate data collection, and Likert scale rating of their day along with their prior VR experience to gain insight into their pre, during, and, post experiment states. Our analysis pointed towards negligible similarities in the experimental and control groups, thus highlighting that visual focus during a task in a VR environment triumphs over any auditory stimuli provided to the player.

Roles & Responsibilities

Our team has three members in total who were tasked with separate responsibilities. I was leading the group to ensure timely delivery of project output as well as creating the VR video game (test) in Unreal Engine and facilitating the conduction of the experiment itself.

Introduction & Motivation

As an advanced multisensory display system, VR consists of multiple stimuli — the visual, tactile, and auditory — responsible for eliciting human emotions in the virtual environment. However, little is known about the role of auditory stimulus in evoking human emotions in VR settings.
Therefore, through this research, we aimed to gain an understanding of:
What is the effect of auditory stimulus in VR on a human’s emotional experience?
What is the effect of auditory stimulus in VR on a human’s attention?
What is the intensity of the emotions experienced?

Literature Review

As with any good research process, we started off with doing our due diligence with some healthy amount of literature review of past and current work in and around the field of VR as well as project-related aspects such productivity and comparison studies. The following were the broad research topics that were divided within our group:
(A) The extent and impact of immersive experiences as compared to 2D screen-based experiences
(B) The impact of immersive experiences in workplace productivity
(C) Considerations while designing immersive experiences
My focus during this phase was specifically on (C) due to the nature of my responsibilities in the project. My intentions going into the literature review phase was to understand the implications of not applying design and ethical considerations while creating immersive experiences and learn from any mistakes that have been made in previous studies in the field of VR along with following any best practices too!

The following are the research papers I went through in this phase pertaining to (C) as mentioned above:
i) Immersive Interactive Technologies for Positive Change: A Scoping Review and Design Considerations
ii) New Terrain in HCI: Emotion Recognition Interface using Biometric Data for an Immersive VR Experience
iii) Some Practical Considerations of Ethical Issues in VR Research

Virtual Reality Task

The experiment was conducted using a Vive Pro 2 while the player played an archery video game. The aim of the game was to hit the highest number of arrows onto a target presented at a distance of 100 unreal units(approximately equivalent to 100 centimeters). The overall environment was decided to be a wide-open forest with a pond at the back of the player’s view and a lone tree with the target at the front of the player’s view. Players were free to look around and let the environment “sink in” to increase their immersion in the game. The environment was created in such a manner to make the players feel at ease while playing the game and help them focus on the task at hand.
Additionally, the players had virtual hands, which try to emulate real hands, that were mapped onto each of their controllers along with the grabbing action being mapped to the back trigger buttons of each of the controllers.

One of the most fundamental aspects of the game mechanics was that the distance between the target and player would be fixed (all players had the same shooting start point) as well as the time constraint to be fixed at 90 seconds. A crucial point to be noted is that the timer for each player was started from the moment the player grabbed and pulled the bowstring. This was done to allow players 10-15 seconds each to look around at their surroundings. Furthermore, the physics of the game were tweaked to be close to real-life physics but with some additional twists. The velocity of the arrow being shot was kept constant i.e. no matter how far back the player pulls on the bowstring, the arrow would move at the same speed. This factor attempted to remove the possibility of varying accuracy due to physical strength differences in our participants.

Another notable game mechanism put in place was the auto-spawning of the arrows. As the player pulled on the bowstring, an arrow would automatically spawn, perfectly aligned and attached to the bow. No manual placement was required in this aspect. This mechanism was used to lower the learning curve and allow the player to focus solely on the angle selection aspect. Lastly, in terms of audio, the mechanism used as a stimulus was birds (crows in this case) cawing as the participant manages to hit the target. On each successful target hit, the sound would of the crows is triggered. This algorithm effectively resulted in an exponentially increasing sound as the player performed well. The sound of the crows was layered on top of the existing sounds of the rustling of the winds in the background.

The figures show the controls, and screenshots of In-Game scenarios

---

Research Methodology

To explore the effect of auditory stimuli in VR on human emotional experience, we designed an experimental study collecting the participants’ physiological responses to the stimuli in our virtual reality environment. We conducted a quantitative analysis looking at participants’ physiological parameter data and a qualitative analysis of their subjective emotional experiences. We hypothesized that participants’ physiological parameter data indicating arousal levels, would be higher as the effect of auditory stimuli in a VR environment.

I'm going to be briefly talking about the various aspects of the research methods used and the reasoning behind them. Additionally, to avoid walls of text, I'll try to summarize as much as possible.

1. Participants

We recruited 10 participants through email and referral from some of the participants, inviting them one by one to participate in a 30-minute study about “emotional experience” in virtual reality. In order to have a diverse sample, we also recruited participants from other majors in addition to Human-Centered Computing (HCC). Five participants are HCC students, two Information System students, two Data Science, and one Electrical Engineering student. The purpose was to obtain various perceptions regarding the virtual reality design from participants who we assume are not familiar with design principles.
Their experiences with the virtual reality environments are also varied, ranging from not comfortable to very comfortable. Each participant was randomly assigned to an experimental or control group. There were five participants in the experimental group and five participants in the control group. Each participant did not know which group they were assigned to (single-blind study).

2. Procedure

After participants completed the consent form, they were interviewed regarding their demographic background, today’s mood, and VR experience. We also used an Empatica E4 device to measure participants’ heart rate (HR), electro-dermal activity (EDA), blood volume pulse (BVP), and temperature during all the stages of the experiement (pre, during and post). The interview was semi-structured in that we can sometimes probe into several questions (e.g., frequency of having VR experience and perception of these experiences) we found the answers were interesting.
In terms of onboarding the subjects, we gave live demonstrations to them regarding the VR headset (HTC Vive Pro 2) as well as the basic controller and environment navigation.
During the VR experience, participants need to complete a time-constrained task (i.e., one and a half minutes) of shooting arrows as many as possible at a target (a dart) hanging on a distant tree in the virtual environment. At the same time, their heart rate was also collected using the Empatica E4.

For the experimental condition, the ambient sound of breeze and birds chirping are incorporated into the forest environment. Furthermore, every time the participants successfully shot the target (i.e. they scored), the crow’s sound would appear and it would be getting louder (intense) as the score increased. The crow's sound, in particular, was intended to evoke arousal in participants since we assume that intense auditory stimuli would affect participants' emotional experience.
For the control condition, no audio stimuli were presented in the forest virtual reality environment. In other words, the control condition would only present the visual stimuli (i.e., forest setting). By manipulating the presence of sounds in virtual environment settings between those two conditions, we want to examine whether the auditory stimuli would affect participants’ emotional experiences.

After participants had completed the virtual reality task, they were interviewed regarding their emotional experience while performing the task. The questions comprised their perceived emotion after completing the task, perception toward their performance, description of their experience while performing the task, and how stimulated (i.e., agitated as compared to relaxed) they were while performing the task. During this interview, we also walked them through the in-game video recording (recorded screen of the virtual reality task) and asked them to talk aloud about whatever they thought and felt during the VR experience. The purpose of this activity was to highlight any particular moment that was deemed significant to the participants’ emotional experience.

There's me with a test subject questioning and listening to the walk-through of their experiece.

---

3. Analysis

We analyzed the responses to the pre and post-task questionnaire, from physiological parameters, and in-game video recording. We decided to use participants’ heart rate (HR) data as a physiological parameter to identify their emotional arousal. An increase in heart rate (the number of contractions of the heart per minute (beats per minute/ bpm)) indicates the initiation of a physiological stress response, defined as a coping response to emotional arousal, external and internal stimuli, and a state of heightened psychological activity (Wascher, 2021).
Previous research also revealed that most of the negative emotions elicited by various means will result in increased HR (Kreibig in Yan et al., 2019). In other words, increased heart rate can be an indication of response to emotional experiences, such as arousal.
In addition, we also identified the time when participants reached their highest heart rate and related it to a specific event captured in the in-game video recording. Therefore, we identified which event evoked such high arousal in each participant. During the writing process of the paper, we related our qualitative data from interviews and in-game video observation to quantitative data obtained from participants’ heart rates. We also refined the themes from these findings around our main research questions.

Results & Conclusion

The results we gathered from the heart-rate measurement and post-task interview did not show a significant effect of auditory stimuli on a participant’s emotional response. Even though there was a slight difference in the physiological responses between the experimental and control group, it could not be directly attributed to the auditory stimuli as we only focussed on heart rate (HR) values in a silo, without correlating them with other physiological factors like the electrodermal activity (EDA). Furthermore, the post-task interview regarding the participants’ emotional state of being relaxed or agitated also did not differ between the two groups.

Our study had some limitations which could be improved for much richer data in the future. For instance, the crow sound only begins after the first successful shot at the target and increases exponentially as the target is shot. This scenario depends on the accuracy of the participant and may not be heard equally by all the participants of the experiment group. Instead, we could have the crow sound start after a certain time delay on starting the game so that all participants can experience the sound regardless of successfully shooting the target. The second limitation was the timeline for the study itself, making the inclusion of a dry-run for some VR experience for participants to practice. Since a few participants did mention that if they had more time in discovering the controls in the VR environment, they would have been more confident in shooting the arrows correctly. The third limitation was not conducting a thorough pilot study before the actual experiment. This would have helped us iterate and polish the interview questions and the rating scales that we finally used to rate the participant's emotions. A 5-point Likert scale was not a good choice to understand the relaxed or agitated emotional state of participants. Perhaps a 10-point scale would have allowed the participants to freely express their preference for an emotional state while leaning toward a more dominant emotion.

Potential future studies in the same domain can yield more evidence if the participant pool is larger.