Cost-Effective Virtual Reality for Smarter Architecture: Predicting How We Hear Spaces

Angela Guastamacchia – angela.guastamacchia@polito.it
Department of Energy, Politecnico di Torino
Torino, Torino 10129
Italy

Popular version of 3aAAb4 – Subjective and objective validation of a virtual reality system as a tool for studying speech intelligibility in architectural spaces
Presented at the 188th ASA Meeting
Read the abstract at https://eppro01.ativ.me//web/index.php?page=Session&project=ASAICA25&id=3869566

–The research described in this Acoustics Lay Language Paper may not have yet been peer reviewed–

When we communicate, clear speech is crucial—it helps us exchange ideas, learn, and build human connections. But often, poor acoustic conditions in rooms like crowded restaurants, wide lecture halls, or meeting spaces can make it difficult to understand speech clearly. Indoor architectural design significantly impacts speech clarity, so studying how different spaces affect communication, especially when hearing-impaired people are involved, is essential for fostering optimal designs that facilitate effective communication.

Virtual Reality (VR) might provide a practical and time-saving solution for this research, allowing us to reproduce various architectural environments and study how people perceive speech within those spaces without needing access to the real environments. Some laboratories have already implemented systems to accurately reproduce acoustics targeting diverse research goals. However, these systems typically rely on complex and costly arrays of dozens of loudspeakers, making studies difficult to set up, expensive, and inaccessible for architectural designers who are not VR experts.

Thus, a question arises: can even a less complex VR system still replicate a realistic experience of listening to speech in an actual room?

At the Audio Space Lab of the Politecnico di Torino, we set up a simpler and more affordable VR system. This system combines a VR headset with a spherical array of 16 loudspeakers to create immersive and realistic audiovisual communication scenarios surrounding the listener in a 360° experience, using an audio technique called 3rd-Order Ambisonics. We then tested whether our VR setup could consistently replicate the experience of listening in a medium-sized, echoey lecture room.

To test this, we compared the speech understanding of thirteen volunteers in the real lecture hall and in its virtual replica. During the tests, volunteers listened to single sentences and repeated what they understood across five different audiovisual scenes, varying the speech source location and the presence or absence of distracting noise. All scenarios included typical background noise, such as the hum of air conditioning, to closely mimic real-life conditions.

In Figure 1, you can see a volunteer in the real lecture room listening to sentences emitted by the loudspeaker positioned to their right, while a distracting noise is presented from the frontal loudspeaker. In Video 1, a volunteer performs the same speech test within the VR system, replicating the exact audiovisual scene shown in Figure 1. Figure 2 shows what the volunteer saw during the test.

Virtual Reality

Figure 1. Volunteer performing the speech comprehension test in the real lecture room.

Video 1. Volunteer performing the speech comprehension test in the virtual lecture room using the VR system.

Virtual Reality

Figure 2. Volunteers’ view during both real and virtual speech comprehension tests.

Our findings are promising: we found no significant differences in speech comprehension between the real and virtual settings across all tested scenes.

Additionally, we asked the volunteers how closely their VR experience matched reality. On average, they rated it as “almost very consistent,” reinforcing that the VR system provided a believable acoustic experience.

These results are exciting because they suggest that even with a less complex VR system, real-life-like speech perception in ordinary environments can be effectively predicted. Our affordable and user-friendly VR system could thus become a powerful tool for architects, acousticians, and researchers, offering an accessible way to easily study speech comprehension in architectural spaces and pursue improved acoustic designs.

Eyes as a window to listening effort with virtual reality

Kristina DeRoy Milvae – klmilvae@buffalo.edu

Department of Communicative Disorders and Sciences, University at Buffalo, Buffalo, NY, 14214, United States

Additional authors: Ian Phillips, Mythili Thamilchelvam, Shifali Chambers, Uzaira Sethi, Jacob Lefler, Stefanie E. Kuchinsky, Douglas S. Brungart

Popular version of 2pPP4 – Virtual reality potential as a platform to measure listening effort
Presented at the 188th ASA Meeting
Read the abstract at https://eppro01.ativ.me/web/index.php?page=Session&project=ASAICA25&id=3868338

–The research described in this Acoustics Lay Language Paper may not have yet been peer reviewed–

Most people take their ability to understand speech for granted until they have a hearing loss. However, approximately 25 million Americans with clinically normal hearing still report experiencing hearing difficulties substantial enough to negatively impact their daily lives. The extra mental effort that these individuals exert during listening appears to come at the cost of reduced ability to participate in social interactions, greater fatigue, and poorer performance on other daily life functions, such as memory for speech. There is currently no standard objective test that quantifies the increased listening effort that these individuals experience, and we are exploring possibilities for this with our research.

Changes in the pupil size of the eye during difficult listening can be used to measure listening effort. Typically, this measurement requires listeners to keep their heads still while a specialized, expensive infrared camera system tracks changes in the listener’s pupils while they listen.  In recent years, virtual reality headsets have emerged as a possible alternative to these laboratory-based eye-tracking systems.  Virtual reality headsets were also originally developed as research tools that were primarily used in the laboratory, but they are now widely marketed for entertainment and gaming applications.  The most advanced virtual reality headsets now incorporate eye-tracking technology capable of making continuous measurements of eye gaze direction and pupil size.  Virtual reality headsets are less expensive than laboratory eye-tracking systems, they are more portable, and they require fewer movement restrictions during the eye measurements.  All of these factors could make virtual reality headsets more appealing than traditional eye trackers for clinical applications.  However, it is not known how comparable listening effort measurements are across these research-grade and virtual-reality eye-tracking systems.

The purpose of our study was to directly compare the ability of research and virtual reality systems to monitor changes in listening effort via changes in pupil dilation.  Participants with normal hearing listened to and recalled digits played in the right ear.  Sometimes they had to ignore distracting digits that were also played in the left ear.  As in our previous studies, we found that the addition of the distracting digits caused a systematic increase in the pupil size measured by a laboratory-grade eye tracker, presumably because it required an increase in listening effort.  We also found that the same effort-related increase in pupil size could be measured with a low-cost virtual-reality headset, opening up the possibility that these systems could be used to conduct objective measures of listening effort in clinical environments.

Our research also revealed important potential limitations to consider with using virtual reality headsets to measure listening effort in a clinical setting. These include subtle quality differences in the measurements, rapidly changing technology, and the programming requirements to set up these systems for this application. Our ongoing work aims to address these concerns as we see potential for clinical use of virtual reality systems for listening effort applications in audiology. The ability to validate patient complaints of hearing difficulties with objective clinical metrics will allow for the development interventions to mitigate listening effort and ultimately improve the quality of life for the millions of people experiencing hearing difficulties.

Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the official policy of the Department of Defense or U.S. Government.

Tools for shaping the sound of the future city in virtual reality

Christian Dreier – cdr@akustik.rwth-aachen.de

Institute for Hearing Technology and Acoustics
RWTH Aachen University
Aachen, Northrhine-Westfalia 52064
Germany

– Christian Dreier (lead author, LinkedIn: Christian Dreier)
– Rouben Rehman
– Josep Llorca-Bofí (LinkedIn: Josep Llorca Bofí, X: @Josepllorcabofi, Instagram: @josep.llorca.bofi)
– Jonas Heck (LinkedIn: Jonas Heck)
– Michael Vorländer (LinkedIn: Michael Vorländer)

Popular version of 3aAAb9 – Perceptual study on combined real-time traffic sound auralization and visualization
Presented at the 186th ASA Meeting
Read the abstract at https://doi.org/10.1121/10.0027232

–The research described in this Acoustics Lay Language Paper may not have yet been peer reviewed–

“One man’s noise is another man’s signal”. This famous quote by Edward Ng from a 1990’s New York Times article breaks down a major learning from noise research. A rule of thumb within noise research states the community response to noise, when asked for “annoyance” ratings, is said to be statistically explained only to one third by acoustic factors (like the well-known A-weighted sound pressure level, which can be found on household devices as “dB(A)” information). Referring to Ng’s quote, another third is explained by non-acoustic, personal or social variables, whereas the last third cannot be explained according to the current state of research.

Noise reduction in built urban environments is an important goal for urban planners, as noise is not only a cause of cardio-vascular diseases, but also affects learning and work performance in schools and offices. To achieve this goal, a number of solutions are available, ranging from switching to electrified public transport, speed limits, traffic flow management or masking of annoyant noise by pleasant noise, for example fountains.

In our research, we develop a tool for making the sound of virtual urban scenery audible and visible. From its visual appearance, the result is comparable to a computer game, with the difference that the acoustic simulation is physics-based, a technique that is called auralization. The research software “Virtual Acoustics” simulates the entire physical “history” of a sound wave for producing an audible scene. Therefore, the sonic characteristics of traffic sound sources (cars, motorcycles, aircraft) are modeled, the sound wave’s interaction with different materials at building and ground surfaces are calculated, and human hearing is considered.

You might have recognized a lightning strike sounding dull when being far away and bright when being close, respectively. The same applies for aircraft sound too. In an according study, we auralized the sound of an aircraft for different weather conditions. A 360° video compares how the same aircraft typically sounds during summer, autumn and winter when the acoustical changes due to the weather conditions are considered (use headphones for full experience!)

In another work we prepared a freely available project template for using Virtual Acoustics. Therefore, we acoustically and graphically modeled the IHTApark, that is located next to the Institute for Hearing Technology and Acoustics (IHTA): https://www.openstreetmap.org/#map=18/50.78070/6.06680.

In our latest experiment, we focused on the perception of especially annoyant traffic sound events. Therefore, we presented the traffic situations by using virtual reality headsets and asked the participants to assess them. How (un)pleasant would be the drone for you during a walk in the IHTApark?

A virtual reality system to ‘test drive’ hearing aids in real-world settings

Matthew Neal – mathew.neal.2@louisville.edu
Instagram: @matthewneal32

Department of Otolaryngology and other Communicative Disorders
University of Louisville
Louisville, Kentucky 40208
United States

Popular version of 3pID2 – A hearing aid “test drive”: Using virtual acoustics to accurately demonstrate hearing aid performance in realistic environments
Presented at the 184 ASA Meeting
Read the abstract at https://doi.org/10.1121/10.0018736

Many of the struggles experienced by patients and audiologists during the hearing aid fitting process stem from a simple difficulty: it is really hard to describe in words how something will sound, especially if you have never heard it before. Currently, audiologists use brochures and their own words to counsel a patient during the hearing aid purchase process, but a device often must be purchased first before patients can try them in their everyday life. This research project has developed virtual reality (VR) hearing aid demonstration software which allows patients to listen to what hearing aids will sound like in real-world settings, such as noisy restaurants, churches, and the places where they need devices the most. Using the system, patient can make more informed purchasing decisions and audiologists can program hearing aids to an individual’s needs and preferences more quickly.

This technology can also be thought of as a VR ‘test drive’ of wearing hearing aids, letting audiologists act as tour guides as patients try out features on a hearing aid. After turning a new hearing aid feature on, a patient will hear the devices update in a split second, and the audiologist can ask, “Was it better before or after the adjustment?” On top of getting device settings correct, hearing aid purchasers must also decide which ‘technology level’ they would like to purchase. Patients are given an option between three to four technology levels, ranging from basic to premium, with an added cost of around $1,000 per increase in level. Higher technology levels incorporate the latest processing algorithms, but patients must decide if they are worth the price, often without the ability to hear the difference. The VR hearing aid demonstration lets patients try out these different levels of technology, hear the benefits of premium devices, and decide if the increase in speech intelligibility or listening comfort is worth the added cost.

A patient using the demo first puts on a custom pair of wired hearing aids. These hearing aids are the same devices sold that are sold in audiology clinics, but their microphones have been removed and replaced with wires for inputs. The wires are connected back to the VR program running on a computer which simulates the audio in a given scene. For example, in the VR restaurant scene shown in Video 1, the software maps audio in a complex, noisy restaurant to the hearing aid microphones while worn by a patient. The wires send the audio that would have been picked up in the simulated restaurant to the custom hearing aids, and they process and amplify the sound just as they would in that setting. All of the audio is updated in real-time so that a listener can rotate their head, just as they might do in the real world. Currently, the system is being further developed, and it is planned to be implemented in audiology clinics as an advanced hearing aid fitting and patient counseling tool.

Video 1: The VR software being used to demonstrate the Speech in Loud Noise program on a Phonak Audeo Paradise hearing aid. The audio in this video is the directly recorded output of the hearing aid, overlaid with a video of the VR system in operation. When the hearing aid is switched to the Speech in Loud noise program on the phone app, it becomes much easier and more comfortable to listen to the frontal talker, highlighting the benefits of this feature in a premium hearing aid.

Virtual Reality Musical Instruments for the 21st Century

Rob Hamilton – hamilr4@rpi.edu
Twitter: @robertkhamilton

Rensselaer Polytechnic Institute, 110 8th St, Troy, New York, 12180, United States

Popular version of 1aCA3 – Real-time musical performance across and within extended reality environments
Presented at the 184 ASA Meeting
Read the abstract at https://doi.org/10.1121/10.0018060

Have you ever wanted to just wave your hands to be able to make beautiful music? Sad your epic air-guitar skills don’t translate into pop/rock super stardom? Given the speed and accessibility of modern computers, it may come as little surprise that artists and researchers have been looking to virtual and augmented reality to build the next generation of musical instruments. Borrowing heavily from video game design, a new generation of digital luthiers is already exploring new techniques to bring the joys and wonders of live musical performance into the 21st Century.

Image courtesy of Rob Hamilton.

One such instrument is ‘Coretet’: a virtual reality bowed string instrument that can be reshaped by the user into familiar forms such as a violin, viola, cello or double bass. While wearing a virtual reality headset such as Meta’s Oculus Quest 2, performers bow and pluck the instrument in familiar ways, albeit without any physical interaction with strings or wood. Sound is generated in Coretet using a computer model of a bowed or plucked string called a ‘physical model’ driven by the motion of a performer’s hands and the use of their VR game controllers. And borrowing from multiplayer online games, Coretet performers can join a shared network server and perform music together.

Our understanding of music, and live musical performance on traditional physical instruments is tightly coupled to time, specifically the understanding that when a finger plucks a string, or a stick strikes a drum head, a sound will be generated immediately, without any delay or latency. And while modern computers are capable of streaming large amounts of data at the speed of light – significantly faster than the speed of sound – bottlenecks in the CPUs or GPUs themselves, or in the code designed to mimic our physical interactions with instruments, or even in the network connections that connect users and computers alike, often introduce latency, making virtual performances feel sluggish or awkward.

This research focuses on some common causes for this kind of latency and looks at ways that musicians and instrument designers can work around or mitigate these latencies both technically and artistically.

Coretet overview video: Video courtesy of Rob Hamilton.