Popular version of paper 1pPPa, “Trends that are shaping the future of hearing aid technology” Presented Monday afternoon, May 7, 2018, 1:00PM, Nicollet D2 Room 175th ASA Meeting, Minneapolis
Hearing aid technology is experiencing a faster rate of change than it has in the history of its existence. A primary reason for this is its convergence with consumer electronics, resulting in an acceleration of the pace of innovation and a change in its nature from incremental to disruptive.
Hearable and wearable technology are non-medical devices that use sensors to measure and inform the user about their biometric data in addition to providing other sensory information. Since hearing aids are worn every day and the ear is an ideal location to place many of these sensors, hearing aids have the potential to become the ideal form factor for consumer wearables. Conversely, hearable devices that augment and enhance audio for normal hearing consumers while also measuring their biometric data have the potential to become a new form of hearing aids for those with hearing loss, combining medical functionality of hearing loss compensation with such consumer functionality as speech recognition with always-on access to Siri. The photo below shows one hearable on the market that allows the wearer to measure their hearing with a smartphone app and adjust the audibility of the hearing to personalise the sound for the individual’s hearing ability, a process that has similarities to the fitting of a traditional hearing aid by an audiologist.
Accelerating this convergence between medical and consumer hearing technologies is the recently passed congressional bill that mandates the creation of a new over-the-counter hearing aid that consumers can purchase in a store and fit their own prescription. E-health technologies already exist that allow a consumer to measure their own hearing loss and apply clinically-validated prescriptions to their hearable devices. This technology development will explode once over-the-counter hearing aids are a reality.
Deep science is also impacting hearing aid innovation. The integration of cognitive function with hearing aid technology will continue to be one of the strongest trends in the field. Neural measures of the brain using EEG have the potential to be used to fit hearing devices and also to demonstrate hearing aid benefit by showing how wearing devices affects activity in the brain. Brain sensors have been proven able to determine which talker a person is listening to, a capability that could be included in future hearing aids to enhance the speech from the desired talker and suppress all other sounds. Finally, science continues to advance our understanding of how hearing aid technology can benefit cognitive function. These scientific and other medical developments such as light-driven hearing aids will advance hearing aid benefit through the more traditional medical channel, complementing the advances on the consumer side of the healthcare delivery spectrum.
Lay-language paper for abstract 3aUW8 “Representation trade-offs for the quality assessment of acoustic color signatures” presented at the 175th Meeting of Acoustical Society of America in Minneapolis.
We use sound to learn about the underwater environment because sound waves travel much better in water than light waves do. Similar to using a flashlight to find your lost car keys in the woods, sound wave pulses are used to ‘light up’ the sea floor. When carefully organized, sound echoes from the surroundings can be shown as sonar imagery such as Figure 1.
Figure 1. A sonar image is generated from a collection of sound recordings by carefully organizing them into a spatial representation, and we can see various features of the sea floor and even shadows cast by sea floor textures and objects, similar to when using a flash light.
Images are easy for us to understand, but not all of the useful information embedded in the sound recordings is represented well by images. For example, a plastic and a metallic trash bin may have the same cylindrical shape, but the sounds they make when you knock on them are easy to distinguish. This idea leads to a different method of organizing a sound recording, and the resulting representation is called acoustic color, Figure 2. This shows how different frequencies emerge and fade as you ‘knock’ on the object with sound from different directions.
Figure 2. Acoustic color of a solid aluminum cylinder. It shows the strength of frequency components when seen from different viewing angle. Source, University of Washington PONDEX 09/10
This representation has the potential to be useful for distinguishing objects that have similar shapes in visual imagery, but have noticeably different acoustic spectral responses. However, it is not easy to extract relevant information that can help discriminate different objects as seen in Figure 2. One of the reasons is the weak and dispersed nature of the object signatures, which makes it difficult to mentally organize them and draw conclusions. We want to explore other ways of organizing the acoustic data in order to make it intuitive for us to ‘see’ what is yet to be uncovered from the environment. Certain animals, such as dolphins and bats are able to take advantage of complicated acoustic echoes to hunt for prey and understand their environment.
One representation under consideration is time-varying acoustic color, Video 1, which provides the ability to observe the time-evolving characteristics of the acoustic color, with some loss in the ability to precisely distinguish frequencies. This helps one understand how different spectral signatures appear and change, and eventually fade out. This short timescale evolution is important information not easily extractable in the typical acoustic color representation.
<Video 1 missing>
Another representation under consideration is an approach called symbolic time series analysis. By representing a short segment of the raw time series as a symbol and assigning the same symbol for segments that are similar, the time series is transformed into a sequence of symbols as illustrated in Figure 3. This allows us to use tools developed for sequence analysis such as the ones used in DNA sequencing for comparing and recognizing patterns in the sequence. It may prove to be an effective approach to extracting underlying patterns from acoustic data that are not as easily accessible in other more common modes of visualizing the data.
Figure 3. Each time series is transformed into a sequence of symbols, then can be further analyzed to characterize temporal patterns. We can use tools developed for other applications such as DNA sequencing, and extract information that are not as easily accessible from more common modes of visualizing the data.
Popular version of paper 3pPA5 Presented Wednesday afternoon, May 09, 2018 175th ASA Meeting, Minneapolis
Hearing loss affects 10% of the global population to some degree but only 20% of sufferers receive treatment1,2. Hearing aids are the most common treatments for hearing loss, with longer battery life and improved ease of use identified as the most desirable advances that will improve acceptance3,4,5. Our research addresses both these issues.
Modern hearing aids have dramatically shrunk in size over the years. This is a positive development since a small hearing aid is less apparent and more comfortable than has been the case in the past. However, with smaller size has come new problems. Controls for modern hearing aids are now much harder to place on the actual device and smaller controls have become increasingly difficult to use, especially for those with low dexterity. Small switches and additional accessories have been the main ways to interact with hearing aids, with increasing adoption of Bluetooth Low Energy (BLE) for connections with smart phones.
The use of BLE and other radio frequency technologies requires additional hardware within the hearing aid, which increases both its price and power consumption. Our work addresses this problem by using high frequency sound waves and ultrasound to communicate between a smart phone and hearing aid (Figure 1). Using hardware already present on the hearing aid allows our technology to be implemented on both old and new hearing aids without any additional hardware costs.
Figure 1 – An illustration of acoustic wireless communication between a smart phone and a hearing aid.
Our work investigated the performance of multiple communication techniques operating at frequencies within the inaudible range of 16 to 24 kHz. To reduce power consumption, the highly efficient audio processing capabilities of the hearing aid were used alongside simple manipulations of the audio signal. These simple manipulations modulate the amplitude and frequency of the sound waves to transmit binary data.
We were able to transmit 48 bits of data over a range of 3 metres while consuming less power than BLE. While 48 bits of data is relatively small compared to data sent via radio frequency transmissions, it represents multiple commands for the remote operation of two hearing aids. These commands can be used to adjust the volume as well as change program settings for different listening scenarios.
There are benefits to using sound waves as a communication channel for other body worn devices apart from hearing aids. The limited transmission range of high frequency audio provides security through proximity as any potential attacker must be within close range and line of sight to conduct an attack. The prevalence of audio technology in personal electronic devices also has the potential for a universal communication medium across varying platforms.
As hardware on both the transmitting and receiving sides of the acoustic channel continues to develop for the core purpose of each technology, acoustic wireless communication will continue to improve as an option for controlling hearing aid technology and other body worn devices.
References 1 N. Oishi and J. Schacht, “Emerging treatments for noise-induced hearing loss,” Expert Opin. Emerg. Dr. 16, 235-245 (2011).
2 Hartley, D., E. Rochtchina, P. Newall, M. Golding, and P. Mitchell, “Use of hearing aids and assistive listening devices in an older Australian population” Journal of the American Academy of Audiology, 21, 642-653 (2010).
3 S. Kochkin, “MarkeTrak VIII: Consumer satisfaction with hearing aids is slowly increasing,” The Hearing Journal 63, 19-20 (2010).
4 S. Kochkin, “MarkeTrak VIII Mini-BTEs tap new market, users more satisfied,” The Hearing Journal 64, 17-18 (2011).
5 S. Kochkin, “MarkeTrak VIII: The key influencing factors in hearing aid purchase intent,” Hearing Review 19, 12-25 (2012).
Zamir Ben-Hur – zami@post.bgu.ac.il Boaz Rafaely – br@bgu.ac.il Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 84105, Israel.
David Lou Alon – davidalon@fb.com Ravish Mehra – ravish.mehra@oculus.com Oculus & Facebook, 1 Hacker Way, Menlo Park, CA 94025, USA.
Popular version of paper 3aAA10, “Localization and externalization in binaural reproduction with sparse HRTF measurement grids”. Presented Wednesday morning, May 9, 2018, 11:40-11:55 AM, 175th ASA Meeting, Minneapolis.
High-quality spatial sound reproduction is important for many applications of virtual and augmented reality. Spatial audio gives the listener the sensation that sound arrives from the surrounding 3D space, leading to immersive virtual soundscapes. To create such a virtual sound scene with headphone listening, binaural reproduction technique is being used. A key component in binaural reproduction is the head-related transfer function (HRTF). An HRTF is a mathematical representation that describes how a listener’s head, ears, and torso affect the acoustic path originating from sound source’s direction into the ear canal [1]. HRTF set is typically measured for an individual in an anechoic chamber using an HRTF measurement system. Alternatively, a generic HRTF set is measured using a manikin. To achieve a realistic spatial audio experience, in terms of sound localization and externalization, high resolution personalized HRTF (P-HRTF) is necessary. Localization refers to the ability of presenting a sound at accurate locations in the 3D space. Externalization is the ability to perceive the virtual sound as coming from outside of the head, like real world environments.
Typical P-HRTF set is composed of several hundreds to thousands of source directions measured around a listener, using a procedure which requires expensive and specialized equipment and can take a long time to complete. This motivates the development of methods that require fewer spatial samples but still allow accurate reconstruction of the P-HRTF sets with high spatial resolution. Given only sparsely measured P-HRTF, it will be necessary to reconstruct directions that were not measured, which introduces interpolation error that may lead to poor spatial audio reproduction [2]. It is therefore important to better understand this interpolation error and its effect on spatial perception. If the error is too significant then a generic HRTF may be the preferred option over a sparse P-HRTF. Figure 1 presents an illustration of the question being answered in this study.
Figure 1. Illustration of the challenge of this paper.
Prior studies suggested to represent the HRTF in the spherical-harmonics (SH) domain. Using SH decomposition, it is possible to reconstruct high resolution P-HRTF from a low number of measurements [3,4]. When using SH representation, the reconstruction error can be caused by spatial aliasing and/or of SH series truncation [4,5,6]. Aliasing refer to loss of ability to represent high frequencies due to limited number of measurements. Truncation error refer to the order limit imposed on the SH representation which further limits the spatial resolution. With small number of measurements, both errors contribute to the overall reconstruction error.
In this study, the effect of sparse measurement grids on the reproduced binaural signal is perceptually evaluated through virtual localization and externalization tests under varying conditions.
Six adult subjects participated in the experiment. The experiment was performed with the Oculus Rift headset with a pair of floating earphones (see Fig. 2). These floating earphones enabled per-user headphone equalization for the study. A stimulus of 10 second band-passed filtered white noise (0.5-15 kHz) was played-back using real-time binaural reproduction system. The system allows reproduction of a virtual sound source in a given direction, using a specific HRTF set that was chosen according to the test condition. At each trial, the sound was played from a different direction, and the subject was instructed to point to this direction using a virtual laser pointer controlled by the subject’s head movement. Next, the participant was asked to report whether the stimulus was externalized or internalized.
Figure 2. The experiment setup, including a Rift headset and floating earphones.
We analyzed the localization results by means of angular errors. The angular errors were calculated as the difference between the perceptually localized position and the true target position. Figure 3 depicts the mean sound localization performance for different test conditions (Q, N), where Q is the number of measurements and N is the SH order. The figure shows averaged error across all directions (upper plot) and errors in azimuth and elevation (lower plots) separately. The externalization results were analyzed as average percentage of responses that the subjects marked as being externalized. Figure 4 shows the externalization results averaged across all directions and subjects.
The results demonstrate that high number of measurements leads to better localization and externalization performances, where most of the effect is in the elevation angles. Compared to the performance of a generic HRTF, P-HRTF with 121 measurements and SH order 10 achieves similar results. The results suggest that for achieving improved localization and externalization performance compare to a generic HRTF, at least 169 directional measurements are required.
Figure 3. Localization results of angular error (in degrees) for different conditions of (Q,N), where Q is the number of measurements and N is the SH order. Upper plot show the overall angular error, and lower plots show separate errors for azimuth and elevation.
Figure 4. Results of externalization performance.
References
[1] J. Blauert, “Spatial hearing: the psychophysics of human sound localization”. MIT press, 1997.
[2] P. Runkle, M. Blommer, and G. Wakefield, “A comparison of head related transfer function interpolation methods,” in Applications of Signal Processing to Audio and Acoustics, 1995., IEEE ASSP Workshop on. IEEE, 1995, pp. 88–91.
[3] M.J.Evans,J.A.Angus,andA.I.Tew,“Analyzing head-related transfer function measurements using surface spherical harmonics,” The Journal of the Acoustical Society of America, vol. 104, no. 4, pp. 2400–2411, 1998.
[4] G. D. Romigh, D. S. Brungart, R. M. Stern, and B. D. Simpson, “Efficient real spherical harmonic representation of head-related transfer functions,” IEEE Journal of Selected Topics in Signal Processing, vol. 9, no. 5, pp. 921–930, 2015.
[5] B. Rafaely, B. Weiss, and E. Bachmat, “Spatial aliasing in spherical microphone arrays,” IEEE Transactions on Signal Processing, vol. 55, no. 3, pp. 1003–1010, 2007.
[6] A. Avni, J. Ahrens, M. Geier, S. Spors, H. Wierstorf, and B. Rafaely, “Spatial perception of sound fields recorded by spherical microphone arrays with varying spatial resolution,” The Journal of the Acoustical Society of America, vol. 133, no. 5, pp. 2711–2721, 2013.
Elisa Konofagou, Ph.D. – ek2191@columbia.edu Department of Biomedical Engineering Columbia University 351 Engineering Terrace, mail code 8904 1210 Amsterdam Avenue New York, NY 10027
Popular version of paper 1pBA6 Presented Monday morning, May 7, 2018 175th ASA Meeting, Minneapolis, MN
Stimulation of the brain has been a topic of curiosity of humans since the beginning of time. Being able to selectively stimulate the brain to enhance performance such as think deeper and remember faster remains, a formidable challenge. Mapping the circuitry of the entire healthy human brain remains an equally unattainable goal. Brain mapping entails the study of biological functions of different regions in the brain. Although many regions of the brain have already been identified, there is very little known as to how the different regions communicate and whether activation patterns observed during specific behaviors are causally related to those behaviors. Such a brain map would not only further the understanding of the brain itself, but also potentially lead to novel cures or treatments for neurological conditions. One way to aid the progress in brain mapping is neurostimulation, a technique used to stimulate or activate neurons in the brain, usually by means of an electrode. When the electrode delivers a stimulus pulse to a targeted brain region, the biological response associated with that area will occur. Ultrasound has been consistently reported for neuronal stimulation for several decades in both animals and humans including eliciting brain activity detected by functional MRI and electroencephalography. In addition, this knowledge can be used to understand the differences between normal and pathological brains to treat patients.
In the peripheral nervous system, ultrasound has been reported since 1929 to stimulate nerves in excised frog muscle fibers and to this day the majority of the studies so far have entailed stimulation of excised nerves. The leading technique to treat peripheral neurological disorders is implantation of electrodes along the peripheral nerve and stimulating the nerve with electrical current. A noninvasive alternative that could treat neuropathic pain and suppress nerve activity constitutes thus an important challenge in interventional neurology.
i)
ii)
iii)
iv)
Figure 1: i) FUS setup for neuromodulation, cameras recording hind limb and tail movements and pupil dilation and eye movement. ii) Recorded left hind paw movement (before (purple) after (green) movement), iii) FUS-induced motor response elicitation: EMG of the right hind limb during contralateral evoked response for different acoustic pressure levels with the success rate increasing at larger pressures and iv) contralateral paw movement elicited by FUS neurostimulation.
Our group has been studying the noninvasive stimulation or inhibition of both the central and peripheral nervous system in live animals. In the brain, we have shown that focused ultrasound is capable of noninvasively stimulating paw movement as well as sensory responses such as pupil dilation and eye movement when different brain regions are targeted, showing for the first time that ultrasound can tap into both the motor and sensory brain regions (Fig. 1). In the periphery, when the ultrasound beam is focused on the sciatic nerve in a live, anesthetized animal, the thigh muscle becomes activated and muscle twitches can be induced at low ultrasonic intensities while the same twitches can be inhibited at higher intensities due to associated temperature rise that inhibits nerve firing. Cellular and fiber responses in excised tissue have confirmed the live animal responses (Fig. 2).
a)
b)
Figure 2:FUS compared to electrical modulation: a) Ex vivo measurements of action potentials in a nerve bundle through FUS (red) and electrical stimulation; b) in vivo EMG responses in murine leg muscle at different FUS pressures and duty cycles. FUS elicits very similar motor responses as electrical stimulation (E.S.; dashed horizontal line), especially at higher pressures and duty cycles.
Brian Kappus – brian.kappus@ultrahaptics.com Ben Long – ben.long@ultrahaptics.com Ultrahaptics Ltd. The West Wing, Glass Wharf Bristol, BS2 0EL, United Kingdom
Popular version of paper 3aEAa4 “Spatiotemporal modulation for mid-air haptic feedback from an ultrasonic phased array” presented Wednesday morning, May 9, 2017, 9 AM, Greenway D 175th ASA Meeting, Minneapolis
Haptic feedback is the use of the sense of touch in computing and interface design to communicate with a user. The average person most often experiences haptic feedback when interfacing with modern mobile devices. These uses are relatively basic: a buzz to alert the user to an incoming call or a vibration when using a touchscreen keyboard.
Interfaces enabled by gesture recognition and virtual/augmented reality, however, typically lack haptic feedback. In this paper, we present “virtual touch” technology developed at Ultrahaptics. It enables the generation of haptic feedback in mid-air, on a user’s bare hands, by the efficient creation and manipulation of ultrasonic waves (i.e. frequencies beyond the range of human hearing).
There are a variety of mechanical receptors present on the hand that are sensitive to different types of sensation including temperature, static pressure, and vibration [1]. Receptors sensitive to vibration can be stimulated through focused acoustic pressure. Ultrahaptics’ technology uses ultrasonic transducers coupled with phase delays so that the resulting interference patterns create focused acoustic pressure at focal points. The pressure is sufficient to create tactile sensations without generating audible sound.
Because the vibration-sensitive receptors on the hand are not capable of perceiving ultrasonic frequencies, to create tactile sensations the acoustic pressure then needs to be switched off and back on again (modulated) at lower frequencies – around the range of 40-400Hz.
Previous versions of this technology have been limited to discrete points of acoustic pressure which are turned on and off at the necessary frequencies to create a tactile effect [2] [3] [4]. However, another way to create the tactile effect is to move the focal point back and forth, its movement away and back again providing the modulation at a frequency perceptible by the receptors. While it is away, pressure at the starting point is low. It then returns to the starting (high) pressure when it returns. From the perspective of this starting point, the acoustic pressure varies in amplitude. This creates tactile sensations.
The technique is called spatiotemporal modulation and using it a closed curve can be repeated almost continuously, forming a robust area of stimulation instead of discrete points. Advantages of spatiotemporal modulation include the ability to render an infinite variety of curves and volumes.
Previously, spatiotemporal modulation was impractical due to the hardware computing requirements. In this paper, we present algorithms developed at Ultrahaptics that realize the required acoustic fields with a fraction of the computing power. This enables fast update rates on reasonable hardware and opens up a new class of haptics.
[Video file missing] Caption: Oil-bath visualization of spatiotemporal modulation creating 3 simultaneous shapes. Acoustic pressure is able to deform the surface of a liquid and is used to visualize the acoustic field using edge lightning. In this example, 3 shapes are traced 200 times per second to create continuous lines of pressure.
[1] S. J. Lederman and R. L. Klatzky, “Haptic perception: A tutorial,” Attention Perception & Psychophysics, vol. 71, no. 7, pp. 1439-1459, 2009. [2] T. Iwamoto, M. Tatezono and H. Shinoda, “Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound,” EuroHaptics, LNCS 5024, 504-513, 2008. [3] T. Carter, S.A. Seah, B.Long, B. Drinkwater, S. Subramanian, UltraHaptics: Multi-PointMid-AirHaptic Feedback for Touch Surfaces, Proceedings of the 26th annual ACM symposium on User interface software and technology. (UIST ’13), New York, NY, USA, 8–11 October (2013). [4] B. Long, S. A. Seah, T. Carter and S. Subramanian, “Rendering Volumetric Haptic Shapes in Mid-Air Using Ultrasound,” Transactions on Graphics (Proceedings of SIGGRAPH Asia) 33 (6) 181 (2014).
Accelerating this convergence between medical and consumer hearing technologies is the recently passed congressional bill that mandates the creation of a new over-the-counter hearing aid that consumers can purchase in a store and fit their own prescription. E-health technologies already exist that allow a consumer to measure their own hearing loss and apply clinically-validated prescriptions to their hearable devices. This technology development will explode once over-the-counter hearing aids are a reality.
Haptic feedback is the use of the sense of touch in computing and interface design to communicate with a user. The average person most often experiences haptic feedback when interfacing with modern mobile devices. These uses are relatively basic: a buzz to alert the user to an incoming call or a vibration when using a touchscreen keyboard.