Adrienne Roman – adrienne.s.roman@vumc.org
Carlos Benitez – carlos.r.benitez@vanderbilt.edu
Alexandra Key – sasha.key@vanderbilt.edu
Anne Marie Tharpe – anne.m.tharpe@vumc.org

The brain needs a variety of stimulation from the environment to develop and grow. The ability for the brain to change as a result of sensory input and experiences is often referred to as experience-dependent plasticity. When children are young, their brains are more susceptible to experience-dependent plasticity (e.g., Kral, 2013) so the quantity and quality of input is important. Because our ears are always “on”, our auditory system receives a lot of input to process, especially while we are awake. But, what can we “hear” when we are asleep? And, does what we hear while we are asleep help our brains develop?

Although there has been research in infants and adults examining the extent to which our brains process sounds during sleep, very little research has focused on young children, a group that sleeps a significant portion of their day (Paruthi et al., 2016). We decided to start our investigation by trying to answer the question, do children process and retain information heard during sleep? To investigate this question, we used electroencephalography (EEG) to measure the electrical activity of children’s brains in response to different sounds – sounds they heard when asleep and sounds they heard when awake.

First, during the child’s regular naptime, each child was hooked up to a portable EEG. Using EEG, a technician could tell us when the child went to sleep. Once asleep, we played the child three made-up words over and over in random order for ten minutes. Then, we let the child continue to sleep until he or she woke up.

When the children awoke from their naps, we took them to our EEG lab for event-related potential (ERP) testing. ERPs are segments of on-going EEG recordings appearing as waveforms that reflect the brain’s response to  events or stimulation (such as a sound played).

The children wore “hats” consisting of 128 spongy electrodes while listening to the same three made-up words heard during the nap mixed in with new made-up words that the children never heard before. We then analyzed the ERPs, to determine if the children’s brains responded differently to the words played during sleep than to the new words the children had not heard before. We were looking for ‘memory traces’ in the EEG that would indicate that the children ‘remembered’ the words heard while sleeping.

We found that children’s brains were able to differentiate the nonsensical words “heard” during the nap from the brand new words played during the ERP testing. This means that the brain did not just filter the information coming in, but also retained it long enough to recognize it after they woke up. This is the first step in understanding the impact of a child’s auditory environment during sleep on the brain.

Kral, A. (2013). Auditory critical periods: a review from system’s perspective. Neuroscience, 247, 117-133.

Paruthi, S., Brooks, L. J., D’Ambrosio, C., Hall, W. A., Kotagal, S., Lloyd, R. M.,

Malow, B. A., Maski, K., Nichols, C., Quan, S. F., Rosen, C. L., Troester, M. M., & Wise, M.S. (2016). Recommended amount of sleep for pediatric populations: a consensus statement of the American Academy of Sleep Medicine. Journal of clinical sleep medicine: JCSM: official publication of the American Academy of Sleep Medicine, 12(6), 785.

Tejal Udhan – tu13b@my.fsu.edu
Shonda Bernadin – bernadin@eng.famu.fsu.edu
FAMU-FSU College of Engineering,
Department of Electrical and Computer Engineering
2525 Pottsdamer Street Tallahassee
Florida 32310

Popular version of paper 1pPPB: ‘Speaker-dependent low-level acoustic feature extraction for emotion recognition’
Presented Monday afternoon May 7, 2018
175th ASA Meeting, Minneapolis

Speech is a most common and fastest means of communication between humans. This fact compelled researchers to study acoustic signals as a fast and efficient means of interaction between humans and machines. For authentic human-machine interaction, the method requires that the machines should have the sufficient intelligence to recognize human voices and their emotional state. Speech emotion recognition, extracting the emotional state of speakers from acoustic data, plays an important role in enabling machines to be ‘intelligent’. Audio and speech processing provides better, noninvasive and easy to acquire solutions than other biomedical signals such as electrocardiograms (ECG), and electroencephalograms (EEG).

Speech is an informative source for the perception of emotions. For example, talking in a loud voice when feeling very happy, speaking in an uncharacteristically high pitched voice when greeting a desirable person, or the presence of vocal tremor when something fearful or sad have been experienced. This cognitive recognition of emotions in turn indicates that listeners are able to infer the emotional state of the speaker reasonably accurately even in the absence of visual presence of information [1]. This theory of cognitive emotion inference forms the basis for speech emotion recognition. Acoustic emotion recognition finds so many applications in modern world ranging from interactive entertainment systems, medical therapies and monitoring to various human safety devices.

We conducted some preliminary experiments to classify four human emotions anger, happy, sad and neutral (no emotion) in male and female speakers. We chose two simple acoustic features, pitch and intensity, for this analysis. The choice of features is based on readily available tools for their calculation. Pitch is a relative highness or lowness of a tone as perceived by the ear and intensity is the energy contained in speech as it is produced. Since these are one- dimensional features, they can be easily analyzed for any acoustic emotion recognition system. We designed decision-tree based algorithm in MATLAB to perform emotion classification. LDC Emotional Prosody Dataset samples are used for this experiment [2]. One sample of each emotion for one male and one female speaker are given below.

{audio missing}

We observed that male speaker does not have many variations in the pitch for all the emotions. The pitch is consistently similar for any given emotion. The median intensity over each emotion class, though changing, remains consistently similar to training data values. As a result, emotion recognition in male speaker has accuracy of 88% for acoustic test signals. Though pitch is almost similar, there is clear distinction in intensities for emotions happy and sad. This dissimilarity in intensity resulted in higher accuracy of emotion recognitions in male speaker data. For female speaker, the pitch ranges anywhere from 230 Hz to 435 Hz for three different emotions, namely, happy, sad and anger. Hence, the median intensity becomes the sole criterion for emotion recognition. The intensities for emotions, happy and angry are almost similar since both the emotions are high arousal emotions. This resulted in low accuracy of emotion recognition in female speaker of about 63%. The overall accuracy of emotion recognition using this method is 75%.

Fig. 1. Emotion Recognition Accuracy Comparison

Our algorithm successfully recognized emotions in male speaker. Since the pitch is consistent across each emotion in male speaker, the selected features, pitch and intensity, resulted in better accuracy of emotion recognition. For female acoustic data, selected features are insufficient to describe the emotions and hence in future research of this work, other features which are independent of voice quality such as prosodic, formant or spectral features will be evaluated.

[1]Fonagy, I. Emotions, voice and music. Sundberg J (Ed.) Research aspects on singing. Royal Swedish Academy of Music and Payot, Stockholm and Paris; pp 51–79, 1981.
[2]Liberman, Mark, et al. Emotional Prosody Speech and Transcripts LDC2002S28. Web Download. Philadelphia: Linguistic Data Consortium, 2002.