After using one of these techniques to measure facial movements, there is still the question of how to use this information to produce speech. Trying to construct a full physical model that would account for all the placements and motions of the articulators would be very difficult and error-prone. Instead, we adapted a statistical technique from the field of Voice Transformation to construct a generalized correspondence between the facial movement measurements and microphone-recorded speech.

In its original conception, Voice Transformation is the process of taking speech from one person, called the source speaker, and making it sound as if it were spoken by another person, called the target speaker. One popular technique for accomplishing this task is called Gaussian Mixture Model Mapping. In this technique, the source and target speakers are recorded reading the same sentences, and these recordings are used to determine parameters of a statistical model, called a Gaussian Mixture Model. This model can be used to predict how the target speaker would say something based on how the source speaker says it. Then, when the source speaker says a new utterance that has not been recorded by the target speaker, the same statistical model can be applied to transform it to sound like the target speaker. Due to the nature of the statistical model, it can generalize beyond the original data used to estimate the models parameters and predict target speaker speech even when the source speakers speech includes new phenomena that were not found in the original data.

From a more general viewpoint, Voice Transformation is a way of taking one type of data and converting it to speech, as long as this data is from a process that is correlated with the produced speech. As this is indeed the case with EMG and ADS data, we used this approach to convert from EMG and ADS data to speech.

For both EMG and ADS, we collected facial movement measurements and recorded speech simultaneously. For EMG, we had a person read 500 sentences. 380 were used to determine a correspondence between facial movements and speech and 120 were used for evaluation. For ADS, we had a person read 188 sentences. 170 were used to determine a correspondence, and 18 were used for evaluation. Evaluation consisted of taking the facial movement measurements, using them to produce synthetic speech, and comparing the synthetic speech with the actual recorded speech. The comparison was performed considering two types of measures which are referred to as objective and subjective.

Objective measures are based on mathematical formulas, and are convenient in that they can be automatically computed. Their disadvantage is that none are known to correspond perfectly to human perception. Due to differences in opinions among humans, a perfect correspondence might not even be possible. Nevertheless, some objective measures, such as the Mel-Cepstral Distortion (MCD) used in our experiments, have been shown to generally correlate to human perception of speech quality, and are useful as rough indicators of performance. In our best trials, the average MCD between speech synthesized from EMG and the actual recorded speech was 6.37. For the best ADS trials, the MCD was 6.69. A Mel-Cepstral distortion of 0 would indicate a perfect match, and for comparison, typical Voice Transformation from speech to speech, which produces intelligible, though somewhat unnatural sounding speech often has MCDs in the range of 5-7.

Subjective measures are based on collecting human opinions. In many subjective measures, a number of humans are asked to score a quality of speech, such as naturalness, on a numerical scale, and the scores are then averaged. The advantage of subjective measures is that the goal often is related to human perception, and these are the most direct methods to measure it. The disadvantage is that conducting experiments with humans can be a time-consuming, expensive process. For that reason, it is common to first use an objective measure and informally listen to synthesized examples to determine whether the results may be good enough to warrant a formal subjective evaluation.

After listening to synthetic speech both partially predicted from facial movements and completely predicted from facial movements, we decided that the intelligibility of the artificial speech was not yet high enough to warrant a formal subjective evaluation. The partially synthesized examples for both EMG and ADS were somewhat intelligible, but the examples fully synthesized from EMG were not. We were able to occasionally recognize words and phrases in the examples fully synthesized from ADS, however. We also found it to be speech-like, both in terms of how it sounded and in the similarities in the spectrograms, which are diagrams that display different sound frequency contributions over time. Sound example 1 is a recorded sentence from our ADS evaluation set. Sound example 2 is the same sentence partially resynthesized from ADS data. The power and fundamental frequency were taken from the recording, but the spectral envelopes were estimated from the ADS data. Finally, Sound example 3 is the same utterance synthesized solely from the ADS data. Due to potential difficulties with predicting fundamental frequencies, we chose to use noise excitation to produce whisper-like speech for this experiment. Although more work is necessary to improve the intelligibility, our preliminary results are encouraging.

For more information on these topics, including links to publications, news articles, and example sound files, please visit http://www.cs.cmu.edu/~atoth/ASA2010.html