Any kind of concatenative synthesizer relies on high-quality recorded speech databases. An example fragment from such a database is shown in Fig. 2. The top panel shows the time waveform of the recorded speech signal, the middle panel shows the spectrogram ("voice print"), and the bottom panel shows the annotations that are needed to make the recorded speech useful for concatenative synthesis.

In the top panel of Fig. 2, we see the waveform for the words "pink silk dress." For the last word, dress, we have bracketed the phone /s/ and the diphone /eh-s/ that encompasses the latter half of the /eh/ and the first half of the /s/ of the word "dress." For American English, a diphone-based concatenative synthesizer has, at a minimum, about 1000 diphone units in its inventory. Diphone units are usually obtained from recordings of a specific speaker reading either "diphone-rich" sentences or "nonsense" words. In both cases the speaker is asked to articulate clearly and use a rather monotone voice. Diphone-based concatenative synthesis [7]has the advantage of a moderate memory footprint, since one diphone unit is used for all possible contexts. However, since speech databases recorded for the purpose of providing diphones for synthesis do not sound "lively" and "natural" from the outset, the resulting synthetic speech tends to sound monotonous.
For years, expert labelers were employed to examine waveform and spectrogram, as well as their sophisticated listening skills, to produce annotations ("labels") such as those shown in the bottom panel of the figure. Here we have word labels (time markings for the end of words), tone labels (symbolic representations of the "melody" of the utterance, here in the ToBI standard, [8] ), syllable and stress labels, phone labels (see above), and break indices (that distinguish between breaks between words, sub-phrases, and sentences, for example).

It turns out that expert labelers need about 100-250 seconds of work time to label one second of speech with the set depicted in Fig. 2 [9]. For a diphone-based synthesizer, this might be a reasonable investment, given that a "diphone-rich" database (a database that covers all possible diphones in a minimum amount of sentences) might be as short as 30 minutes. Clearly, manual labeling would be impractical for much larger databases (dozens of hours). For this, we would require fully automatic labeling, using Speech Recognition tools. Fortunately, these tools have become so good, that speech synthesized from an automatically labeled speech database is of higher quality than speech synthesized from the same database that has been labeled manually [10].

With the availability of good automatic speech labeling tools, Unit-Selection Synthesis has become viable for obtaining customer-quality TTS. Based on earlier work done at ATR in Japan [11][12][13], this new method employs speech databases recorded using a "natural" (lively) speaking style [14]. The database may be focused on narrow-domain applications (such as "travel reservations" or "telephone number synthesis"), or it may be used for general applications like email or news reading. In the latter case, unit-selection synthesis can require on the order of ten hours of recording of spoken general material to achieve high quality. In contrast with earlier concatenative synthesizers, unit-selection synthesis automatically picks the optimal synthesis units (on the fly) from an inventory that can contain thousands of examples of a specific diphone, and concatenates them to produce the synthetic speech. This process is outlined in Fig. 3, which shows how the method must dynamically find the best path through the unit-selection network corresponding to the sounds for the word 'two.' The optimal choice of units depends on factors such as spectral similarity at unit boundaries (components of the "join cost" between two units) and on matching prosodic targets set by the front-end (components of the "target cost" of each unit). In addition, there is the problem of having anywhere from just a few examples in each unit category to several hundreds of thousands of examples to chose from. Obviously, also, the unit selection algorithm must run in a fraction of real time on a standard processor.

In respect to quality, there are two good explanations why the method of unit-selection synthesis is capable of producing customer quality or even natural quality speech synthesis. First, on-line selection of speech segments allows for longer fragments of speech (whole words, potentially even whole sentences) to be used in the synthesis if they are found with desired properties in the inventory. This is the reason why unit-selection appears to be well suited for limited-domain applications such as synthesizing telephone numbers to be embedded within a fixed carrier sentence. Even for open-domain applications, such as email reading, however, advanced unit selection can reduce the number of unit-to-unit transitions per sentence synthesized and, consequently, increase the segmental quality of the synthetic output. Second, the use of multiple instantiations of a unit in the inventory, taken from different linguistic and prosodic contexts, reduces the need for prosody modifications that degrade naturalness [15][16].

Conclusions
The advent of high-quality text-to-speech (TTS) may have created the false notion of speech synthesis being a "solved problem," that is, the idea that speech synthesis can replace a live human speaker (or a speaker's recording) in any application, service, or product. This is definitely not the case, given the enormous richness and expressive capabilities of the human voice that is impossible, or at least impractical, to match with a speech synthesizer. What unit selection speech synthesis can do, however, is deliver surprisingly good quality speech for somewhat narrow applications, such as, for example, travel reservations, weather reports, etc. [17] The high quality is achieved by recording special domain voice databases. For a given domain (e.g., "travel"), voice talents are being recorded while reading examples from that domain, such as "Your flight to <destination> has been confirmed." The idea is to cover as much material as possible that is well suited for the given application. Also, the reading style used (e.g., friendly but affirmative) has to be appropriate for the application. What unit-selection TTS cannot do today (at least not in any practical way) is to turn an average voice, reading in a "newsreader" (i.e., reserved, toned-down) style, into a highly desirable "spokesperson" voice for marketing a new product (i.e., speaking in a highly expressive, enthusiastic style). The reason for this inability is simple: there is no way that all the necessary speech data (many hundreds of hours) can be recorded from one single speaker, given the time it would take and the fact that a speaker's voice might change over time. We also do not know enough yet to use signal processing to turn normal speech into highly expressive/emotional speech.