StyleCap: Automatic Speaking-Style Captioning from Speech
Based on Speech and Language Self-supervised Learning Models

Abstract

We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. As a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning, our StyleCap uses paired data of speech and natural language descriptions to train neural networks that predict prefix vectors fed into a large language model (LLM)-based text decoder from a speech representation vector. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions.

Content

• 1. Dataset
• 2. Data augmentation by rephrasing sentences using an LLM
• 3. Samples of generated captions

1. Dataset

We used PromptSpeech [1], which includes natural language instructions (style prompt) of various speaking styles. This dataset was constructed for PromptTTS [1] that can synthesize speech in accordance with a style prompt. We used the training subset of PromptSpeech real version, which includes 26,588 human-annotated style prompts of multi-speakers' speech samples from LibriTTS [2]. We divided the 26,588 prompts into training (24,953 by 1,113 speakers), development (857 by 40), and test (778 by 38) subsets and paired them with the corresponding speech samples from LibriTTS.

The related data is available for download from the following link.
・PromptSpeech
・LibriTTS corpus
・train_dev_test_set.zip (Each subset's file id list)

2. Data augmentation by rephrasing sentences using an LLM

We introduce sentence rephrasing augmentation using an LLM to increase the diversity of speaking-style descriptions in the training data. Specifically, we first ask the pre-trained Llama 2-Chat (7B) [3], which is an LLM optimized for dialogue use cases, to generate five sentences from one given sentence on the basis of the following prompt:

"Rewrite the following sentence that describes someone's style of speaking in a different way, keeping the meaning of the original description. Original Description: [subject to be rephrased]."

Then, we select one sentence from the five rephrased sentences on the basis of their BERTScore [4] values: i.e., we pick on the sentence with the highest score if the score is higher than 0.80.

Samples of sentence rephrasing

3. Samples of generated captions

In this section, we present samples of generated captions for the test set using each of the comparison models explained in section 3.2 of our paper. The table demonstrates the samples when each model was trained both without sentence rephrasing augmentation and with sentence rephrasing augmentation.

References

• Z. Guo, Y. Leng, Y. Wu, et al., "PromptTTS: Controllable text-to-speech with text descriptions," in Proc. ICASSP, 2023.
• H. Zen, V. Dang, R. Clark, et al., "LibriTTS: A corpus derived from lib- rispeech for text-to-speech," in Proc. INTERSPEECH, 2019, pp. 1526-1530.
• H. Touvron, L. Martin, K. Stone, et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023.
• T. Zhang, V. Kishore, F. Wu, et al., "BERTScore: Evaluating text generation with BERT," in Proc. ICLR, 2020.