Request for Guidance on Adding Hindi Support to CSM

[goravaa]

Hi SesameAILabs team,

I’m interested in extending the CSM model to support Hindi for text-to-speech tasks. My goal is to adapt CSM to generate conversational speech in Hindi. Here are some of the aspects I’ve considered so far:

• Data:
  I plan to gather or leverage existing Hindi speech and text datasets to fine-tune the model effectively.

• Tokenizer:
  The current implementation uses a tokenizer (from Meta Llama) that is optimized for English. I might need to adapt or re-train a tokenizer (e.g., using SentencePiece) on Hindi text to better handle Hindi script and vocabulary.

• Training Strategy:
  I’m considering a transfer learning approach where the model is fine-tuned on Hindi-specific data. This may require adjustments to both the text encoder and the audio decoder to capture the linguistic and acoustic nuances of Hindi.

Could you please provide guidance on the following:

• Are there any existing plans or recommendations for adding multilingual support or specifically Hindi support to CSM?

• What best practices would you suggest for fine-tuning a model like CSM for a language with a different script and phonetic structure?

• Are there any particular modifications or additional components (such as phoneme-based representations) that might improve Hindi speech generation?

I’m open to collaboration and would greatly appreciate any feedback or pointers from the community.

Thank you for your work on CSM, and I look forward to your suggestions.

[ZackHodari]

We haven't done any experiments relating to this yet, so this is all speculation:

I'd avoid changing the tokenizer (LLama) and codec (Mimi) initially. They may be a bottleneck, but I think there's better places to spend your effort initially.

You could attempt to do the same kind of fine-tuning other people are looking at for English (try the discord), but use your own Hindi data. Note that the base model training only contained small amounts of non-English data. If the base model is powerful enough, fine-tuning might give you decent results. The only way to find out is to try

[goravaa]

Hey Zack, thanks for the input! I'm starting on the Hindi dataset and will fine-tune using the current setup as my baseline. If I run into any hiccups (like tokenizer issues or acoustic tweaks), I'll adjust accordingly. I'll keep you posted on how it goes. Appreciate the help!

[hanspaa2017108]

@goravaa any update on Hindi fine-tune !? Looking forward to some updated. How did you prepare dataset? Can you share one example of the transcript. Thanks

[tarun7r]

@goravaa Hi, any update on fine-tuning hindi model? if so did you change the tokenizer as well and what format of dataset did you use?

[goravaa]

Hi @tarun7r and @hanspaa2017108,

Thanks for following up! I’m still in the process of preparing the Hindi dataset—quality Hindi audio data is quite challenging to source, so it’s taking a bit more time.

For initial testing, I created a small dataset (you can find it here: https://huggingface.co/datasets/garavv/AudioLlama) just to validate the training scripts. The process has been resource-intensive, and I ran into OOM (out-of-memory) issues with full fine-tuning. As a next step, I’m planning to switch to LoRA-based fine-tuning to make it more feasible on my current hardware.

I’ll share updates and results here as soon as I have something concrete.

[anjalyv]

Hi @goravaa ,

This is the notebook I followed for fine-tuning on Gujarati data:
https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_(1B)-TTS.ipynb

However, with the inference script, audio can be generated without context. I was also unable to use the model with the provided inference script for the pre-trained CSM model that uses the generator.

Were you able to fine-tune the model for Hindi?