Real-time speech events

[brucek5]

Is there a means to insert an event handler as the speech audio is outputted that would enable reporting in real-time the "Visemes or Phonemes" so speech could be synchronized with animated mouth movements? Example is like the c# Microsoft "SpeechSynthesizer" class where it fires a "VisemeReached" event.

[Lyrcaxis]

Hi. The base model DOES support retrieving phoneme timings, but it's not supported in KokoroSharp.

I had tried to do that initially, but it requires multiple calls to ONNX-Runtime side per chunk for a streaming-like workflow, and I didn't find a proper way to retrieve and pass the KV state in again after each step proceeds.

What should be feasible, though, is to retrieve a timings array along with the response that allows people to map phonemes of an already-retrieved transcription chunk to whatever they like. Should support this kind of use-case without any slowdowns.

---

TLDR: This is not supported currently, but I see merit in it, so would be a nice feature to add.