Warning when using CUDA

[caprimotion]

Hi there, thanks for the great stuff. Really easy to use.
When using the NVidia RTX5090, i get the following warning. Do you have a solution?

2025-09-23 16:32:24.7821519 [W:onnxruntime:, transformer_memcpy.cc:83 onnxruntime::MemcpyTransformer::ApplyImpl] 39 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2025-09-23 16:32:24.8046794 [W:onnxruntime:, session_state.cc:1280 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2025-09-23 16:32:24.8100260 [W:onnxruntime:, session_state.cc:1282 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2025-09-23 16:32:25.0043608 [W:onnxruntime:CSharpOnnxRuntime, scatter_nd.h:51 onnxruntime::cuda::ScatterNDWithAtomicReduction::ScatterNDWithAtomicReduction] ScatterND with reduction=='none' only guarantees to be correct if indices are not duplicated.

[caprimotion]

As a thank for all of this: those needing a faster response from the Synthesizer can use this 'streaming' subclassed version of the Synthesizer. It will output the audio bytes as soon as they become available via an event. In my case this was the difference between waiting 800ms for a full sentence or 350 ms for the first audible text. With the GPU it went down to 250ms.

The speed at which you get bytes might overflow the buffer in the NAudio.Wave.BufferedWaveProvider. I have an AudioFeeder class to fix that (scroll further down):

class KokoroWavSynthesizerStreaming: KokoroWavSynthesizer
{
public event EventHandler OnAudioAvailable;
public KokoroWavSynthesizerStreaming(string modelPath, SessionOptions options = null) : base(modelPath, options)
{
}

public async Task SynthesizeAsync(string requestor, string text, KokoroVoice voice, KokoroTTSPipelineConfig pipelineConfig = null)
{
    int[] tokens = Tokenizer.Tokenize(text.Trim(), voice.GetLangCode(), pipelineConfig.PreprocessText);
    List<int[]> list = pipelineConfig.SegmentationFunc(tokens);
    KokoroJob job = EnqueueJob(KokoroJob.Create(list, voice, pipelineConfig.Speed, null));
    List<byte> bytes = new List<byte>();
    if (list.Count > 1)
    {
        new List<char>();
    }

    foreach (KokoroJob.KokoroJobStep step in job.Steps)
    {
        byte[] partialResult;
        step.OnStepComplete = delegate (float[] samples)
        {
            bytes.AddRange(KokoroPlayback.GetBytes(KokoroPlayback.PostProcessSamples(samples)));
            if (Tokenizer.PunctuationTokens.Contains(step.Tokens[^1]))
            {
                float num = pipelineConfig.SecondsOfPauseBetweenProperSegments[Tokenizer.TokenToChar[step.Tokens[^1]]];
                bytes.AddRange(KokoroPlayback.GetBytes(new float[(int)(num * (float)KokoroPlayback.waveFormat.SampleRate)]));
            }

            partialResult = bytes.ToArray();
            AudioAvailableEventArgs e = new AudioAvailableEventArgs();
            e.AudioBytes = partialResult;
            e.requestor = requestor;
            OnAudioAvailable?.Invoke(this, e);
            bytes.Clear();
        };
    }

    while (!job.isDone)
    {
        await Task.Delay(10);
    }
}

}

////////////////// AUDIO FEEDER CLASS
using NAudio.Wave;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace YOUR_NAMESPACE
{
///

/// The AudioFeeder class can receive audio via the AddAudio method. The audio is provided to the BufferedAudioReceiver owned by the caller.
/// The class implements a feeder thread to do the work. Therefore, you must call Stop on the feeder to gracefully stop the thread when you no longer want to use it.
/// The class is thread-safe.
///

public class AudioFeeder
{
///

/// The feeder thread that will keep feeding available audio to the BufferedWaveProvider
///

Thread _audioFeeder;

    /// <summary>
    /// Indicates that the audio feeder thread must be stopped
    /// </summary>
    bool _gracefulStopRequested = false;

    BufferedWaveProvider _bufferedWaveProvider = null;
    /// <summary>
    /// The bufferWaveProvider to fill with received streaming audio
    /// </summary>
    public BufferedWaveProvider BufferedWaveProvider { get { return _bufferedWaveProvider; } }

    /// <summary>
    /// Store of received audio from the synthesizer
    /// </summary>
    ConcurrentQueue<byte[]> _byteQueue = null;

    /// <summary>
    /// creates a new AudioFeeder
    /// </summary>
    /// <param name="bwp"></param>
    public AudioFeeder(BufferedWaveProvider bwp)
    {
        _bufferedWaveProvider = bwp;

        _byteQueue = new ConcurrentQueue<byte[]>();

        ThreadStart ts = new ThreadStart(FeedAudio);
        _audioFeeder = new Thread(ts);
        _audioFeeder.Start();
    }

    /// <summary>
    /// Add audio bytes to the internal storage. It will be fed to the Buffered Wave Provider automatically
    /// </summary>
    /// <param name="audioBytes"></param>
    public void AddAudio(byte[] audioBytes)
    {
        _byteQueue.Enqueue(audioBytes);
    }

    /// <summary>
    /// Start the feeding of audio to the buffered wave provider. Call this prior to requesting synethesis.
    /// </summary>
    /// <returns></returns>
    public void FeedAudio()
    {
        //run a continous loop to feed available audio to the buffered wave provider without overflowing its internal buffer
        while (!_gracefulStopRequested)
        {
            //if byte available, feed them
            if (_byteQueue.Count > 0)
            {
                int sourceIndex = 0;
                int spaceAvailable = 0;

                //get the first chunk of bytes as received in OnAudioAvailable
                byte[] bytes;
                _byteQueue.TryDequeue(out bytes);

                //while bytes are available in this chunk of bytes to feed, feed them
                while (sourceIndex < bytes.Length && !_gracefulStopRequested)
                {
                    //determine space availabed in the provider
                    spaceAvailable = _bufferedWaveProvider.BufferLength - _bufferedWaveProvider.BufferedBytes;

                    int bytesToFeed = spaceAvailable;
                    if (spaceAvailable > (bytes.Length - sourceIndex))
                    {
                        //if more space is available than we have bytes,set the bytes to feed to (only) what we have
                        bytesToFeed = bytes.Length - sourceIndex;
                    }
                    _bufferedWaveProvider.AddSamples(bytes, sourceIndex, bytesToFeed);

                    //register the bytes fed into the provider
                    sourceIndex += bytesToFeed;

                    //Pause to allow the buffer to be read by the output device
                    Thread.Sleep(100);
                }
            }

            //prevent draining the CPU
            Thread.Sleep(100);
        }
    }

    /// <summary>
    /// Request a gracefull stop of the audio feeder. show complete within less than a second
    /// </summary>
    public void Stop()
    {
        _gracefulStopRequested = true;
    }
}    

}

[caprimotion]

this part was missing:
public class AudioAvailableEventArgs : EventArgs
{
public byte[] AudioBytes { get; set; }
public string requestor { get; set; }
public AudioAvailableEventArgs() : base() { }
}

[Lyrcaxis]

Hi! I'm glad you like the repo!

Is your initialization similar to this?

var options = new SessionOptions();
options.AppendExecutionProvider_CUDA();
KokoroTTS tts = KokoroTTS.LoadModel(sessionOptions: options);
tts.SpeakFast("Hello from GPU!", KokoroVoiceManager.GetVoice("af_heart"));

I would recommend setting options.LogSeverityLevel to either 'Info' or 'Verbose' to be able to determine what's wrong.

Also, are there any actual issues after the warning? Or simply scary to see?

ONNX does convert most of the stuff, but some intermediate tensors might remain on CPU -- it's usually not an issue, unless you see actual performance hits.

[caprimotion]

Thanks for the swift response.
My init is similar. Although i use the underlying Synthesizer instead of the simple version.
Setting the log level to INFO results in the same warning.
Setting it to error only...gives me warning :) :
2025-09-23 16:59:26.6338909 [W:onnxruntime:CSharpOnnxRuntime, scatter_nd.h:51 onnxruntime::cuda::ScatterNDWithAtomicReduction::ScatterNDWithAtomicReduction] ScatterND with reduction=='none' only guarantees to be correct if indices are not duplicated.

And yes, the output is just fine.

[Lyrcaxis]

Cool, then I think it's nothing to worry about -- indices are likely not duplicated 😆

[caprimotion]

It is probably the NG model that was build to be used by the CPU.

[rwfsmith]

I've been trying this out, I made a little local AI assistant sort of thing that uses Whisper Turbo, GPT-OSS-20B with LM Studio(Including Playwright MCP for web lookups), and Kokoro. Still needs a lot of work, but it's only been a week or so that I've been working on it.

It's pretty fast to get started, but I get a lot of clicks in the audio and it seems to hit issues sometimes with pronouncing things, like for "you’d" it says "yoo-dee" instead of just "yood". Any idea how to improve that?

[Lyrcaxis]

@rwfsmith awesome project! Yeah, the audio nicification could use improvements.
A way to get the raw audio played back would be to set tts.NicifySamples = false;.

I tweaked the values here empirically, but they're not done very well, apparently:
https://github.com/Lyrcaxis/KokoroSharp/blob/main/KokoroSharp/HighLevel/KokoroPlayback.cs#L87

---

As for the you'd --> yood thing, that'd be part of Preprocessing. I tried to cover a lot of the stuff, but A LOT were left uncovered.

You could definitely just preprocess your text as string before passing it to tts.SpeakFast(), and it'll work
(e.g. tts.SpeakFast(txt.Replace("you'd", "yood")), or you could contribute to the repo if you're feeling generous ^^