Slow on Linux platform

[satyaai]

Thanks for the great open-source project, it's really helpful.

While running on Windows, the processing seems faster than on the Linux platform. I tested it on a Windows machine with an i9 14th Gen 3.1GHz CPU and on a Linux machine on GCP with a c3-highcpu-22 instance featuring a Xeon 4th Gen 2.2GHz CPU. Considering the CPU specs, it's expected to be slower on the Linux machine. However, I also tried running it on the WSL installation of the Windows machine, and there it took almost 1.5 times longer than the native Windows process.

I tested it post building from source, it is faster than running the binary release but still slower than the windows process.
Windows process (Binary release)- 2.3 sec
Linux with binary release- 8.9 sec
Linux with build from source-3.9 sec for the same .wav audio file.

So, can you please comment on whether it's by design or due to some internal architecture that it runs slower on Linux platforms compared to Windows? Or am I missing something here?

[DanielSWolf]

I certainly didn't intend for Rhubarb to run slower than necessary on any platform. Given that it's faster when you build it from sources, my guess is that you used a different compiler (or a later version of the same compiler).

All CI build logic is in .github\workflows\ci.yml. If you come up with a config that results in better performance, I'd be happy to change it.

[ajayraj12]

I am integrating Rhubarb Lip Sync into my real-time virtual avatar application, where performance is crucial. However, I am experiencing significant delays in processing audio to generate lip sync data.

I have tried optimizing it using multi-threading, reducing CPU load, and tweaking the --threads parameter, but the processing time remains high. My current setup utilizes:

System: 8-core CPU
Threads Allocation: max(1, TOTAL_CPU_CORES - 2)
Processing Approach: Python threading and subprocess.run()
Despite these optimizations, lip sync generation takes too long, affecting the real-time performance of my application. I would really appreciate any suggestions on further optimizing the process, such as:

Better utilization of multi-threading/multi-processing
Any possible caching or pre-processing techniques
Alternative approaches for handling audio input efficiently
Code Implementation:
Here’s my current implementation:

python
Copy
Edit
import subprocess
import json
import threading
import os
from pydub import AudioSegment
import tempfile
import multiprocessing

TOTAL_CPU_CORES = multiprocessing.cpu_count()
OPTIMAL_THREADS = max(1, TOTAL_CPU_CORES - 2)

print(f"Initializing Rhubarb with {OPTIMAL_THREADS} threads (Total Cores: {TOTAL_CPU_CORES})")

def get_audio_duration(audio_file):
audio = AudioSegment.from_file(audio_file)
return round(audio.duration_seconds, 2)

def run_rhubarb(audio_file):
try:
duration = get_audio_duration(audio_file)
rhubarb_path = get_rhubarb_executable()

    cmd = [
        rhubarb_path,
        "-f", "json",
        "--threads", str(OPTIMAL_THREADS),
        audio_file
    ]

    process = subprocess.run(cmd, capture_output=True, text=True)

    if process.returncode != 0:
        raise RuntimeError(f"Rhubarb Error: {process.stderr}")

    output_json = json.loads(process.stdout)

    mouth_cues = [{"start": entry["start"], "end": entry["end"], "value": entry["value"]} for entry in output_json.get("mouthCues", [])]

    return {
        "metadata": {"soundFile": audio_file, "duration": duration},
        "mouthCues": mouth_cues
    }

except Exception as e:
    return {"error": str(e)}

def run_rhubarb_threaded(audio_file):
result = {}

def worker():
    nonlocal result
    result = run_rhubarb(audio_file)

rhubarb_thread = threading.Thread(target=worker)
rhubarb_thread.start()
rhubarb_thread.join()

return result

def get_rhubarb_executable():
rhubarb_path = os.getenv("RHUBARB_PATH")
if not rhubarb_path or not os.path.exists(rhubarb_path):
raise FileNotFoundError("Rhubarb executable not found. Set RHUBARB_PATH environment variable.")
return rhubarb_path

def convert_mp3_to_wav(mp3_path, wav_path):
command = f'ffmpeg -i "{mp3_path}" -acodec pcm_s16le -ar 16000 "{wav_path}" -y -loglevel error'
subprocess.run(command, shell=True)

def process_lipsync(audio_file):
ext = os.path.splitext(audio_file)[1].lower()
if ext == ".mp3":
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp_wav:
wav_file = tmp_wav.name
convert_mp3_to_wav(audio_file, wav_file)
else:
wav_file = audio_file

return run_rhubarb_threaded(wav_file)

Request for Suggestions:
Is there a better way to parallelize the processing of audio files with Rhubarb?
Should I use Python’s multiprocessing instead of threading for better CPU utilization?
Are there any recommended configurations for Rhubarb's --threads parameter for an 8-core system?
Any other system-level or code level optimizations I can try?
I would greatly appreciate any feedback or suggestions. Thanks in advance!

[DanielSWolf]

This issue, the way you originally described it, is about performance on Linux being worse than on Windows when using the pre-built binaries. This is something that should be trivially fixable by switching to a different compiler or a later version of the same compiler. Please let me know your build setup that yields faster binaries.

Regarding processing speed in general: I'm afraid there is no easy fix for this. Rhubarb is primarily designed for use in the production of video games or similar, when time isn't much of an issue. While I understand that there are real-time applications like yours where this is a problem, my immediate focus is improving quality rather than speed.

If the issue is processing many files rather than a single one, you'll probably get faster overall performance if you disable Rhubarb's multithreading altogether and run several instances in parallel.

[j2l]

If the issue is processing many files rather than a single one, you'll probably get faster overall performance if you disable Rhubarb's multithreading altogether and run several instances in parallel.

Good to know! Thank you
(I'm also on Linux)