Automatic Speech Recognition (ASR)

[hongbo-miao]

Inference

• https://github.com/OpenNMT/CTranslate2
• https://github.com/jkawamoto/ctranslate2-rs
• https://github.com/SYSTRAN/faster-whisper
  • Support word-level timestamps
• https://github.com/CodersCreative/faster-whisper-rs
  • Allow loading whisper models from custom path CodersCreative/faster-whisper-rs#2 (comment)
• https://github.com/m-bain/whisperX
  • VAD + ASR
  • Support multispeaker
  • Support word-level timestamps
  • Does not support fine-tuned mode: Support inference from fine-tuned 🤗 transformers Whisper models m-bain/whisperX#25
  • Server
    • https://github.com/alexgo84/whisperx-server
    • https://github.com/chinaboard/whisperX-service
    • https://github.com/Nyralei/whisperx-api-server
• https://github.com/speaches-ai/speaches/
  • Good for concurrent API calls: question: concurrent API calls? speaches-ai/speaches#254
  • Support fine-tuned models
    • using a custom model speaches-ai/speaches#174
    • Add possibility to set custom models speaches-ai/speaches#352
• https://github.com/NVIDIA-NeMo/NeMo
  • [NeMo 2.6] RNNT ASR inference fails on A100 (CUDA 12.8, PyTorch 2.9) with CUDA Graphs error CUDA failure! 35 NVIDIA-NeMo/NeMo#15145
• https://github.com/k2-fsa/sherpa-onnx
  • https://github.com/thewh1teagle/sherpa-rs
• (Removed Whisper support in latest version) https://github.com/vllm-project/vllm
  • https://docs.vllm.ai/en/v0.7.0/getting_started/examples/whisper.html
  • https://docs.vllm.ai/en/latest/examples/offline_inference/audio_language.html
  • https://huggingface.co/blog/fast-whisper-endpoints
• https://github.com/gpustack/vox-box
• (Not maintained) https://github.com/litongjava/whisper-cpp-server
• (Not maintained) https://github.com/Vaibhavs10/insanely-fast-whisper
• (Not maintained) https://github.com/JigsawStack/insanely-fast-whisper-api
• (Not maintained) https://github.com/toverainc/willow-inference-server

Mobile

• https://github.com/argmaxinc/WhisperKit
  • Can run on iOS
  • Difference with whisper.cpp
    • Comparison to Whisper.cpp argmaxinc/WhisperKit#250
    • Major difference with whisper.cpp? argmaxinc/WhisperKit#109
  • Support fine-tuned models: Is there a way to use fine-tuned whisper models? argmaxinc/WhisperKit#369
  • MLX support: MLX model support argmaxinc/WhisperKit#124
  • Wait to support newer swift-transformers
    • WhisperKit and MLX-Swift clashing argmaxinc/WhisperKit#366
    • Can swift-transformers support version 0.1.13 and above and swift-argument-parser support version 1.4.0 and above? argmaxinc/WhisperKit#290
• https://github.com/ggml-org/whisper.cpp
  • Can run on iOS
  • Support word-level timestamps
  • Cannot use fine-tuned model easily: Fine-tuning whisper.cpp models ggml-org/whisper.cpp#2022
• (Not support Whisper today) https://github.com/ml-explore/mlx-swift-lm
  • From Apple
  • Can run on iOS
  • Issues
    • Has anyone successfully run Whisper in Swift? ml-explore/mlx-swift-examples#146
    • A more library-style name? ml-explore/mlx-swift-examples#419
    • Can you rename this library to mlx-swift-inference or something? ml-explore/mlx-swift-examples#389

Model

• https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2
  • https://parakeettdt.com/

Evaluation

• https://github.com/jitsi/jiwer
• https://github.com/analyticsinmotion/werpy
• https://github.com/leo19941227/speech_metrics
• https://github.com/psambit9791/pysnr
• https://github.com/maxmarsc/pyaudioaliasingmetrics
• (Not maintained) https://github.com/jpuigcerver/xer
• (Not maintained) https://github.com/belambert/asr-evaluation

Star History

Communication

• https://github.com/gradio-app/fastrtc