Upgrade to pyannote-audio 4

[borgoat]

There's a couple of new pyannote models:¹ pyannote/speaker-diarization-community-1 (offline) and pyannote/speaker-diarization-precision-2 (hosted by pyannote)

I did a minimal upgrade to pyannote-audio 4.0 here to be able to use it, although I believe to make it work properly we probably need additional arguments: the token parameter changed now since one may have to provide a pyannote AI token to use their cloud model.

Footnotes

1. https://www.pyannote.ai/blog/community-1 ↩

[hbredin]

That's great @borgoat.

FYI, I just released version 4.0.1 of pyannote.audio that fixes support for pyannoteAI premium diarization models.

Therefore, once you update this PR to use 4.0.1, running the following command will perform transcription locally but diarization on pyannoteAI cloud with state-of-the-part Precision2 model.

whisperx --diarize \
         --diarize_model pyannote/speaker-diarization-precision-2 \
         --hf_token {pyannoteAI-api-key} \
         audio.wav

{pyannoteAI-api-key} can be obtained from dashboard.pyannote.ai (you'll automatically get a bunch of free credits).

Enjoy!

[stdweird]

@borgoat any chance of rebasing your branch and resolving the conflicts?

[borgoat]

I rebased it now, there are just a couple of things to note:

• I had to set 3.10 as the minimum python version (same as pyannote)
• Right now we're using the HF token for pyannote too - I guess that's incorrect @hbredin assuming one may want to use pyannote.ai?

[ErikHeggeli]

Just got it to work, but had to up torch and python. Otherwise throwing "std::bad_alloc"
Also offline use is way easier with this new version.
Got a working version in my forked repo.

[stdweird]

@ErikHeggeli i am mainly switching to 4 to make it work offline. if i look in your test branch, the other main difference is the modified yaml file. what is that about? is that something required and should it be included here? (or perhaps you can open PR for it?)

[ErikHeggeli]

@ErikHeggeli i am mainly switching to 4 to make it work offline. if i look in your test branch, the other main difference is the modified yaml file. what is that about? is that something required and should it be included here? (or perhaps you can open PR for it?)

Can not see the yaml file you are talking about, what is it called?

[ErikHeggeli]

If it is from the test branch it is not needed. That's something needed to make the earlier versions of pyannote work offline.

[stdweird]

@ErikHeggeli nvm. i see that your branch ships everything needed to make it offline, not only the changes for pyannote 4.

[ErikHeggeli]

Yes, clone the main branch.
Get the models from HuggingFace:
git clone https://hf.co/pyannote/speaker-diarization-community-1 /path/to/directory/pyannote-speaker-diarization-community-1

Make sure the models are actually downloaded and not just some reference/pointer, just check that the models aren't 1 kb. (This happened to me). Produced this error "_pickle.UnpicklingError: invalid load key, 'v'."

And then in diarize.py provide full path to "/path/to/directory/pyannote-speaker-diarization-community-1", if path is wrong you will get "huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name'"

Then it should work offline. Only thing I haven't been able to test is GPU offloading, only have CPU available atm.

[stdweird]

wrt the reference pointer, you need git-lfs installed.
wrt offline, euhm, looks like regular whipser usage pulls in at least on file from torch hub and per language one other model, so still puzzling a bit; but i'll get there. it's much easier than reverse engineering needed to make pyannote 3.X work offline ;)

[ErikHeggeli]

Yes that was most likely the alignment model (or whisper asr model). You have to download and give the path to those as well, but that always worked as intended offline from my experience.

[hbredin]

• Right now we're using the HF token for pyannote too - I guess that's incorrect @hbredin assuming one may want to use pyannote.ai?

Using --hf_token {pyannoteAI-api-key} should work just fine.
See https://huggingface.co/pyannote/speaker-diarization-precision-2#usage

[to-audiobook]

Just got it to work, but had to up torch and python. Otherwise throwing "std::bad_alloc" Also offline use is way easier with this new version. Got a working version in my forked repo.

thanks to @hbredin I just learned the std::bad_alloc exeption is caused by incompatibilities between torch and torchcodec versions. Because of that we'd better force specific torchcodec versions, depending on which torch version you guys decide to use. torchcodec's github page has a table showing the versions compatibilities.

[GUUser91]

i had to edit /mnt/2tb/whisperX/whisperx/diarize.py
and replace

self.model = Pipeline.from_pretrained(
    model_config, use_token=use_auth_token
).to(device)

with

self.model = Pipeline.from_pretrained(
    model_config, token=use_auth_token
).to(device)

for whisperx to work

[boegel]

Any updates on finalizing this?

[stdweird]

@borgoat can you fix the extra code from the @GUUser91 comment. it's needed for pyannote 4.0.3 (and i assume later versions)

[edihasaj]

this is an awaited feature! would be very good to have it

[zckrs]

friendly ping on this PR @Barabazs 🙏

[edihasaj]

would be good to move this forward @Barabazs

[m-bain]

Sorry can look into this can you resolve the conflict and ill merge

[zckrs]

If needed chore: upgrade pyannote-audio 4 (rebased) 👼

[Barabazs]

what's the reason for this change?

[hbredin]

Because pyannote 4.x no longer returns a single pyannot.core.Annotation object.
It now returns a structured output with speaker_diarization: Annotation and speaker_embeddings: np.ndarray attributes.

[Barabazs]

Thank you!

[Barabazs]

Closing in favor of #1349