Struggling deleting duplicates

[mstoffel-sag]

Hi,

great project, thanks for that. But I really struggle with the workflow. I have scanned 75k pictures from two users. I have duplicates in there with even the same filename. I selected username via autoselect to add 3. I expect only two duplicates for each picture.

Somehow I don't get the whole multimode group thing. I just want to select all duplicates that I want to keep. Every time I do find similar it takes very long and then im only presented with one group and a lot of pending. When I resolve these. It does the scan again.... Sorry that I dont get it.

[RazgrizHsu]

Hi, thanks for trying out the project.

I noticed the threshold in your screenshot appears to be around 0.63.
This is likely the main cause of the issues you're experiencing.
A threshold that low tends to match photos that look quite different from each other, and it also makes the search process significantly slower.

I'd suggest starting with a threshold around 0.93, which focuses on near-exact duplicates.
This should make the search much faster and the results far more relevant.

Enabling Multi Mode with Max Group set to around 10 would allow you to review multiple duplicate groups at once, rather than going through them one by one.

For the auto selection, setting up your preferred criteria (such as user priority) lets the system automatically pick which photo to keep in each group.
You can then scroll through to verify and use the batch action buttons to process them quickly.

A progressive approach tends to work well for large collections.
Start with a high threshold (0.93+) to clear obvious duplicates first, then gradually lower it to catch more similar photos as needed.

The reason the tool is designed this way is that people have quite different use cases.
Some just want to find exact duplicates, while others might want to discover similar content like memes or burst shots.
I originally built this to find meme images across my library, which required a much lower threshold and a different workflow.
That's why the tool offers flexible threshold and mode settings to accommodate various scenarios.

Here are my settings for reference, using 0.93 threshold with Multi Mode (Max Group: 10) and Exclude SimilarOver > 5.

[mstoffel-sag]

Thanks you very much. These settings are giving me really good results:

But it covers just 100 duplicates of my 70k photos and after I resolve the first group it starts the search again even if there are 30 pending groups....

[RazgrizHsu]

Regarding the "search starts again after resolving" issue.

This behavior is controlled by the "Auto Find Next" checkbox in Search Settings.
When this option is enabled, every time you resolve or delete the current group, the system automatically searches for the next unprocessed photo and places the new result into the current tab.
That's why you're seeing the search start again after resolving.

You can always switch to the pending tab manually to process the groups that were already found.
The pending tab holds all previously searched groups waiting for your review.
You don't have to resolve them through the current tab one by one.

Alternatively, if you prefer to work through all pending groups before any new search happens, you can turn off "Auto Find Next".
With it disabled, after resolving the current group, the system will automatically switch to the pending tab instead of triggering a new search.

Regarding "only covering about 100 photos".

With Max Group set to 10, each Find Similar run returns up to 10 groups.
Each group contains one main photo plus its similar matches, so a single batch typically covers a few dozen to around a hundred photos.
This is the normal amount for one search batch.

The full similarity comparison across 70k photos involves a very large amount of computation.
Comparing every pair requires roughly 2.45 billion vector distance calculations (70,000 × 69,999 / 2).
Even with vector database indexing to speed up queries, searching all 70k photos at once and displaying every result simultaneously would be impractical in terms of computation time, memory usage, and browsing experience.
That's why the system is designed to search and process in batches.

The recommended workflow would be: press Find Similar to get a batch → process all groups in both current and pending tabs → press Find Similar again for the next batch, and repeat until everything is processed.

The https://github.com/RazgrizHsu/immich-deduper#usage-guide in the README has a more detailed walkthrough of the full workflow.

[mstoffel-sag]

Thanks, I will try to clean up by filtering inside of immich. When I need a really a similar comparison I will come back to this great tool