Frame ghosting

[AIisCool]

As I'm sure we've all seen, despite AMAZING results, SEEDVR2 does have an issue; whether it's hardcoded in the model or something to correct via code: when there is medium to fast motion between frames, you can see ghosting/afterimage of the previous frame which leads to distortions for really fast motion. I can also see a similar ghosting/distortion issue with Topaz's Starlight model.

So the point of this discussion is: What is causing it and how do we correct it if such a task is possible?

[naxci1]

The batch_size should be as large as possible. For example, it used to be 101 for me, but I've now set it to 121, and these problems don't occur. I don't recommend going below 101. Depending on your system hardware, the higher you set it, the better the results you'll get. I also set Taled_vae to 768. This is also important; a higher number results in better quality and faster processing.

[naxci1]

[AIisCool]

@naxci1 Tried those exact setting and the issue is still there for me.

Input:

Output:

You can see his left arm is distorting with the plant behind him. If you wish to try with that video, I uploaded it here.

[naxci1]

The link did not work. If the video is corrupted like this, SeedVR2 cannot fix it unfortunately. However, if the video is not corrupted like this and SeedVR2 corrupts it, I think it can be fixed. Upload the original video again.

[AIisCool]

Not sure why the link did not work but I tried again.

No corruption in the input video, it only becomes distorted after processing.

[naxci1]

input video:

The hand is already moving out of place in the intro video. Is this a low-fps video?

On mega.nz, it only shows uploading; I guess there's no download permission.

[naxci1]

It would have been more useful to deinterlaced the video beforehand.

[AIisCool]

No, the FPS is 23.976. As for deinterlacing it's from a Blu-ray download that I resized the resolution down by 50%. That said I have seen the same distortion issue with pure progressive video so interlacing isn't the cause.

Not sure why mega isn't working but I tried proton this time.

[naxci1]

I downloaded the video, deinterlaced it, but it didn't work. The video was corrupted when re-rendered. It would have been easier if it were the original video. I'll run tests with it and let you know the results, but if it were the original video, the DVD version, this problem wouldn't have happened.

[naxci1]

What is perceived as distortion as the hand moves in the image is actually a common video and photography phenomenon known as "motion blur."

[naxci1]

https://yandex.kz/video/preview/3174338796135403253

I also found the 1080p version of the movie, the same scene has the same problem, since the DVD ripping is done in lower quality, all videos have this problem and it cannot be fixed.

[AIisCool]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

I tried another video. You can see the pattern on her skirt is all distorted after SEEDVR processing. Yes there is motion and it's blurry as she moves, but in the original the pattern isn't distorted.

Input:

SEEDVR2:

[mindplay-dk]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

Huh, now in this example I definitely see what you're talking about - it's as though it's painted the leaves from the plant onto his arm.

If I had to guess, I'd say the input is probably too blurry for the model to make sense of, and so it makes a "wrong best guess".

On the skirt in the other example, I don't see a problem - just looks like motion blur in the input and the output, right?

[AIisCool]

Yes the leaves into his arm is the issue I've been trying to convey. As for her skirt, while yes it's in motion, the pattern looses it's definition that while in motion in the original, doesn't become blurred/distorted.

[naxci1]

share workflow

[naxci1]

That's all I could fix. Unfortunately, it can't fix motion blur. Topaz couldn't fix it either, so I tried that too. It would be nice if this problem was added to the new model.

6_apo8_nyx3_yeni_00002.mp4

[mindplay-dk]

I don't think we're supposed to "fix motion blur"? Fast movement in 24-25 FPS film should have some motion blur, it makes the perceived motion feel smooth - modern video games go as far as adding motion blur to give it a cinematic feel. :-)

[naxci1]

Actually, the title needs to be changed, the problem is "motion blur"

[adrientoupet]

It would be even more impressive if it had been officially released and I had time to make an updated video about it to document everything :) but thank you, appreciate the support... hopefully we'll get there soon!

[AIisCool]

You're welcome, I wish I was far more knowledgeable and able to assist.

[AIisCool]

Hi @adrientoupet I had a thought about the distortions. When I tried upscaling a video using UltimateSDUpscale and tied in the WAN 2.2 model and it's VAE, it didn't have any distortions. I wonder if this may help, if it's possible to somehow integrate WAN's temporal consistency. This is of course assuming the distortions can't be fixed via coding and isn't just part of the SEEDVR model.

[naxci1]

@AIisCool

Could you share your workflow? It might be due to the WAN model difference.

[AIisCool]

Upscale Image.json

[adrientoupet]

Thank you @AIisCool - I suspect the distortions should be fixable in the code, but I'll report back once I had a chance to dive deeper into it.
Out of curiosity, how does ultimateSDUpscale compare to SeedVR2 ?

[AIisCool]

It was decent, but took too much to try and keep it from changing the input. That said the one thing I liked was that it wasn't causing color banding, which I see happening on the output of SEEDVR2 at times even though I have it set to save as ProRes 4444XQ

Input:

SEEDVR2:

I even tried using DVO Clarity to reduce the banding in the input clip, but it still comes out with more pronounced banding from SEEDVR2.

I think it's due to SEEDVR2 not being trained on degraded input, but perhaps there's a way to reduce the banding?

[AIisCool]

@adrientoupet I found another good example of this while testing SEEDVR2 on some traditional 2D animation.

Everything is fine in this frame:

But then the next fame has the "ghost" of the outlines of the previous around the character:

Not present on the original input:

I've uploaded the original clip here should you wish to test with it.

[mindplay-dk]

I wonder if this is an edge case for animation, which doesn't always contain "motion" in the traditional sense.

I wonder if it's even a good idea to have the same model trained on both film and animation? Wildly different types of motion... like, I wonder what the model learns from seeing duplicated frames, which are often present in (esp. traditional) animated content? 🤔

[mindplay-dk]

Oh, interesting, this particular clip in fact does not appear to have duplicate frames - it was created at the native frame rate of the original film. Hmm, I wonder if the artifacting you're seeing here is something it learned from animated content with duplicate frames? It's not unlikely that (the wrong kind of) compression could create artifacts like that while attempting to compensate for choppy motion?

[adrientoupet]

Hi @AIisCool - thank you for sharing the clip, I'll give it a go and report back.

[adrientoupet]

Tried it, I'm able to reproduce the issue. Unfortunately I don't have a fix for it just yet.
Need to look into temporal overlap/prepend as discussed here #217 (comment) to see if that could help, or do further investigations. That will be for next time as I don't have bandwidth to look into it for the coming release. Thank you

[AIisCool]

Tried it, I'm able to reproduce the issue. Unfortunately I don't have a fix for it just yet. Need to look into temporal overlap/prepend as discussed here #217 (comment) to see if that could help, or do further investigations. That will be for next time as I don't have bandwidth to look into it for the coming release. Thank you

I'm glad it's reproducible, and not just a "me" issue.
No, thank you for continuing to put endless hours into improving this! Please no rush. I hope that whether it comes in November, December, or January, it is fixable.

[AIisCool]

@adrientoupet I had a thought, though it's of course completely uneducated on the matter. I notice that SEEDVR2 makes immobile objects very clear, along with the backgrounds that are stationary become very clean/clear. Perhaps it's overcompensating and that is the cause of the "ghosting" when something/someone moves and is then for super-fast movement it cannot deal with the temporal shift?

[adrientoupet]

I'm curious to see if you're getting more luck with the last version ? Using batch size long enough to be temporally consistent but not too long that they create issues, enable uniform batch size, add temporal overlap, prepend... Let me know when you have a chance, thank you.

[R3G4L101]

On 2.5 I am getting the after image ghost trails with movement. Makes the quality look distorted. Previous nightly build doesn't have this effect on the same videos.

[adrientoupet]

Can you share workflow ,& video please so I can troubleshoot?

[R3G4L101]

Here you go, workflow attached at bottom. I took another look and it is in both, but the trailing is more pronounced in 2.5. I never noticed from before because the trailing is more subtle in motion. Used the same seed for both.


SeedVR2.5 Video TEST.json

[adrientoupet]

Thank you very much @R3G4L101 - Any chance you can upload your input footage somewhere so I can troubleshoot on the same thing? Cheers

[R3G4L101]

SeedVR2.Comparison.mp4

Comparison vid

[naxci1]

Hi @R3G4L101

[adamfluty]

Hi, I am also having this issue. I have both detelecined (30 -> 24 fps) and deinterlaced the raw DVD rip. It is clean to start with as seen in the first image. Batching at all, whether 9 or 121, it creates major "leak" between frames. 90% of the time its previous frames being retained and I have even seen future frames leak in as well.
Ready-to-upscale

Batch 9

Here you can see where the brim of his hat and fingers will be in the NEXT frame

With batch 121, the issue is more localized to the moving character, not the space around them

Batch 0 makes each frame flawless but with bad shimmering in the video, as you would expect.
I am not tiling, and I have played with the temporal overlap as well with no improvement.

[thehhmdb]

I've noticed this issue as well.

[thehhmdb]

Did anyone find any fixes or workarounds for the frame ghosting?

[cd1188]

I have the same problem when people are talking. There are slightly blurry and darker frames in the face.
It's definitely not motionblur! Even with constantly moving objects, there is a flicker.
I set the batch size to 51 and I feel like every fourth frame is sharp and then the following frames become blurry...
Seems to be a pattern of 4 or 5 frames?
As a test I have also set the temporal-overlap to 0 and its the same bad flicker.
I noticed that when I remove a frame at the beginning of the video, the sharper images also shift by one frame.
I had already considered rendering the video four times and then extracting the sharp images, but I don't know if that will work.
I think that would break the motion consistency if I combine different videos.

Edit: extracting every 4th frame is difficult, because the pattern is only consisten within each batch.
The problem is that each batch has a odd frame count and after each batch, you have to shift the extraction pattern for one frame.
-> temporal overlap of 1 helps here to be in the pattern of 4 frames for the next batch

Edit2:
The Idee from extracting each 4th frame from 4 different videos with shifted start-frame is also not good.. you see now shimmering on each frame.