Questions about training data format and preprocessing / filtering

[undobug]

Hi, thank you for sharing this impressive work and for releasing the code!

I have some questions regarding the training data format and preprocessing pipeline, and I hope you could provide some clarification or guidance.

1. Training sample length and preprocessing

I noticed that in example/trainset_example/fps25_mp4, the provided training samples are relatively short, around 5–8 seconds each.
Are these clips already preprocessed and segmented from longer raw videos? (In the paper, the average video length is reported to be around 150 seconds)

2. Data filtering and quality control

Would it be possible to share data filtering scripts, or at least provide more detailed recommendations? In particular:

Face size / scale:
You mentioned that overly large faces may negatively affect lip-sync accuracy (#73). Is there a recommended face size range (e.g., relative to image resolution)? Do you filter samples based on face bounding box size?

Head motion:
Is there any filtering based on head pose variation (pitch / yaw / roll)? Do you remove samples with excessive head movement or large pose changes?

Other filtering criteria:
Do you apply any filtering based on audio–visual sync confidence, face detection stability, image quality, occlusion, or blur?

[RobinROAR]

Hi, thank you for sharing this impressive work and for releasing the code!

I have some questions regarding the training data format and preprocessing pipeline, and I hope you could provide some clarification or guidance.

1. Training sample length and preprocessing

I noticed that in example/trainset_example/fps25_mp4, the provided training samples are relatively short, around 5–8 seconds each. Are these clips already preprocessed and segmented from longer raw videos? (In the paper, the average video length is reported to be around 150 seconds)

2. Data filtering and quality control

Would it be possible to share data filtering scripts, or at least provide more detailed recommendations? In particular:

Face size / scale: You mentioned that overly large faces may negatively affect lip-sync accuracy (#73). Is there a recommended face size range (e.g., relative to image resolution)? Do you filter samples based on face bounding box size?

Head motion: Is there any filtering based on head pose variation (pitch / yaw / roll)? Do you remove samples with excessive head movement or large pose changes?

Other filtering criteria: Do you apply any filtering based on audio–visual sync confidence, face detection stability, image quality, occlusion, or blur?

First, thank you for your support！
The data samples we provided are only intended to help run the training script and are not related to the actual training data. Below, I will briefly introduce the processing of training data.

Since Ditto trains DiT in motion space and essentially aims to select the most common facial motion representations, it has high requirements for the quality of training videos. The training data needs speakers whose lip movements, expressions, and head movements match the content and rhythm of their speech, with no excessive displacement, content-irrelevant facial movements, or occlusions during recording. In actual training, we mainly used professional anchor broadcast videos recorded in studio green screen environments, with fixed camera and model positions. The content was primarily news broadcasts, supplemented by talk shows and singing. Due to the high quality of the data itself, we segmented long videos into approximately 2-minute clips and only discarded some segments through face detection. We also incorporated some open-source data such as VFHQ and HDTF to supplement audio diversity. For such data, we used SyncNet to filter lip-sync consistency, which also indirectly filtered out some data with large side angles. Due to portrait rights and other reasons, we apologize that we cannot make the training set public. We hope the above information is helpful for your research.