Question about the mismatch between training and inference prompts

[lky-violet]

Very impressive work! When I saw that the understanding encoder is used to compress image information into a “text prompt”, it immediately reminded me of the recent deepseek-ocr idea. Your work is truly inspiring.

I have a few questions I would like to discuss:

The core idea of the paper is that the image understanding encoder itself can serve as an effective image description tool, converting sparse textual descriptions into dense “textual prompts” that better guide image generation. However, as mentioned in Section 2.2, in the text-to-image task, there is no reference image during inference, only the instruction prompt. Meanwhile, according to Equation (3), in RecA training, the instruction prompt is not involved. This leads me to the following concerns:

1. Does this introduce a mismatch between training and inference? Or is the output of the image understanding encoder implicitly encoding the original instruction prompt information?
2. Traditionally, T2I training maps a short instruction prompt to a detailed image. RecA instead trains using dense understanding prompts to generate detailed images, but during inference it can still generate detailed images from short instruction prompts. Do you think this improvement comes from modifying the internal knowledge of the UMM? Since if an instruction is short, shouldn’t the generated image also be less detailed? Could this be a limitation of current benchmarks?

I may have misunderstandings since I am new to UMMs, and I would greatly appreciate your clarification. Thank you again for the excellent work!

[HorizonWind2004]

Thanks for your insightful question! Yes, the core idea of RecA is somewhat similar to that of Deep-OCR.

(1) Yes. I think the most interesting part is this mismatch. when tuned only on I2I data, model's T2I performance has improved substantially. I think the improvement stems from "similarity" between text prompt embeddings and image embeddings. Moreover, our ablation study shows that if we mixed the T2I data, the overall performance drops significantly. In other words, the improvement is far less than when no T2I data are integrated.

(2) I agree with your point that this improvement comes from modifying the internal knowledge of UMMs. A better alignment, I guess. T2I is a less-to-more procedure. Even if the prompt is as short as "a photo of a cow", it may also render a lot of irrelevant details, such as grassland and blue sky.