Importing external memory as an image while preserving the contents

[mahkoh]

VUID-VkImageCreateInfo-pNext-01443:

If the pNext chain includes a VkExternalMemoryImageCreateInfo or VkExternalMemoryImageCreateInfoNV structure whose handleTypes member is not 0, initialLayout must be VK_IMAGE_LAYOUT_UNDEFINED

VUID-VkImageMemoryBarrier-oldLayout-01197

If layouts are not ignored, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier

Therefore, unless there is some other way to change the layout of the image subresource, it seems that an image created from external memory MUST initially be transitioned with oldLayout=VK_IMAGE_LAYOUT_UNDEFINED.

However, existing applications instead transition from oldLayout=VK_IMAGE_LAYOUT_GENERAL if they want to preserve the image contents. I've talked to mesa developers and it seems that oldLayout=VK_IMAGE_LAYOUT_GENERAL is supposed to be used for this usecase with oldLayout=VK_IMAGE_LAYOUT_UNDEFINED being incorrect.

The assumptions seem to be twofold:

• Since the contents of the memory have already been defined via some other mechanism (e.g. some other process has used that memory as an image and rendered to it), the memory itself is morally already in the VK_IMAGE_LAYOUT_GENERAL layout.
• The driver does not require the initial transition from VK_IMAGE_LAYOUT_UNDEFINED for bookkeeping. All work that it might do during this initial transition must instead already be done during vkCreateImage (except that the memory must not be modified).

I hope this can be clarified in the spec.

[mahkoh]

One possibility might be to allow initialLayout=VK_IMAGE_LAYOUT_GENERAL when importing external memory.

[cubanismo]

The mesa devs are incorrect. When developing the internal memory extensions, it was agreed within the working group, which included mesa developers, that undefined was the correct layout to use, and in this special case of the initial transition of an image bound to external memory from undefined->[other layout], implementations must ensure the image content is preserved.

[mahkoh]

in this special case of the initial transition of an image bound to external memory from undefined->[other layout], implementations must ensure the image content is preserved.

Is this mentioned anywhere in the spec?

Wlroots, other wayland compositors that use vulkan, and GTK currently transition from GENERAL. If some or all mesa drivers destroy the surface contents when transitioning from UNDEFINED (whether that is incorrect or not), then these applications cannot switch to using UNDEFINED.

[cubanismo]

Sorry, I misremembered. I was mixing up initialLayout and the first layout transition. The correct behavior is, as VU-01443 quoted above describes, the image must be created with the UNDEFINED layout. However, that will not affect the actual layout of the image data.

As defined in the Memory Aliasing section:

Aliases created by binding the same memory to resources in multiple Vulkan instances or external APIs using external memory handle export and import mechanisms interpret the contents of the memory in consistent ways, and data written to one alias can be read by the other alias.

So if you alias memory from some other Vulkan image using external memory, or from a compatible external resource (E.g., OpenGL using GL_EXT_memory_object but NOT using EGL and EGLImage), the layouts are identical between each alias, regardless of what you pass in to initialLayout. E.g., if you create a Vulkan image in instance one and populate it there, then transition it to the VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL layout, exported the memory and imported it to Vulkan instance 2 and created a new identical Vulkan image with initialLayout = VK_IMAGE_LAYOUT_UNDEFINED there and bound it to the imported memory, the actual layout of that new image would be VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, and it could be accessed as such (Following the usual queue family transition away from the special external queue).

Finally, issue 2 in the VK_EXT_external_memory extension appendix states the following:

2. Are new image layouts or metadata required to specify image layouts and layout transitions compatible with non-Vulkan APIs, or with other instances of the same Vulkan driver?

RESOLVED: No. Separate instances of the same Vulkan driver running on the same GPU should have identical internal layout semantics, so applications have the tools they need to ensure views of images are consistent between the two instances. Other APIs will fall into two categories: Those that are Vulkan compatible (a term to be defined by subsequent interopability extensions), or Vulkan incompatible. When sharing images with Vulkan incompatible APIs, the Vulkan image must be transitioned to the VK_IMAGE_LAYOUT_GENERAL layout before handing it off to the external API.

Which reinforces the above and notes, as Mesa devs claim apparently, that images from incompatible external APIs are considered to be in the GENERAL layout. I assume that's what they're referring to.

Sorry for the bad info above.

[mahkoh]

Sorry for the bad info above.

Don't mention it.

E.g., if you create a Vulkan image in instance one and populate it there, then transition it to the VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL layout, exported the memory and imported it to Vulkan instance 2 and created a new identical Vulkan image with initialLayout = VK_IMAGE_LAYOUT_UNDEFINED there and bound it to the imported memory, the actual layout of that new image would be VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, and it could be accessed as such

That is very interesting and a much stronger statement than what I expected. I don't immediately see how that follows from the part you quoted, though:

Aliases created by binding the same memory to resources in multiple Vulkan instances or external APIs using external memory handle export and import mechanisms interpret the contents of the memory in consistent ways, and data written to one alias can be read by the other alias.

In my mind, the image layout has two components:

1. It affects the memory layout in GPU memory.
2. It affects image metadata in the Vulkan driver itself.

I imagine the transition from the foreign or external queue with srcLayout=dstLayout is supposed to take care of populating the user-space metadata if it exists?

---

Maybe this could be clarified somewhere in the descriptions of the external and foreign queues? Something like:

If an image memory barrier defines a queue transfer from the external or foreign queue and the image memory is shared with another API and the other API has modified the image layout of the shared memory, then the image layout of the image subresource immediately before the transfer is the last layout defined by the other API.

I think this would answer my question from my initial post:

Therefore, unless there is some other way to change the layout of the image subresource,

A: Yes, this can happen while the image is in the foreign or external queue.

[cubanismo]

Maybe this could be clarified somewhere in the descriptions of the external and foreign queues? Something like: [...]

I don't see why this case should be singled out for clarification. The same "interpret the contents in consistent ways" language is used to define the exact same interaction between locally aliased images. The spec has no concept of image layouts as driver metadata. That's why layouts need to be specified explicitly at so many locations in the API.

[mahkoh]

I hadn't yet seriously considered the case where memory is aliased within the same VkDevice. I agree my suggestion doesn't make sense because it doesn't cover that case.

I've reread parts of the spec and I found this useful sentence which seems to confirm what you're saying:

The layout of each image subresource is not a state of the image subresource itself, but is rather a property of how the data in memory is organized,

And yet for whatever reason I had been under the impression that this was not the case. I think part of this is the VkImageCreateInfo::initialLayout parameter. If the image layout is only a property of the memory, what is the purpose of this parameter? It seems to be unused in mesa drivers.

There is also this confusing sentence in the spec:

Upon creation, all image subresources of an image are initially in the same layout, where that layout is selected by the VkImageCreateInfo::initialLayout member.

Does this not contradict the previous quote? Let's say I attach memory to an image (A) and transition that image to GENERAL. Then I attach the same memory to another image (B) and I have to specify UNDEFINED in the initialLayout of B. But the layout is not UNDEFINED.

I think the only correct interpretation of that is that UNDEFINED in initialLayout does not "select" anything but indicates that the layout is unknown at this point in time.

However, even then the sentence still contradicts the first quote. Namely this part:

Upon creation, all image subresources of an image are initially in the same layout

I could have used image A to transition different subresources to different layouts before creating B. Therefore the subresources are not all in the same layout.

Does that make sense?

Edit: Oh, and how can image subresources have a layout upon creation of the VkImage if no memory has been attached to the image yet?