BaseTester: support plugin EPs with compiled nodes and registered kernels #27176
Conversation
adrianlizarraga
changed the title
| const auto& provider_type = ep.Type(); | ||
|
|
||
| constexpr std::array kEpsThatDoNotUseKernelRegistry{ | ||
| constexpr std::array kEpsThatDoNotUseKernelRegistryForAllNodes{ |
There was a problem hiding this comment.
maybe call them "EPs that compile nodes" instead of "EPs that do not use kernel registry for all nodes"
| constexpr std::array kEpsThatDoNotUseKernelRegistryForAllNodes{ | ||
| kOpenVINOExecutionProvider, | ||
| kTensorrtExecutionProvider, | ||
| kTensorrtExecutionProvider, // uses kernel registry for Memcpy* nodes, but compiles all others. |
There was a problem hiding this comment.
just to confirm, the plugin EP name will be different, right? that assumption is made here:
onnxruntime/onnxruntime/test/unittest_util/base_tester.cc
Lines 689 to 691 in f83d4d0
There was a problem hiding this comment.
Good question. @chilo-ms do you know if the plugin EP name is different?
There was a problem hiding this comment.
I assume so, but let me confirm with Nvidia for their plugin TRT RTX EP.
Also, do we want to include provider-bridge TRT RTX EP, kNvTensorRTRTXExecutionProvider, in this list as well?
There was a problem hiding this comment.
we should only add kNvTensorRTRTXExecutionProvider to this list if its name differs from the plugin EP version, right?
There was a problem hiding this comment.
we should only add kNvTensorRTRTXExecutionProvider to this list if its name differs from the plugin EP version, right?
Yes, waiting for Nvidia's reply for the name of their plugin TRT RTX EP, will update.
There was a problem hiding this comment.
@ishwar-raut1 as you mentioned offline, the name to register EP DLL is also set for plugin TRT RTX EP name. Could you help confirm that the EP name doesn't conflict with “TensorRTExecutionProvier” and "NvTensorRTRTXExecutionProvier” (will be added) in the kEpsThatCompileNodes list?
There was a problem hiding this comment.
The name to register the EP DLL is also set for the plugin TRT RTX EP name. if the same is used to register the dll then there could be a conflict.
There was a problem hiding this comment.
To run onnxruntime_provider_test with plugin TRT RTX EP, the plugin EP name must not conflict with NvTensorRTRTXExecutionProvider; otherwise, the test will fail at this check that Ed mentioned:
onnxruntime/onnxruntime/test/unittest_util/base_tester.cc
Lines 689 to 691 in f83d4d0
There was a problem hiding this comment.
I think we can conditionally add NvTensorRTRTXExecutionProvider to all_provider_types only if USE_NV is defined, which occurs only when the provider-bridge NV TRT RTX EP is built with ORT.
The result would be that this error would only occur if the user tried to use both the provider-bridge and plugin EPs at the same time (with a build that included the provide-bridge EP).
…hen ORT is built with the provider-bridge version of the EP
Description
Updates the
BaseTesterclass used by theonnxruntime_provider_testtool to support plugin EPs that use a kernel registry but compile other nodes. For example, TRT EP only uses registered kernels for Memcpy* nodes, but compiles every other node.Without this change, plugin EPs that use a mix of compiled nodes and registered kernels cannot be tested with
onnxruntime_provider_test.Motivation and Context