[Bug] Torch conflict when using vllm-plugin-FL on BI-V150

[shh2000]

1. Iluvatar release their vllm0.11+corex, which can serve Qwen/Qwen3-0.6B correctly
2. Using the release image of vllm0.11+corex, I cannot compile vllm0.13.0, which is required by vllm-plugin-fl, because of NVCC or CUDA's version
3. Fortunately, I tried directly download and install vllm0.13.0 wheel built on NVIDIA with "--no-deps" and it worked. However, when serving model like Qwen/Qwen3-0.6B, it triggered exception at the very beginning "from vllm.entrypoints.cli.main import main"

File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 42, in <module>
    import torch.distributed._symmetric_memory
  File "/usr/local/corex-4.4.0/lib64/python3/dist-packages/torch/distributed/_symmetric_memory/__init__.py", line 16, in <module>
    from torch._C._distributed_c10d import _SymmetricMemory, Work as _Work
ImportError: cannot import name '_SymmetricMemory' from 'torch._C._distributed_c10d' (unknown location)
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

I think it is the conflict between vllm0.13.0 and torch-corex, which is repaired in vllm-corex series.
4. Thus, I tried install vllm-plugin with FlagCX and FlagGems. Though some small exception occured, there's still trivial methods to avoid or fix them. After that, I have an environment with:

vllm                              0.13.0
vllm_fl                           0.0.0
flag_gems                         4.2.1rc0
flagcx                            0.8.0              /home/FlagCX/plugin/torch

5. Unfortunately, no matter I enable or disable FlagCX or FlagGems, the conflict between vllm/vllm-fl and torch-corex still exists

I wonder if there is any help about solving the mismatch between vllm-plugin-fl and multi-backend torch, like torch-npu, torch-metax, torch-corex? Each torch-backend may have their own APIs to cooperate with their SDK/Driver/..., not only Computing operators covered by FlagGems and Communicating operators covered by FlagCX.

[ceci3]

It seems that the torch in corex doesn't yet support symmetric_memory. The _symmetric_memory doesn't actually use in this file; it's just import and according comment, it seems move _symmetric_memory source code to this file. Also, symmetric_memory is called in other files via try-except. It seems the issue is caused by the lack of distinction between importing into the current file in the vllm library. So you can simply comment out this import line here.

[shh2000]

It seems that the torch in corex doesn't yet support symmetric_memory. The _symmetric_memory doesn't actually use in this file; it's just import and according comment, it seems move _symmetric_memory source code to this file. Also, symmetric_memory is called in other files via try-except. It seems the issue is caused by the lack of distinction between importing into the current file in the vllm library. So you can simply comment out this import line here.

Thanks for you advice! I comment out the line with import _symmetric_memory, however, there's new exception with

File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 821, in cutlass_scaled_mm_supports_fp8
    return torch.ops._C.cutlass_scaled_mm_supports_fp8(cuda_device_capability)
  File "/usr/local/corex-4.4.0/lib64/python3/dist-packages/torch/_ops.py", line 1267, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' '_C' object has no attribute 'cutlass_scaled_mm_supports_fp8'

, still at the very beginning "from vllm.entrypoints.cli.main import main". It would be of great time&corex-server cost to test and comment out all conflict lines conflicted with torch-corex. If there're any high-level suggestion?

[shh2000]

After manually comment out 4 lines, I can finally avoid exception at the very beginning. However, there are still some lines that throw errors, and it appears they cannot be simply commented out like:

File "/usr/local/lib/python3.10/site-packages/vllm_fl/ops/activation.py", line 10, in __init__
(EngineCore_DP0 pid=1393037)     super().__init__()
(EngineCore_DP0 pid=1393037)   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/activation.py", line 75, in __init__
(EngineCore_DP0 pid=1393037)     self.op = torch.ops._C.silu_and_mul
(EngineCore_DP0 pid=1393037)   File "/usr/local/corex-4.4.0/lib64/python3/dist-packages/torch/_ops.py", line 1267, in __getattr__
(EngineCore_DP0 pid=1393037)     raise AttributeError(
(EngineCore_DP0 pid=1393037) AttributeError: '_OpNamespace' '_C' object has no attribute 'silu_and_mul'
(EngineCore_DP0 pid=1393037) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W128 09:23:05.142905504 ProcessGroupNCCL.cpp:1518] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

at

File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1393037)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1393037)   File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1393037)     super().__init__(

Any advice about these kind of exception?

[shh2000]

After manually comment out 4 lines, I can finally avoid exception at the very beginning. However, there are still some lines that throw errors, and it appears they cannot be simply commented out like:

File "/usr/local/lib/python3.10/site-packages/vllm_fl/ops/activation.py", line 10, in __init__
(EngineCore_DP0 pid=1393037)     super().__init__()
(EngineCore_DP0 pid=1393037)   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/activation.py", line 75, in __init__
(EngineCore_DP0 pid=1393037)     self.op = torch.ops._C.silu_and_mul
(EngineCore_DP0 pid=1393037)   File "/usr/local/corex-4.4.0/lib64/python3/dist-packages/torch/_ops.py", line 1267, in __getattr__
(EngineCore_DP0 pid=1393037)     raise AttributeError(
(EngineCore_DP0 pid=1393037) AttributeError: '_OpNamespace' '_C' object has no attribute 'silu_and_mul'
(EngineCore_DP0 pid=1393037) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W128 09:23:05.142905504 ProcessGroupNCCL.cpp:1518] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

at

File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1393037)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1393037)   File "/usr/local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1393037)     super().__init__(

Any advice about these kind of exception?

The source code of these new exception is:

class SiluAndMul(CustomOp):
    """An activation function for SwiGLU.

    The function computes x -> silu(x[:d]) * x[d:] where d = x.shape[-1] // 2.

    Shapes:
        x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d)
        return: (num_tokens, d) or (batch_size, seq_len, d)
    """

    def __init__(self):
        super().__init__()
        if current_platform.is_cuda_alike():
            self.op = torch.ops._C.silu_and_mul
        elif current_platform.is_xpu():
            from vllm._ipex_ops import ipex_ops

            self.op = ipex_ops.silu_and_mul
        elif current_platform.is_cpu():
            self._forward_method = self.forward_native

I think these should already be replaced by FlagGems right? But it occured when USE_FLAGGEMS=1 😭