Segmentation Fault in macOS CI Runner Due to MPS Incompatibility

[ansh-info]

Hi @awmulyadi ,

Hope you are doing fine and thank you for your time 😄
I wanted to bring to your attention an issue we've encountered with our CI tests on GitHub Actions, specifically related to the macOS runner. PR #179

Background:

• Previously, we utilized Python's built-in venv for managing virtual environments and installed dependencies using pip.
• Recently, we've transitioned to using Conda as our default environment manager.
• We've also implemented a universal caching mechanism for all agents, ensuring compatibility and successful test runs on all Windows, macOS and Linux platforms.

The Issue:

• While all tests pass on Windows and Linux runners, we're encountering a segmentation fault when running a specific test on the macOS CI runner.
• Locally, on my Mac, all tests, including the problematic one, pass without issues.

Details:

• Test Failure: The test test_utils_embeddings_sentencetransformer.py fails with the following error:

/Users/runner/work/\_temp/7150a77c-6a6e-419c-b0b8-dd1ff5961c22.sh: line 7:  5462 Segmentation fault: 11  coverage run --include=aiagents4pharma/talk2knowledgegraphs/\* -m pytest --cache-clear aiagents4pharma/talk2knowledgegraphs/tests/
aiagents4pharma/talk2knowledgegraphs/tests/test\_utils\_embeddings\_sentencetransformer.py
Error: Process completed with exit code 139.

• Environment: : macos-latest(earlier we were using macos-13, which by default uses cpu build)

Root Cause:

• MPS Backend Limitations

• Link to my fork to view the actions

Proposed Solution:

1. Force CPU Usage in CI:

import os
import torch

device = torch.device("cpu") if os.environ.get("CI") == "true" else torch.device("mps" if torch.backends.mps.is_available() else "cpu")

Conclusion:

By implementing one of the above solutions, we can ensure that our CI tests run reliably across all platforms without compromising the core application logic or test integrity.

Please let me know your thoughts on this approach or if you have any alternative suggestions.

Best regards,
Ansh

[awmulyadi]

Hi, Ansh! Thanks for letting me know regarding this issue. I discovered this issue a couple of months ago when testing T2KG on macos-latest, as I reported here. I haven't had a chance to look at this matter closely, other than observing that people encountering the issue decided to downgrade their libraries (check here).

It is nice that you came up with the proposed solutions there 👍 . However, it is not clear where we should put those lines of code. It would be great to elaborate on this solution by pointing out the file that should be modified in order to resolve such an issue.

Regards,

[ansh-info]

I explored a few solutions for this, and since I'm still new to PyTorch, I’d really appreciate your input.

Initially, I discovered that the CI runners don’t support GPU, and the previous tests were using the CPU build without any issues. When I added Conda and macos-latest, it started assuming Apple Metal support by default—something that isn't available in the CI environment.

Note: From the issue you shared, it seems the problem only appeared after switching to macos-latest. So, in my case as well, it could be due to macos-latest rather than Conda. Previously, we were using macos-13, which is Intel-based and uses x86 architecture, but macos-latest now defaults to the M-series architecture (ARM-based)

To address this, I introduced a conditional check: if the environment variable is set to "CI", it runs on CPU; otherwise, on macOS, it attempts to use Metal.

This approach worked initially, but I realized that historically, we’ve never actually utilized GPU—whether locally or in CI. So, I simplified the logic to always run on CPU, just like before.

Now, everything is handled by a single line:

First version:

I tested this on my Mac, and all tests passed with 100% coverage.

Below is the cleanest version of the fix. Do you prefer this approach?

and optionally in your sentence_transformer.py SentenceTransformer accept a device kwarg, so we can force all weights
to live on CPU right when you load the model(I didn’t do this because I didn’t want to touch the core logic of the application, so I only modified the tests.).

something like this:

self.model = SentenceTransformer(
    self.model_name,
    cache_folder=self.model_cache_dir,
    trust_remote_code=self.trust_remote_code,
    device="cpu",  # Ensures all internal device handling uses CPU
)

so the complete code for test_utils_embeddings_sentencetransformer.py will look like this(withot the optional step):

"""
Test cases for utils/embeddings/sentence_transformer.py
"""

import numpy as np
import pytest

from ..utils.embeddings.sentence_transformer import EmbeddingWithSentenceTransformer


@pytest.fixture(name="embedding_model")
def embedding_model_fixture():
    """
    Fixture for creating an instance of EmbeddingWithSentenceTransformer.
    """
    model_name = "sentence-transformers/all-MiniLM-L6-v1"  # Small model for testing
    embedding_model = EmbeddingWithSentenceTransformer(model_name=model_name)
    # Move underlying model to CPU for testing
    embedding_model.model.to("cpu")
    return embedding_model


def test_embed_documents(embedding_model):
    """
    Test the embed_documents method of EmbeddingWithSentenceTransformer class.
    """
    # Perform embedding
    texts = ["This is a test sentence.", "Another test sentence."]
    embeddings = embedding_model.embed_documents(texts)
    # Check the result
    assert len(embeddings) == len(texts)
    assert len(embeddings[0]) > 0
    assert len(embeddings[0]) == 384
    assert embeddings.dtype == np.float32


def test_embed_query(embedding_model):
    """
    Test the embed_query method of EmbeddingWithSentenceTransformer class.
    """
    # Perform embedding
    text = "This is a test query."
    embedding = embedding_model.embed_query(text)
    # Check the result
    assert len(embedding) > 0
    assert len(embedding) == 384
    assert embedding.dtype == np.float32

I still have to test if it works, but the free credits/trial on my GitHub account have expired, so I couldn't proceed further.

Let me know your thoughts!

[ansh-info]

also the tests for test_utils_enrichments_ols.py fails locally on my mac:

due to assertation ->

original line

    assert descriptions[3].startswith(HP_DESC)

to fix it, I had to write it like this for both
def test_enrich_documents(enrich_obj) and
def test_enrich_documents_with_rag(enrich_obj):

    assert HP_DESC in descriptions[3]

[ansh-info]

Hi @awmulyadi,

I was able to run the tests successfully, and here are the results. After doing some additional investigation, I discovered that macos-latest in GitHub Actions currently points to macOS 14 (Sonoma).

However, I checked my local environment (MacBook M3), which is running macOS 15. I also verified this against the GitHub Actions runner documentation, and it confirms the mapping. The test passed on my machine, so I updated the workflow YAML to explicitly use macos-15 instead of macos-latest. With this change, the tests pass both locally and in CI.

To be thorough, I reverted the code back to the version from the main branch to confirm the issue. As expected, the tests failed in CI due to an MPS (Apple Metal) related error. This confirms that the failure was related to the .mps device not being available in the CI environment. Switching the device back to .cpu resolved the issue.

-> when I reverted back

-> With .cpu final update all the test passes

Here is the link to my fork with latest changes

Please let me know if you'd like me to keep the macOS version pinned or if there's a preferred approach moving forward.

[awmulyadi]

It is nice that you confirmed the tests were running successfully with the changes 👍 . I recommend we set it to be macos-latest so that we don't have to set it up again when the newer version is being released in the future. Let me know your thoughts on this :)

[ansh-info]

Hi, I just wanted to clarify a small point regarding the macos-latest tag.
It can be a bit misleading—macos-latest currently refers to macOS 14 (Sonoma), whereas the latest available macOS ARM build is macOS 15. For consistency and to reflect the actual latest version (which also matches the one on my local MacBook), we might want to explicitly use macos-15 instead.