Failure to run encrypted models on openvino/model_server:latest due to Memory Fault

[anjalirai-intel]

Description of the problem

I was running openvino/model_server with encryption using gsc. I encrypted the model files using gramine-sgx-pf-crypt and added the corresponding entries to the manifest.template file, such as { path = "/mnt/tmpfs/model_encrypted", uri = "file:/mnt/tmpfs/model_encrypted", type = "encrypted" }. Additionally, I included fs.insecure__keys.default = "88f1c4333d2d1e85a81d3b72f92fb396" in the manifest.template file.

I was successfully able to run openvino/model_server:2023.0, and the OpenVINO server started without issues. However, when I built the latest openvino image openvino/model_server:latest and tried the same setup, it failed with a memory fault debug: memory fault at 0x0000cc95 (IP = 0x39b6f85e2).

I have attached the manifest file for openvino/model_server, and you can download the model by following these steps:

mkdir models
curl --create-dirs https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.xml https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.bin -o models/1/face-detection-retail-0004.xml -o models/1/face-detection-retail-0004.bin

Steps to reproduce

I have attached the manifest file for openvino/model_server, and you can download the model by following these steps:

mkdir models
curl --create-dirs https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.xml https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/face-detection-retail-0004/FP32/face-detection-retail-0004.bin -o models/1/face-detection-retail-0004.xml -o models/1/face-detection-retail-0004.bin

Expected results

openvino/model_server:latest should be able to run successfully with encrypted model

Actual results

openvino/model_server:latest is failing with memory fault

I have attached the log files as well

openvino_model_server.zip

[adarshan-intel]

This memory fault issue is reproducible on bare metal by downloading the ovms 2024.5 binary from docs.openvino.ai for ubuntu 24.04
Attaching the files for both manifest.template files, gramine-direct and gramine-sgx logs
logs.zip

[kailun-qin]

This memory fault issue is reproducible on bare metal

IIUC, the issue also applies to core Gramine. If so, could you pls raise a discussion/issue in the core Gramine repo?

[adarshan-intel]

This memory fault issue is reproducible on bare metal

IIUC, the issue also applies to core Gramine. If so, could you pls raise a discussion/issue in the core Gramine repo?

Raised on issue on gramine - gramineproject/gramine#2086