strix halo support?

[iandouglas]

My AI rig is an AMD Ryzen AI Max+ 395, aka "strix halo", with an integrated GPU that works, I believe, via ROCm. AFAIK llama.cpp has support for this, but I'm curious if your version also supports this or if additional work would be needed? I'd love to try out your project! (I'm on the Goose team at Block)

[ShelbyJenkins]

@iandouglas, hello.

It will work right now using CPU/RAM backend. This person on reddit tested performance differences between RAM and ROCm/HIP with a 395+, and using it as just normal RAM works well enough for testing.

Adding full support for the lmcpp crate

The lmcpp crate is pretty simple. It just builds/downloads for which ever platform is available (or configured), and offers a type safe interface to llama-server.

1. Adding tooling to detect if the proper software is available. Right now we use the NVML crate for Nvidia just to "test" that it's available before we try to build for it.
2. Adding the download paths and configuration options for ROCm.

For the forthcoming llm_models crate

This is a slightly more complex as it allocates models to GPUs.

1. Adding tooling to detect the integrated memory as a GPU device.
2. Configuration options.

I've done this before for Apple's Metal, and it was fairly straight forward and simple. I need to investigate a way to access a system with ROCm support to do this. So my intention was to build/buy one at some point like I did with my macbook, but I could also rent a container with an AMD GPU. If it's a high priority, I could probably do it in a few days. Otherwise, my primary goal right now is to finish the llm_models crate.