Support for python package without cli dependencies

[gaby]

Problem

Currently installing magika also installs a cli and dependencies. When using magika as a library, these are not needed.

Proposal

Provide a core package that allows only installing the library, example pip install "magika[core]". This can be done through the pyproject.toml.

[reyammer]

Hello @gaby, great to hear from you, happy new year, and sorry for the long delay in getting back to you! Resurrecting after the holidays break :-)

This looks like something useful (especially in the context of the pydantic-ai issue), thanks for bringing this up.

The request is to have something that would allow one to NOT carry along the rust binary when installing via pip, right?

If so, what would be the trick? I know of the "magika[xyz]" way to install optional dependencies, but is it easy to do the same to indicate what to NOT package in the file? If you know how to do it, would you mind creating a PR?

[Unrelated: I've skimmed through the discussion with the pydantic-ai folks, thanks for being a magika's advocate :-). Feel free to open issues for any additional friction there might be, we are happy to put the work in to help additional integrations. Thanks!]

[gaby]

Hello @reyammer

The rust binary/dependency issue with 3.14 is a separate one I believe.

The main purpose of this issue is to allow users to install magika as a library. Currently installing the package not only gives you a library, it also forces a cli via click on users. Not every user needs this/will use it.

Goal:

• pip install magika, stays the same as now.
• pip install magika[core], only installs the dependencies needed for using it as a library ex: skip installing click.

On a side-note, I tried locally to get rid of numpy but failed, its only used in 1 file. I'm not fully verse on using numpy, so I don't know if its required or if replacing it with vanilla python will hurt performance.

[reyammer]

Hello,

The rust binary/dependency issue with 3.14 is a separate one I believe.

Correct. I tried a while ago (#1206), but onnxruntime didn't support 3.14 yet. IIRC I've read around they will add support by the end of Jan 2026, so we should be unlocked soon on this. Let's keep an eye on it.

The main purpose of this issue is to allow users to install magika as a library. Currently installing the package not only gives you a library, it also forces a cli via click on users. Not every user needs this/will use it.

Oh it's much worse than that :-) Currently pip install magika installs the python CLI dependencies such as click... but also the full magika Rust binary (which embeds its own copy of the models + makes it less installable on non-mainstream platforms/architectures for which we don't have wheels). The pure-python wheel does not carry the rust binary, but the others do. So, yes, I think it would be very nice to offer an option to just install "magika the python library".

pip install magika[core], only installs the dependencies needed for using it as a library ex: skip installing click.

I checked around, and I don't think this is possible. My understanding from PEP 508 is that "each extra results in the declaration of additional dependencies of the distribution when the extra is used in a dependency specification.". As in: extras are strictly additive (please keep me honest!).

In that case, the way forward could be: we create a new package, say magika-core, which only ships the Magika python module, with no dependencies for the python or Rust clients. If that works out, one day we can even have the original magika package depends on such new magika-core, but we can always do it later. Creating magika-core (which I just registered on pypi, just in case), should be pretty easy: it's a pure-python package. And in that package, we can also get rid of click and I think also python-dotenv as dependencies.

Thoughts?

On a side-note, I tried locally to get rid of numpy but failed, its only used in 1 file. I'm not fully verse on using numpy, so I don't know if its required or if replacing it with vanilla python will hurt performance.

We could try. It's not critical as onnxruntime does depend on numpy anyways, but IIRC we only use it to create the input feature vector for onnxruntime and to compute argmax on the output vector, but given that the big bottleneck is the model inference anyways, maybe we can try to get rid of it? This would simplify our pyproject and we would avoid to add constraints to numpy versions for onnxruntime and/or whatever project is integrating with magika. Created #1268 to track.

/cc @invernizzi

[reyammer]

Hello @gaby, any more thoughts on this before we move to next steps? Feedback is welcome! :-)