Nix installable packages

[Brikaa]

No description provided.

[Brikaa]

@HexF I get dependency problems when trying to inject some packages into the environment. For example this happens when installing python requests and adding its site-packages PYTHONPATH

Traceback (most recent call last):
  File \"/piston/jobs/351474a4-a3bd-4127-a0e9-cedf32106275/file0.code\", line 4, in <module>
    import requests
  File \"/nix/store/23x8xw10z0slsdvy9jjnblvf0hyl78yh-python3.9-requests-2.26.0/lib/python3.9/site-packages/requests/__init__.py\", line 100, in <module>
    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
  File \"/nix/store/23x8xw10z0slsdvy9jjnblvf0hyl78yh-python3.9-requests-2.26.0/lib/python3.9/site-packages/requests/__init__.py\", line 85, in check_compatibility
    raise Exception(\"You need either charset_normalizer or chardet installed\")
Exception: You need either charset_normalizer or chardet installed

[Brikaa]

Creating a ready environment using pkgs.python3.withPackages does not have this problem.

[Brikaa]

But Python libraries you would like to use for development cannot be installed, at least not individually, because they won't be able to find each other resulting in import errors. Instead, it is possible to create an environment with python.buildEnv or python.withPackages where the interpreter and other executables are wrapped to be able to find each other and all of the modules.

[HexF]

What if piston.mkRuntime in the nix flake takes a function, which allows us to pass in all the package names? That way we could dynamically construct the environment from nix. The only issue I can think of is the speed at which some environments will load.

{pkgs, piston, ...}:
let
    basePkg = pkgs.python3;
in piston.mkRuntime libraries: 
    let
pkg = basePkg.withPackages libraries;
in
    {
    language = "python3";
    version = pkg.version;

    aliases = [
        "py3"
        "py"
        "python"
    ];

    run = ''
    ${pkg}/bin/python3 "$@"
    '';

    packages = pkg.pkgs;

    tests = [
        (piston.mkTest {
            files = {
                "test.py" = ''
                    print("OK")
                '';
            };
        })
    ];
}

[Brikaa]

Flakes don't seem to support command-line arguments so how will we do the ad-hoc package installations?

[HexF]

https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-eval.html
--arg ?

[HexF]

Could we possibly make this piston.mkRuntimePackages or similar, just so if packages aren't required we can use the old format?

[Brikaa]

Error: Command failed: nix eval --json /piston/src#pistonRuntimeSets.all.python3.metadata --arg sf true
error: '--arg' and '--argstr' are incompatible with flakes
Try 'nix --help' for more information.

[Brikaa]

There seems to be this: https://nixos.org/manual/nix/stable/language/builtins.html#builtins-getFlake if we export the runtimes as a function that takes the packages as an argument. But it seems to copy the entire working directory into the nix store every time and it is slow.

[Brikaa]

Maybe we can copy flake.nix into the job directory and inject the packages into it lol

[Brikaa]

The reason builtins.getFlake is slow is that it is ignoring the version control system and copying everything to the nix store every time. Putting git+file// before the path solves the problem.

[HexF]

Finally got back around to looking at this... wow its been a while.
The reason why --arg isn't supported is kinda weird. It's to do with nix's pure builds, and if you can suddenly start throwing arguments at an eval call, the actual build outputs can change.
While this is what we want, nix doesn't actually support this for flakes.

What I'm thinking currently is just exporting the packages key to the API, calling nix build on all the paths we get out, then passing these paths into the run and compile scripts.
This way the run and compile scripts can deal with how to setup the environment correctly for the package set.
I think this was the way you were originally going to implement it.