Another question about a plugin architecture

[wabiloo]

Hi. Me again, with another question related to enabling a plugin architecture.

I have a CLI tool (cloup + click) that has multiple levels of "hard-coded" commands.
I would like to make it possible to have "plugin" commands that are distributed separately from the main tool, to only selected users. I am therefore trying to figure out a way to enable (sub-)commands to have a mix of hard-coded sub-commands and "discoverable" sub-commands.

For example the tool may support

mycli command1 subcommandA
mycli command1 subcommandB
mycli command2 subcommandC

But the user can install a subcommandplugin.py files inside ~/.mycli/plugins/command1/, which would then enable

mycli command1 subcommandpluginX

And similarly, a pluginY.py file inside ~/.mycli/plugins/ would then enable

mycli pluginY

I have been able to make it work when introducing a special sub-command that enables a list of plugins, such as

mycli addons pluginZ

with the following

import os
import click
import cloup

plugin_folder = "~/.mycli/plugins"

class AddonGroup(click.Group):
    def list_commands(self, ctx):
        if not os.path.isdir(plugin_folder):
            click.secho(f"Plugin folder {plugin_folder} does not exist", fg="red")
            return []

        rv = []
        for filename in os.listdir(plugin_folder):
            if filename.endswith(".py") and filename != "__init__.py":
                rv.append(filename[:-3])
        rv.sort()
        return rv

    def get_command(self, ctx, name):
        ns = {}
        fn = os.path.join(plugin_folder, name + ".py")
        with open(fn) as f:
            code = compile(f.read(), fn, "exec")
            eval(code, ns, ns)
        return ns[name]


@cloup.command(cls=AddonGroup, help="Other functionality provided through plugins")
def addons():
    pass

But I don't like the need for this extra "addons" sub-command.

Any recommendation?

BTW, with cloup, I'd add any "discovered" plugin in a specfic section, but this would be alongside other hard-coded sections.

Thanks in advance

NB. to be clear, I've avoided talking about "lazy-loading", because it's not really something I need or care about. If they're lazy-loaded, that's fine, but it's more important to me that they are discoverable at run-time and can be listed (alongside hard-coded commands) in the --help section

[wabiloo]

One thing that I tried (but did not work) was to have AddonGroup inherit from cloup.Group instead of click.Group, but it seems that when doing this, list_commands() is never called.
It may therefore be that I have to overload other cloup.Group methods instead

[wabiloo]

I found one possible way to do it, but I'm not sure how "good" it is...

import os
import click
import cloup

plugin_folder = "~/.mycli/plugins"

PLUGIN_SECTION_NAME = "Plugin commands"

class AddonGroup(cloup.Group):
    def __init__(self, name=None, commands=None, **attrs):
        super().__init__(name, commands=commands, **attrs)
        plugins = self._find_plugins()
        if plugins:
            plugin_section = cloup.Section("Plugins")
            for plugin_name, plugin in plugins.items():
                self.add_command(plugin, section=plugin_section)

    def _find_plugins(self):
        plugins = {}
        if not os.path.isdir(plugin_folder):
            click.secho(f"Plugin folder {plugin_folder} does not exist", fg="red")
            return plugins

        for filename in os.listdir(plugin_folder):
            # TODO - add logic to only find plugins relevant to the current group command
            if filename.endswith(".py") and filename != "__init__.py":
                command_name = filename[:-3]
                plugins[command_name] = self._load_plugin(command_name)
        return plugins

    @staticmethod
    def _load_plugin(name):
        ns = {}

        fn = os.path.join(plugin_folder, name + ".py")
        with open(fn) as f:
            code = compile(f.read(), fn, "exec")
            eval(code, ns, ns)
        return ns[name]

    def list_sections(self, ctx):
        sections = super().list_sections(ctx)

        # move the Plugins section - if it exists - to the end
        if sections and sections[0].title == PLUGIN_SECTION_NAME:
            sections.append(sections.pop(0))
        return sections


@cloup.command(cls=AddonGroup, help="Other functionality provided through plugins")
def addons():
    pass

@cloup.command()
def bla():
    print("This is a hard-coded command indeed")

addons.section("Hard-coded commands", bla)

It "works" (and also solves the styling issue I had been mentioning in #165), but I wonder whether this is really the best way.

Also, because the discovery of plugins happens in the init(), I don't have access to the context (ctx) and therefore cannot easily determine where in the command hierarchy is, to discover the applicable plugins