"Yet another type of module repository" considered harmful and a modest proposal to piggy back on an existing alternative (tl;dr: Docker)

[oakad]

Setting up a central registry for open access project dependencies is nice and easy, "just another web service". However, Bazel is a sort of heavy lifting tool and as such mostly suits large scale corporate stuff - most projects won't even think of looking for a dedicated build tool before reaching certain substantial scale.

However, once we are in the corporate land, maintaining any sort of IP repository becomes a difficult task:

1. Repository must support fine grained access to different items stored therein, subject to access and security policies.
2. Said access policies usually come with some sort of centralized authentication/authorization system.

To solve the above, rather difficult, problem companies rely on third party products, of which one of the most prominent is JFrog Artifactory (dedicated package management and distribution) as well as "registry" addons for enterprise GitHub, GitLab and such.

Neither of these support bzlmods and it will take a great deal of time for them to support Bazel, if ever.

However, all of these (and many other) solutions support docker image hosting. In fact, docker images are so ubiquitous these days, that storing and controlling access to them is considered a basic feature of any IP storage product, either cloud based or self hosted.

Which makes me think:

1. Bzlmod bundle can be trivially packaged as a docker image (those are nothing more than glorified TARs, so not much overhead there).
2. The bundles can then be pushed and pulled to whatever docker registry Bazel users desire, providing an easy way to host private bzlmod registries while having whatever fancy authentication, authorization and transparent forwarding/caching from other registries essentially for free.
3. Hosting the central bzlmod registry as docker registry is probably easier these days than messing up with custom web services and will provide a bunch of extra features out of the box.
4. There was a bit of discussion in one of the issue threads regarding using "plain git checkouts" as local bzlmod registries, but now I think docker image based solution may be advantageous and is much closer conceptually to the existing bzlmod registry model.

Nota bene: I'm talking here about packaging the MODULE.bazel and associated files into a Docker image serving as a bzlmod definer package; not the actual sources or any build artifacts, which stay the same as they are now.

[sluongng]

Since we already support pointing --registry= to a directory, I think it would be better if you just materialize that directory using whichever method you choose. It's probably gona be the easiest if you just start a local http server with registry authentication baked inside, which supports downloading those container images, and then point the --registry= flag to your http server. I don't see any benefits of using a container registry versus something as simple as multiple registries using Github/Gitlab pages static hosting, though.

Adding oci container pull + auth code into Bazel would make it overly complex and defeat the goal of simplifying the core of Bazel and making it more customizable. Perhaps a future feature to add to Bazel would be to call out to a custom helper executable to handle the registry downloads. That would let people craft their own storage and auth mechanism and improve the customization of Bazel.

[oakad]

Some directory with private bzlmods right in the depending project is, of course, one way to go about this sort of thing and routinely done. After all it's the same as the old "workspace" approach with some extra steps and extra edits. Kind of negates the utility of having an automated module registry though. :-)

1. As to auth code, there's already an effort to reinvent a bicycle here: https://github.com/tweag/credential-helper . How is go helper linked to moby sdk or some such is any worse or "more complicated" than go helper linked to a bunch of cloud sdks escapes my understanding.
2. As to static hosting, most (all?) repository hosting solutions don't provide "intra-repository" permissions. So, if we have 2 modules with potentially incompatible permissions (even partially incompatible) we must put those into 2 distinct repos, completely impractical undertaking. That's why things like Artifactory exist in the first place.

On a less specific note, I must point out that the problem is far from new (I had to implement some automation for Maven registries back in a day, and there were great many quirks to resolve, despite Maven's promise of "it's just some files in static storage"). There are quite a few "module registries" around, they are all "kind of simple" (apart from NuGet, which is just lame), they all try to provide the same functionality, and they all have just enough quirks to make every interested party's life miserable.

Docker/OCI registry approach is generic enough and popular enough to solve a lot of problems wholesale. Maven based on Docker would be a better thing, NuGet based on Docker would be a better thing, PyPi, Pecl/Pie, NPM - all of them could be trivially ported to docker infra and it would be a better thing for everybody trying to host that stuff. :-)