Emit WITs during build

[itowlson]

We want to make it more seamless for application components to consume dependencies: that is, once you add the dependencies to your spin.toml, the imported interfaces become readily available for you to program and build against in your main component. We've explored various ways of doing this, but the current favourite is to have spin build extract the dependency WITs and emit them to a well-known location.

We have prior art here from the spin deps plugin, but we now want that functionality in the Spin CLI and integrated into spin build. A consequence of this is language neutrality: the spin deps plugin had language commands that did some nice work on the code side, but Spin generally tries not to know about specific languages.

We need to design it so it can be called from outside spin build. We've always had a philosophy that you can build Spin apps using your toolchain of choice, whether that's Spin, make, or whatever. So we should expose the "create or update WITs" functionality in such a way that other toolchains can use it (although spin build will do it automatically). Ideally it should be possible to run continuously or via LSP so that WITs are updated as dependencies are added and removed (and so compiler errors and code completion update), but this may be impractical: in any event, it's not required on day one.

Language templates or SDKs will want to update to integrate seamlessly with the generated WITs but that's out of scope for this issue.

We should agree where to store extracted WITs. spin deps put them under .spin, which is natural - it is a bit of mix of various kinds of data but we can create a subdirectory to isolate the extracted WITs.

[fibonacci1729]

Thanks for writing this up!

We need to design it so it can be called from outside spin build

My vote would be to add this as a new subcommand that is executed by default when spin build. I would propose spin deps generate or something in that vein.

We should agree where to store extracted WITs.

If we do use .spin I would vote we put the synthesized wits in a subdirectory like .spin/deps/wit

[tschneidereit]

We should agree where to store extracted WITs. spin deps put them under .spin, which is natural - it is a bit of mix of various kinds of data but we can create a subdirectory to isolate the extracted WITs.

I think we should place them in the directory a component resides in, not all in one place. Otherwise, a component's build system has to somehow have knowledge about the overall workspace layout—which I think is simply impossible (short of very specific hard-coding) in case a custom location for the spin.toml manifest is used.

And I think my vote would be for something quite simple and top-level, such as ./spin-deps.wit, with a comment in the header explaining what this is. An empty default version of this file (or rather, one containing an empty world) could be included in the templates, and already be wired up for bindings generation, too.

[fibonacci1729]

I like that proposal!

For the rust templates we could add spin_sdk::imports!() (similar to Ivans PoC) where imports!() is a simple wrapper around wit_bindgen::generate!().

[itowlson]

I don't want to use spin deps because a plugin with that name already exists.

And within that I am not sure what to call it. spin deppydoo generate invites: generate what? The dependency? The obvious answer is spin deppydoo generate-wits - and that's long, and who knows or cares what a wit is, am I going to end up with Oscar Wilde in my project.

But... thinking about it, like I should have done before I started writing this paragraph, this is going to be used only when someone wants to use Spin in a non-spin build scenario. And so they are already sufficiently in the weeds of Wasm-targets-this-and-j2w-that-componentise-the-other that they'll have to be aware of WITs anyway, and will likely be tucking it all away in a makefile or script. So it's probably fine.

[lann]

Maybe spin generate? It would just be dependency WITs initially but we've talked about generating e.g. protobuf from WIT exports in the past too.

$ spin generate
Writing dependency imports for component 'foo' to '.spin/dependencies/foo.wit...

[karthik2804]

I don't want to use spin deps because a plugin with that name already exists.

FWIW it never made it into the official plugins registry.

[tschneidereit]

I think the existence of a (non-registry-published) plugin shouldn't prevent us from using spin deps: this I think is solidly a case where the future is bigger than the past.

However, I'm not even sure I particularly like that subcommand to begin with. I also don't really like spin generate, because that sound a lot like project scaffolding (and in fact I could see a possible future in which we'd want to use it for just that).

I would suggest something like spin resolve-exports, except ultimately I think it'd be good to synthesize the world in this generated WIT file not just from dependencies' exports, but also from the target environment. spin resolve-worlds sounds ... pretentious though 😔 And uninformative on top.

Maybe spin resolve-component-environments? Too wordy/long? Probably. At least it's sort of correct?

[itowlson]

Maybe spin resolve-component-environments? Too wordy/long? Probably. At least it's sort of correct?

I'm afraid I would have no idea what that meant - sorry. "Resolve" doesn't really communicate much to me, and what I'd infer from it is misleading; and "environments" doesn't in any way suggest imports / dependencies / libraries. Yes, there would be docs, but I'd like the command name to have associations that make it easy to find / recognise / remember - although, since this is probably primarily scripting, it may be a "use twice in your life and done" where that doesn't factor in.

I would also tend to avoid super long top-level commands for the rather mundane reason of help text alignment - they push all other command help right, making it harder to track from shorter commands to their help. That not a "we can't have them", just a "prefer shortish where practical." I'd certainly resist the help experience of commands people use being degraded by specialist commands.

[fibonacci1729]

Historically we've also discussed a spin prepare as well that was meant to capture "work to be done before build and after new" but that is still pretty vague/generic. I agree that we shouldn't toss a name aside because it was used in an experiment that is evolving into a core suite of commands. I personally like spin deps prepare (or generate). I imagine we'll be adding an experience for guiding the addition of dependencies to the manifest (similar to the spin-deps-plugin prototype) and that command will invoke whatever we name this workflow we are discussing.

[tschneidereit]

@itowlson I guess I should've been more clear that I didn't really think that was a name to seriously consider—even I can immediately see that it is way too long, apart from the other issues you rightfully point out.

I guess I agree that spin deps prepare is okay. And maybe it even works okay if and when part of what it does is to apply the target environment as well when generating the WIT file, not just dependency exports.

I imagine we'll be adding an experience for guiding the addition of dependencies to the manifest (similar to the spin-deps-plugin prototype) and that command will invoke whatever we name this workflow we are discussing.

One thing to note is that this doesn't only need to happen when adding a new dependency: at least conceptually (meaning if no optimizations/shortcuts are taken) it needs to happen on every build, as the built component's dependencies might've changed their interface.

... which in turn means, as Ivan mentions, that most developers shouldn't need to ever use this directly: the reason we need it is that not all developers will want to use Spin for build orchestration, opting to directly use something like cargo build instead. In such a case, they still need a way to freshen dependency interfaces, and would use this command.

[itowlson]

I imagine we'll be adding an experience for guiding the addition of dependencies to the manifest (similar to the spin-deps-plugin prototype) and that command will invoke whatever we name this workflow we are discussing.

One thing to note is that this doesn't only need to happen when adding a new dependency: at least conceptually (meaning if no optimizations/shortcuts are taken) it needs to happen on every build, as the built component's dependencies might've changed their interface.

No, it needs to happen on add (or soon after), because code completion / background error checking / etc. The developer wants to write code against their shiny new library!

[tschneidereit]

No, it needs to happen on add (or soon after)

I agree, hence "[..] this doesn't only need to happen when adding [..]"

[itowlson]

🤦 My apologies for misreading, @tschneidereit!

[tschneidereit]

No worries at all—it was a bit subtle and easy to overlook