feat: add variable interpolation in compose files

[ganthern]

Hey there 👋

I thought I'd take a crack at #3. My use case:

I wanted to install Immich in my homelab, which recommends the compose file method at the link. My lab uses quadlet files. Immich consists of several containers and uses variables for configuration in the compose file, which I was too lazy to convert by hand.

podlet is almost there to do it automatically, but doesn't support interpolation because compose_spec_rs doesn't (PR to podlet is incoming once/if this is accepted).

Notes

• I tried not to introduce more dependencies, hence the simplistic hand-rolled recursive descent parser instead of using something like nom, combine or pest.
• currently, only the first encountered error is reported. it would be possible to extend this, but the rest of the library also doesn't aggregate errors yet, so I didn't.
• I didn't actually compare this with the interpolation in docker-compose or podman-compose. I'm also not sure what the best way to go about this would be. It's likely there are differences in how whitespace is handled and maybe more.

Intended usage

I was hoping to extend podlet to achieve a usage like

podlet compose --env ./a.env --env - --env ./b.env ./compose.yaml

which is to say it's possible to replace variables given in one or multiple .env files and the actual environment (--env -); with later sources overwriting earlier ones.

So, there it is. Waiting for reviews and any guidance to get this merged!

[Tokarak]

This is huge — this could be the step to migrating these massive self-hosted softwares (Immich, Nextcloud, Seafile, etc.) — and any homelab setup — from Docker compose to Podman Systemd.

The code is mostly LGTM:

• I was a little skeptical about the hand-rolled parser, but it seems very well implemented. In terms of security it should be fine, because the only public method is pub(crate) and takes in only a HashMap and a string slice (I'm referring to Parser::start()).
• This is not completely zero-copy (Every string will be reallocated at least once, even if it doesn't have any variables), but I assume that that shouldn't really matter in practice? The largest compose file I found mentioned online is only 3000 lines long, i.e. only a few kilobytes of memory. If performance ever became an issue, the pattern here is very similar to crate nom, which should hopefully squeeze out those last few memory optimisations.
• Another more legitimate reason to switch the parser to nom (or some other crate which OP kindly mentioned: "nom, combine or pest") is pretty errors. Should quite significantly improve the UX. It would make it easier to debug a compose file with podlet.
• There are some bugs with white space handling; I resolved them here Amendments and bug-fixes ganthern/compose_spec_rs#1. Otherwise, I don't see any bugs!

Just one problem: How do we escape a } character in a modifier? Does the spec allow wrapping a modifier in quotation marks? I mean something like this: ${VAR:-"This is a default value with a '}' character"}. AFAICT this is a limitation of the spec, which is fine, because the variable can be set from env with a '}'.

[akostadinov]

Yes, this is very needed because nowadays docker-compose is the usual choice for packaging complex software like this.

In fact I would like to see podlet integrated with systemd in the same way quadlet is. So that one can adjust .env and then the pipeline would just run the software, instead of all the manual work.

Hope somebody from the project can review this.

[ganthern]

Hello 👋

This is huge — this could be the step to migrating these massive self-hosted softwares (Immich, Nextcloud, Seafile, etc.) — and any homelab setup — from Docker compose to Podman Systemd.

Thanks! Also thank you for the review (I completely agree with it) and the bug fix.

Just one problem: How do we escape a } character in a modifier? Does the spec allow wrapping a modifier in quotation marks? I mean something like this: ${VAR:-"This is a default value with a '}' character"}. AFAICT this is a limitation of the spec, which is fine, because the variable can be set from env with a '}'.

I could swear I thought about this (and/or checked against docker-compose) when I implemented this, but I'll do it again once I find the time and write it down this time. It's been a while :)

Yes, this is very needed because nowadays docker-compose is the usual choice for packaging complex software like this.

In fact I would like to see podlet integrated with systemd in the same way quadlet is. So that one can adjust .env and then the pipeline would just run the software, instead of all the manual work.

Hope somebody from the project can review this.

Thank you as well! Interesting thought, I didn't even consider keeping the docker-compose around. Probably because I'm not hosting many apps yet.

[ganthern]

I had a look at escaping } in default values and error messages by testing a bit.

• docker-compose doesn't seem to support any escaping of closing braces.
• it's also not defined in the spec and there's an issue about that: Please provide a more detailed "braced string" syntax description compose-spec/compose-spec#549
• podman-compose seems to be the same (and doesn't support nested variables either)

I'd guess it would make most sense to adopt backslash escapes as the most familiar
variant. It's also what the POSIX spec the interpolation is loosely based on requires.
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_02

I haven't thought much about how that would interact with YAML escapes.
In any case it's probably for another project to decide.