Validation false positive for nested interpolation in docker-compose

[AlexSkrypnyk]

When provisioning in Lagoon, the following variable assignment in the docker-compose.yml is reported as invalid, while Docker Compose considers this valid and can process it accordingly.

MYVAR1: ${MYVAR1:-${MYVAR2:-defaultvalue}substring}

When provisioning in Lagoon, the following error is received:

There are issues with your docker compose file that lagoon uses that should be fixed.

Please note that any modern shell supports nested variable interpolation, so supporting this in Lagoon seems a reasonable expectation.

[shreddedbacon]

Interesting, because this uses the docker-compose spec library to validate. There could be something missed though.

Can you provide how you're defining this (full docker-compose file would be nice if possible)

[AlexSkrypnyk]

x-environment: &default-environment
  MYVAR1: ${MYVAR1:-${MYVAR2:-defaultvalue}substring}

services:
  cli:
    image: uselagoon/php-8.1-cli-drupal:23.10.0
    environment:
      <<: *default-environment

[AlexSkrypnyk]

@shreddedbacon

this uses the docker-compose spec library to validate

I only was able to find this

build-deploy-tool/legacy/build-deploy-docker-compose.sh

Line 203 in 6bd767a

### use the `build-validate` built in validater to run over the provided docker-compose file

Not sure how to follow to the library though.

The official docker compose docs state that nested interpolation is supported
https://docs.docker.com/compose/compose-file/12-interpolation/

Looking at the tests here, https://github.com/uselagoon/build-deploy-tool/tree/6bd767a68605db8e16a553d0f61766969852d3ca/test-resources/docker-compose, there is not a single instance of nested interpolation of variables among those fixtures.

[shreddedbacon]

We use the github.com/compose-spec/compose-go library to parse the docker-compose files. The error you see is from it.

We don't have a test case for nested interpolation because until today you're the first person to do it.

When I run your provided nested variable into a test case, I get the following

invalid interpolation format for x-environment.MYVAR1: "${MYVAR2:-defaultvalue". You may need to escape any $ with another $

but if I do it like the docker docs state and use ${MYVAR1:-${MYVAR2:-defaultvalue}}, it parses fine.

[shreddedbacon]

For some reason, maybe there is a fix in a newer compose-go library version, it doesn't like the substring in your provided nested variable.

I'll check the compose-go library if there is a fix in a newer version, otherwise will have to log an issue with them.

[AlexSkrypnyk]

What is the version of the compose-go are you using?
There is a bug that was fixed in https://github.com/compose-spec/compose-go/releases/tag/v1.13.5

https://github.com/compose-spec/compose-go/pull/405/files

[shreddedbacon]

Yes, just saw that. I'll see if we can update our version. We are running a forked version at the moment due to some Lagoon-isms though that I'll need to backport.

[shreddedbacon]

Will see what I can do, but it may not be today.

[tobybellwood]

The provided code does not build with docker compose v2, only v1

It can't handle the "substring" appended inside the variable
MYVAR1: ${MYVAR1:-${MYVAR2:-defaultvalue}}substring

[AlexSkrypnyk]

The use case is quite wide: base a value on COMPOSE_PROJECT_NAME but allow to override it from internal variables.

Example from DrevOps:

DREVOPS_LOCALDEV_URL: &default-url ${DREVOPS_LOCALDEV_URL:-${COMPOSE_PROJECT_NAME:-example-site}.docker.amazee.io}

I want the default URL to be based on COMPOSE_PROJECT_NAME by default (which defaults to the current directory) - this allows to run the same project codebase from multiple directories and the URL will be auto-prefixed with the current project dir name. But it also will allow to override that URL locally by providing DREVOPS_LOCALDEV_URL if needed without changing any code in docker-compose.yml.

[AlexSkrypnyk]

@tobybellwood
please see the link above - we have pin-pointed the issue - it is due to the outdated composer-go library used by Lagoon.

[tobybellwood]

oh - ahahaha - I realised that i'd pinned my docker compose v2 to v2.17.3 for testing last week... And hence why it was broken on my local (and hence was hitting the exact same bug)