Single command to download the latest backup

[AlexSkrypnyk]

Is your feature request related to a problem? Please describe.
Currently, to retrieve a backup, one needs to perform several actions:

1. lagoon retrieve backup -p project -e main
2. Record the backup ID
3. Keep polling for a backup URL: lagoon get backup -p project -e main -B <backup ID>
4. Download the backup from the URL via browser or through curl with additionally set headers
5. Move the backup to the desired location

Describe the solution you'd like
I would like to have a single command that downloads a database from the production environment.
lagoon get backup latest

Lagoon CLI provides similar command format to deploy the latest commit: lagoon deploy latest

Describe alternatives you've considered
Create a shell wrapper script around current commands.

Additional context
Other hosting platforms have a single-command database retrieval options. End-users will benefit from a more simplified DX.

[shreddedbacon]

https://uselagoon.github.io/lagoon-cli/commands/lagoon_get_backup/ there is a command get backup

Once you have retrieved it, you can use get backup to get the download link without having to go to the UI.

The idea of getting latest is interesting, but there are multiple backups of different names, you'd have to specify which one. Also, backups can take considerable time to retrieve, sometimes hours depending on the size/type. How long does the CLI wait before it would give up waiting for the retrieval? There has to be a timeout.

I think with the three commands already available to list the backups, retrieve the backup, and also provide the link to download it when the backup is ready, you have enough commands to write your own loop logic around this and poll interval according to your known backup sizes.

[AlexSkrypnyk]

@shreddedbacon
Thank you for providing an explanation.

I agree that all of your concerns are valid, but I also think that they could be addressed by setting sensible defaults that would fit 80% of cases

multiple backups of different names, you'd have to specify which one

latest, service is mariadb (or set through .env file or environment variables)

How long does the CLI wait before it would give up waiting for the retrieval?

up to 10 minutes than fail (can be overridden in .env file or environment variables)

I think with the three commands already available to list the backups...

I was hoping to discuss avoiding to have 3 commands and a wrapper script in this issue.
I think that having a more streamlined DX will help others to adopt easier - as mentioned above, other platforms provide single-command operations that developers are used to.

Thanks again

[shreddedbacon]

Yeah, I understand your issue with the multiple commands. But the comment was more for other people that may be watching that maybe didn't realise that the three commands exist too.

Not everyone uses mariadb for their database service name. So it isn't necessarily a sensible default, it might be to you.

I'll have a think about this some more, no guarantees on any implementation though.

[AlexSkrypnyk]

could be a config in lagoon yaml to put those defaults, whatever they are.

I'm happy to be a guinea pig to test canaries

[shreddedbacon]

We need to think a bit more about a latest backup command. There is no actual concept of a "latest backup" beyond it being the latest one in the list. The "latest backup" could change depending on when you run the command too, and depending on your project/environment backup intervals, this could be once a day, or many times a day.

I'm not sure if we will implement any wait or polling logic into the CLI in this case, will need to discuss this with the team more broadly to see how everyone feels about it. This is mostly because some backups can be very large, and it is kind of pointless polling the API every few seconds for potentially hours, especially if you know that your backups are large and take a long time to retrieve. I know with someone writing their own polling logic can do whatever they want, but should you?

If we did implement wait logic, you'd have to handle a wait timeout in your logic too, and also the potential that the backup retrieval fails entirely. What happens if you specify a wait timeout of say 30 minutes, but the backup takes 45 minutes to retrieve? What happens if the backup never retrieves due to a retrieval failure? What will your scripts do then?

We could make lagoon list backups --latest --name mariadb which would return the backup information of the latest mariadb backup at the time the command was run. You would then have to use this ID for every other command though. This at least would make it easier than having to write lookup logic in a tool to get this ID yourself.

If we do end up implementing a wait type command, I'm opposed to any defaults for a backup name, as backup names can be unique. If you work across multiple projects you'd need to change the default depending on which project. Now, this may be fine for a chunk of users that work off a standard deployment type, but not all deployments are the same, or use the same service names. We also don't read the project .lagoon.yml file. And storing a default in the CLI configuration file would be equally problematic for users that work across multiple project types.

Like with #384 though, some flags may have an alternative variable name that could be provided. This may be one of those sorts of cases too.

[AlexSkrypnyk]

@shreddedbacon
I really appreciate the technical explanation.

But look at it from the product point of view (and maybe @tobybellwood can share his view): users want to use a single CLI command to do what they need. Acquia does it, Pantheon does it, Platform.sh does it - they all provide the tooling for their users to be as development experience-friendly as possible. From what you wrote, I do not see anything that could not be overcome by coding additional logic into LagoonCLI. Besides, all of the backups discovery functionality already exists in the LagoonCLI. Any concerns around defaults could be resolved by allowing users to provide their own per-project settings that the teams can store with their projects.

[shreddedbacon]

PRs are always welcome

[shreddedbacon]

Can you provide some links or documentation references to these other providers that do what you're asking from us please. I'd like to understand how they are solving this problem more.

[AlexSkrypnyk]

https://docs.acquia.com/acquia-cloud-platform/add-ons/acquia-cli/commands/api:environments:database-backup-download

https://docs.pantheon.io/terminus/commands/backup-get

[tobybellwood]

We'll add those into the research we do as we look at how the backups in Lagoon function more generally - but that's a way off at the moment, with other parts of the ecosystem demanding more urgent attention.

Personally, I'm against the CLI tools "waiting" for upstream actions. Subscriptions in GraphQL cause us more headaches than almost any other part of the API, and just having an app hanging around waiting for some other upstream function to complete (without a reasonable level of progress measuring) seems like an anti-pattern, and it's one that we've avoided in the other parts of the CLI.

There may be ways we can augment or replace the existing commands to make tracking and following them simpler from a CLI pov, so we'll consider that.