Builds can fail when docker-host ghosts #473
Description
On clusters where node turnover is high, it's possible for the docker-host to be rescheduled in the middle of a deployment and cause the build to fail. Many of the solutions to help reduce this type of failure involve cluster tuning or Lagoon changes that may have increased cost implications.
This is a proposal to try and mitigate that failure in the build-deploy-tool. Instead of failing, wrap the docker image build + push process in a retry mechanism. If at any point during build or push phases the docker-host goes away, wait for the docker host to comeback (with a timeout) and then start the entire build + push phases over again. I think a single retry could be enough, but it could be an env var so it's configurable.