Improved build logs when errors occur during container startup

[rocketeerbkw]

One of the steps in a deployment that can have failures is Applying Deployments. This is the part where k8s is attempting to rollout the new container images.

Currently, if a rollout fails, the build logs might not be totally clear. We just output the "raw" k8s conditions:

nginx-476fb88445-f9cw7  Running Ready   containers with unready status: [nginx]
nginx-476fb88445-f9cw7  Running ContainersReady containers with unready status: [nginx]

But there are multiple reasons that a container can be "unready:" Applications failures (pods crashing), application delays (pods fail to start in time), miconfiguration (pods listening on wrong port).

I think we can do a better job of looking at these error cases and providing more useful errors to users in the build logs. We can translate the raw k8s messages into steps users can take in Lagoon to fix deployment issues.

[shreddedbacon]

Yeah, this would be fairly simple to do. Extend the failure message to run a kubectl events --for pod/${POD} -o json and scan the events for failure messages like readiness or liveness probe failures, or exit codes etc. Then we could have some messages defined that could be shown depending on which events are returned.

[rocketeerbkw]

Example condition to catch: The dockerfile has a CMD ["customServer"], but the customServer isn't on the $PATH so it isn't executable. In this case, the rollout fails, there are no logs to show, the condition is only "unready" and the user has no idea what is wrong.

Attempting to run the containers locally would also give the user the error.

From the lagoon deploy logs:

##############################################
STEP Applying Deployments: Failed at 2025-03-06 16:25:00 (UTC)
The information below could be useful in helping debug what went wrong
##############################################
Terminated
Rollout for basic failed, tried to gather some startup logs of the containers, hope this helps debugging:
======== /tmp/kubectl-build-deploy/logs/container/basic/basic-6df6fdff85-f8tsb-basic.log =========

##############################################
If there is any additional information about the status of pods, it will be available here
POD/SERVICE NAME	STATUS	CONDITION	MESSAGE
basic-6df6fdff85-f8tsb	Running	PodReadyToStartContainers	
basic-6df6fdff85-f8tsb	Running	Initialized	
basic-6df6fdff85-f8tsb	Running	Ready	containers with unready status: [basic]
basic-6df6fdff85-f8tsb	Running	ContainersReady	containers with unready status: [basic]
basic-6df6fdff85-f8tsb	Running	PodScheduled

From kubectl describe pod basic-6df6fdff85-f8tsb:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned basic-main/basic-6df6fdff85-8hrsj to ip-10-200-171-92.eu-west-1.compute.internal
  Normal   Pulled     14m                   kubelet            Successfully pulled image "harbor.cluster.example.com/basic/main/basic@sha256:869a7666e034bcbdeab1687ef1e7bcee4a41c4e50df87151f261dd6559f63f82" in 216ms (216ms including waiting). Image size: 687858110 bytes.
  Normal   Pulled     14m                   kubelet            Successfully pulled image "harbor.cluster.example.com/basic/main/basic@sha256:869a7666e034bcbdeab1687ef1e7bcee4a41c4e50df87151f261dd6559f63f82" in 176ms (176ms including waiting). Image size: 687858110 bytes.
  Normal   Pulled     13m                   kubelet            Successfully pulled image "harbor.cluster.example.com/basic/main/basic@sha256:869a7666e034bcbdeab1687ef1e7bcee4a41c4e50df87151f261dd6559f63f82" in 208ms (208ms including waiting). Image size: 687858110 bytes.
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image "harbor.cluster.example.com/basic/main/basic@sha256:869a7666e034bcbdeab1687ef1e7bcee4a41c4e50df87151f261dd6559f63f82"
  Normal   Created    13m (x4 over 14m)     kubelet            Created container basic
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "customServer": executable file not found in $PATH: unknown
  Normal   Pulled     13m                   kubelet            Successfully pulled image "harbor.cluster.example.com/basic/main/basic@sha256:869a7666e034bcbdeab1687ef1e7bcee4a41c4e50df87151f261dd6559f63f82" in 234ms (234ms including waiting). Image size: 687858110 bytes.
  Warning  BackOff    3m56s (x52 over 14m)  kubelet            Back-off restarting failed container basic in pod basic-6df6fdff85-8hrsj_basic-main(7d0bfc9c-6844-47df-ba9a-a6d4ee380b35)

[shreddedbacon]

Just on this. I think we should try and move away from exposing the raw k8s output in the log, and instead maybe try to extract relevant information and format it in a way that describes what may be the problem. Or some combination of the two where it makes sense.