k8s restart can fail while job being canceled

[johnbradley]

If a user cancels a job and then immediately restarts before the k8s cluster has finished cleaning up the various pvcs/configmaps/jobs the restart will error.

[dleehr]

As described, canceling a job through bespin-api sends a message through the queue to lando, instructing it to cancel the job and tear down kubernetes resources.

lando marks the job as CANCELED as soon as it has made those API calls to kubernetes, but the resources may not be deleted immediately. Since the job is in CANCELED state too early, the user may attempt to restart it. The restart will fail because lando will attempt to create more resources with the same names, and kubernetes does not allow those conflicts.

This is somewhat of an edge case and is recoverable - sounds like if the user simply waits a few minutes for the state to settle down, their job can be restarted. So it's not a high priority issue but there is some room to better model the behavior.