So if you want to migrate a particular env from schema version 3 to version 6, do you deploying 3 different init containers manually, or is there a single migration container that always contains all migrations and it by itself understands from which level to which level should the migration be run? Schema migration failure is a stop the world a scenario for our entire stack.
> Schema migration failure is a stop the world a scenario for our entire stack.
That's the big problem, and I don't believe that Kubernetes can help with that. It's too intimately tied to your application internals. The recommended practice for migrations is to do them in multiple zero-downtime steps, e.g. first deploy code that handles both old and new schema, then migrate, then deploy to remove transitional code.
Sorry, I forgot to mention that it is acceptable to stop the world. We do in-house deployments and a lot can fail on the customer's side (IT disabling the DB Server, network failure, etc.). We'd rather have the stop and be focused on fixing it ASAP, than be tricked into thinking that the issue may resolve itself.
What I need is a recipe on how to handle db migrations, which are quite frequent, every customer has different current version of the app and therefore is on different db schema. We have our own tool to do that, I suppose we could wrap into a kubernetes-something, the only thing special is that it should run : after old version containers are brought down, before new version of containers are brought up, run only single time, include backing up the db and stopping the entire process on failure.
