Good way to avoid rollbacking DB setup for svc deploy

[toshitanian]

I lately work on building and on deploying several services using App Runner that uses database. Setting up manifest for both app and database is very easy by running the command below.

$ copilot svc init --name my-svc -t "Request-Driven Web Service"
# Edit network.vpc
$ copilot storage init -n my-db-cluster -t Aurora -w my-svc  --engine PostgreSQL --initial-db test-db
$ copilot deploy svc 

With this deploy command, a Cfn stack is created and the provisioning starts running. The stack has both Aurora DB setup and App Runner setup. The DB setup takes about 10 minutes to complete, App Runner set up takes other 10 minutes.

The problem is that if App Runner service crashes due to some error and App Runner fails to setup, the Cfn tries to rollback all the stack including DB setup while I want the DB being kept. In my most cases, we just need to fix the environment variable settings or some small program fixes to have the application run correctly. But to confirm that the bug is fixed by deploying the service, I need to wait for the DB setup and App Runner setup. It's difficult to do try-and-errors.

Separating DB setup to other svc is a way to solve this, but setting up security group across svc and db secrets are comprecated.
#1975 (comment)

So, I want to know how if theres's good way to avoid rollbacking DB setup for svc deploy

[efekarakus]

Hi @toshitanian !

The problem is that if App Runner service crashes due to some error and App Runner fails to setup, the Cfn tries to rollback all the stack including DB setup while I want the DB being kept.

😭 I totally understand. I can think of two possible mitigations:

1. [preferred] Deploy the app runner service and DB addon first with a stable image https://gallery.ecr.aws/nginx/nginx that will guarantee that the stack deployment will succeed, and then make the remaining manifest changes so that the database doesn't get rolled back.
2. You can use the new --no-rollback flag available in the deploy command: https://aws.github.io/copilot-cli/docs/commands/deploy/, this way the rollback won't happen on failure and it might give you a bit more time to figure out what went wrong. Unfortunately, you'll still need to trigger the rollback before you can deploy again 😢

[toshitanian]

@efekarakus Thank you for your reply!
Option 1 looks good to our use case 👍 Will try with that way