This is the first part of a two-part article that describes how to implement a Blue/Green deployment of an application when a schema change to a database is made. This schema change will break existing code. I also assume that the amount of data in the database is very large.
We have a Staging and a Production instance of the application. After a production release is performed, we want to be able to roll back to the staging instance.
The application, called Sports Store, is a Microsoft MVC 5 application using a SQL server database. The application uses Azure as the execution environment. Azure Deployment Slots are used to facilitate rollbacks. We also use Azure DevOps as the CI/CD tool, and all aspects of the process are automated.
This is an oversimplified example, but my hope is that it touches all the points a real application would.
We use a simple Customers table, as shown below. We will be renaming the LastName field to SurName. Remember, this application has been running in production using the LastName field. We also assume that there are 1 million records in the Customer Table.
Our application has a simple way to browse the data and perform CRUD Operations. LastName will be changed to SurName.
Here is the data entry screen. Again, LastName will be changed to SurName.
My implementation is based on the excellent article Zero-Downtime Deployment With a Database.
The biggest design decision I had to make involved the database. I could have a separate database for Production and Staging. This would make the deployments easier, since each environment would have its own isolated database.
However, because of the large amount of data involved, I decided to have a single database shared by both environments. This makes deployments trickier but would not require the movement of vast amounts of data between environments. The diagram below depicts the design I chose.
I also decided to associate a version number with the web application and the database. There is also a configuration setting for the web application to identify the database version it is compatible with.
In the above diagram, we see the database version is 2.0.3. The current production version is 2.2.6, and the stage version is 2.2.5. Both Production and Stage are compatible with database version 2.0.3. As long as Production and Stage are compatible with the same database, we can safely roll back a release. A release is rolled back by changing where the Production IP points to.
We use Azure Deployment Slots to implement our Blue/Green Environment. The screenshot below shows our single Sport Store Database. We also see the production instance of the Web application, OcambSSBlueGreenWeb, and the staging instance, Staging (OcambSSBlueGreenWeb/Staging).
Continuous Deployment Pipeline Diagram
We use Azure Devops to deploy artifacts to our Blue/Green Environment. The artifacts are created by the build process and use the codebase in the Master branch.
There is a deployment pipeline, Deploy – BlueGreen Master Database, with a single stage, Production DB, that deploys the database to production. This is a shared database, so great care must be taken when deploying database changes.
The deployment pipeline for the website, Deploy – BlueGreen Master Website, has two stages.
Staging is used to deploy the website to the Staging Slot. Final testing is done to ensure the deployment is acceptable. It is important to understand that prior to this, extensive testing was completed as sprints unfolded.
Production is used to swap the Production IP Address. Once this is completed, the new changes are available to production users. There is also a pipeline to roll back a release.
Continuous Deployment Pipeline – Azure DevOps
The screenshot below shows the release pipeline for the database. The artifacts were built from the Master branch. An approval is required before the artifact is deployed.
The screenshot below shows the release pipeline for the website. The artifacts were built from the Master branch. There are two stages to the deployment. Each stage requires an approval.
The first stage deploys artifacts to the Staging environment. We perform a set of regression tests to ensure we are ready to move the release to Production. During this time, Production is using the current release. Both Production and Staging are compatible with the database, so they can operate side by side.
When we are comfortable with the release, we change the Production IP Pointer to point to the staging environment by approving the Production stage. No code is deployed here; we only change the Production IP Pointer. If something goes wrong in production, we can easily roll back.
The screenshot below shows the release pipeline for a rollback. This only switches the Production IP Pointer to point back to the previous release. There are no artifacts involved in this case.
This article discussed the approach we are taking to implement a zero-downtime deployment of an application when a breaking schema change is made to the database. We discussed the challenges we faced and our design.
We showed how we set up deployment slots in Azure and how we use Azure DevOps to implement a Blue/Green environment with rollback capability.
The next article will dive into the details of how this was implemented.