Zero Downtime Deployment with a Database - Example

In my previous article, I covered the challenges and design of an application that uses zero downtime deployments with breaking schema changes to a database. In this article, I cover the implementation details.

The current state of our database table is shown below.

Image1.png

We will be renaming the LastName field to SurName.

Image2.png

The user interface will also need to change.

Image3.png

The implementation for the user interface ensures that a value is entered for LastName. This will be implemented for SurName as well.

Implementation Details

We will be using diagrams like the one shown below to step through the implementation. The database version is shown above the database element. We start with 2.0.0.

Under the database element, we see the fields involved. We start with the current database with LastName that does not allow nulls.

Application 2.2.2/Database 2.0.0

The current production version of the application is 2.2.2 and is compatible with database version 2.0.0. The previous version, 2.2.1, is in the staging environment and is also compatible with database version 2.0.0.

Image4.png

Application 2.2.2/Database 2.01

The first thing we need to do is add SurName to the table. It is set to allow nulls so both versions of the application will operate. Remember, the user interface validates the entry of the data. The database reinforces this rule. This brings our database version to 2.0.1.

Image5.png

No application code is changed, but the compatible database version is changed. As new rows that contain LastName are added, null SurName fields will be present. The database change needs to be tested carefully in the QA environment since no easy rollback is available when deployed to production.

Image6.png

Application 2.2.3/Database 2.0.1

Version 2.2.3 of the website implements the SurName field in the user interface. Front-end validation ensures that the SurName field has been entered.

Image7.png

To support a possible rollback, we also need to maintain the LastName field. This application uses the Entity Framework and the Repository Pattern.

A screenshot of the code changes for the SaveCustomer method is shown below.

Image8.png

We need a postdeployment script that updates the SurName field with the LastName values.

Image9.png

Below, we see the implemented SurName field. All of the LastName values are now also in the SurName field. As the user operates the application, the LastName field is also updated.

Image10.png

Rollback - > 2.2.2

If something goes wrong with 2.2.3, we can roll back to 2.2.2. As we are fixing 2.2.3, users will be using 2.2.2. As they do, only the LastName field will be updated. When we deploy the fix to 2.2.3, we need to rerun the postdeployment script.

Application 2.2.4/Database 2.0.2

2.2.4 removes LastName from the application codebase, but we need to keep LastName in the database and change it to allow nulls. As new rows that contain SurName are added, null LastName fields will be present.

Image11.png

Rollback - > 2.2.3

If something goes wrong with 2.2.4, we can roll back to 2.2.3. As we are fixing 2.2.4, users will be using 2.2.3. As they do, the LastName field will still be maintained.

Application 2.2.5/Database 2.0.3

Version 2.2.5 of the application depicts the implementation of some other new feature and the completion of the SurName implementation.

Final database changes are made, and the LastName field is replaced with the SurName field.

Image12.png

Rollback - > 2.2.4

Version 2.2.5 implements some other feature. If something goes wrong, we can roll back to 2.2.4, which contains the final application code for the SurName change.

Closing Thoughts

This trivial change represents a worst-case scenario for a database change. To support rollbacks, we basically need to support both the LastName and SurName fields and gradually remove the LastName field.

In my development cycle, I used a separate local development SportsStore database. This is necessary because I needed to carefully test the database deployments. I also carefully made backup copies of the local database so I could restore and retest the deployments.

In a real-world example, careful thought, planning, and testing need to be completed to achieve true zero downtime deployments with rollback.

If you are interested, you can have a look at the current Production and Staging version of SportsStore.

Zero-Downtime Deployment with a Database – Introduction

sqldevops.png

This is the first part of a two-part article that describes how to implement a Blue/Green deployment of an application when a schema change to a database is made. This schema change will break existing code. I also assume that the amount of data in the database is very large.

We have a Staging and a Production instance of the application. After a production release is performed, we want to be able to roll back to the staging instance.

The application, called Sports Store, is a Microsoft MVC 5 application using a SQL server database. The application uses Azure as the execution environment. Azure Deployment Slots are used to facilitate rollbacks. We also use Azure DevOps as the CI/CD tool, and all aspects of the process are automated.

This is an oversimplified example, but my hope is that it touches all the points a real application would.

Our Challenge

We use a simple Customers table, as shown below. We will be renaming the LastName field to SurName. Remember, this application has been running in production using the LastName field. We also assume that there are 1 million records in the Customer Table.

Image1.png

Our application has a simple way to browse the data and perform CRUD Operations. LastName will be changed to SurName.

Image2.png

Here is the data entry screen. Again, LastName will be changed to SurName.

Image3.png

My implementation is based on the excellent article Zero-Downtime Deployment With a Database.

Design

The biggest design decision I had to make involved the database. I could have a separate database for Production and Staging. This would make the deployments easier, since each environment would have its own isolated database.

However, because of the large amount of data involved, I decided to have a single database shared by both environments. This makes deployments trickier but would not require the movement of vast amounts of data between environments. The diagram below depicts the design I chose.

Image4.png

I also decided to associate a version number with the web application and the database. There is also a configuration setting for the web application to identify the database version it is compatible with.

Image5.png

In the above diagram, we see the database version is 2.0.3. The current production version is 2.2.6, and the stage version is 2.2.5. Both Production and Stage are compatible with database version 2.0.3. As long as Production and Stage are compatible with the same database, we can safely roll back a release. A release is rolled back by changing where the Production IP points to.

Deployment Slots

We use Azure Deployment Slots to implement our Blue/Green Environment. The screenshot below shows our single Sport Store Database. We also see the production instance of the Web application, OcambSSBlueGreenWeb, and the staging instance, Staging (OcambSSBlueGreenWeb/Staging).

Image6.png

Continuous Deployment Pipeline Diagram

We use Azure Devops to deploy artifacts to our Blue/Green Environment. The artifacts are created by the build process and use the codebase in the Master branch.

There is a deployment pipeline, Deploy – BlueGreen Master Database, with a single stage, Production DB, that deploys the database to production. This is a shared database, so great care must be taken when deploying database changes.

Image7.png

The deployment pipeline for the website, Deploy – BlueGreen Master Website, has two stages.

Staging is used to deploy the website to the Staging Slot. Final testing is done to ensure the deployment is acceptable. It is important to understand that prior to this, extensive testing was completed as sprints unfolded.

Production is used to swap the Production IP Address. Once this is completed, the new changes are available to production users. There is also a pipeline to roll back a release.

Continuous Deployment Pipeline – Azure DevOps

The screenshot below shows the release pipeline for the database. The artifacts were built from the Master branch. An approval is required before the artifact is deployed.

Image8.png

The screenshot below shows the release pipeline for the website. The artifacts were built from the Master branch. There are two stages to the deployment. Each stage requires an approval.

The first stage deploys artifacts to the Staging environment. We perform a set of regression tests to ensure we are ready to move the release to Production. During this time, Production is using the current release. Both Production and Staging are compatible with the database, so they can operate side by side.

When we are comfortable with the release, we change the Production IP Pointer to point to the staging environment by approving the Production stage. No code is deployed here; we only change the Production IP Pointer. If something goes wrong in production, we can easily roll back.

Image9.png

The screenshot below shows the release pipeline for a rollback. This only switches the Production IP Pointer to point back to the previous release. There are no artifacts involved in this case.

image10.png

In Conclusion

This article discussed the approach we are taking to implement a zero-downtime deployment of an application when a breaking schema change is made to the database. We discussed the challenges we faced and our design.

We showed how we set up deployment slots in Azure and how we use Azure DevOps to implement a Blue/Green environment with rollback capability.

The next article will dive into the details of how this was implemented.

Agile Transformation Contracts

046327647-contract-signature.jpeg

It can be challenging to get team members new to Agile all on the same page because, often, there are no clear responsibilities defined for the Agile coach and the team.

This Agile Transformation Contract can be used by teams and an Agile coach to define these responsibilities. It should be signed by the coach and each team member. Feel free to download the contract and modify it to your specific needs.

Agile Transformation Contract – Coach

Agile Transformation Contract – Team Member

Anxiety is an Impediment to Success

The definition of anxiety is:

071929285-business-man-stress-and-anxiet.jpeg
"a feeling of worry, nervousness, or unease, typically about an imminent event or something with an uncertain outcome."

When we work under High Anxiety, it becomes very difficult for us to deliver a high-quality and timely work product.

The situation is worse when our bosses feel the pressure from above and become stressed and anxious.

As an Agile Coach and consultant, I have found myself in situations where I have felt the anxiety building. This article offers some tips I use to help myself when I feel anxious and describes how I work with people around me who are anxious.

High Anxiety is Evil

It is normal to feel stress. Our professions are stressful; we have deliverables we need to complete. We attend daily standups and report our progress to everyone. This can definitely cause anxiety. 

When we spend much of our time worrying and find it hard to focus on doing our work, we are experiencing High Anxiety. This can really inhibit our ability to perform our work and be successful.

Head it off

The first priority is being able to tell when it is coming. A huge component of anxiety is fear. When you find it difficult to concentrate on your task because you are fearful of succeeding, that is a warning sign. If you spend more time worrying than working, you are in a High-Anxiety situation. 

Tackle the Problem

It is easy for anxiety to take over your mind and cloud your thinking. As you feel the anxiety building, take time off and tackle the problem. Write down some notes about what you will do to be successful. 

There are times when I lie in bed, worrying about something. When this happens, I spend a few minutes updating my notes. I have my Chromebook by my bed ready. Having opened up my notes and updated them, I can often fall asleep again.

This does not need to be a long and drawn out affair, spend a few minutes and take formal notes. Then execute on the notes you took. When the notes are no longer correct, change them and try again. This inspect and adapt approach causes you to focus on the problem. As you begin to succeed, the anxiety will fade.

Manage Your Anxiety

Once, I was often in a state of High Anxiety. If a situation arose that caused me anxiety, I would move right into the dysfunctional state of High Anxiety. I would wish that the anxiety would go away.

Then I had an epiphany. I realized that anxiety was a part of life. I had no control over many things, and anxiety was like gravity. I learned that, by heading off the anxiety and tackling the problem, I could lessen the time that I spent in a High-Anxiety state. 

At first, there was not much improvement. Something would happen and bam, there I was in a High-Anxiety state, sometimes for days. But, eventually, the days became hours and sometimes even less than that. Now I use this approach multiple times each week.

High Anxiety Happens to Your Boss Too

Everything I have said about anxiety happens to your boss too. She is human and experiences stress just like you do, maybe more. When your boss is under High Anxiety, her decision-making may be affected.

The most likely effect of her anxiety on you is an erratic change in plans—either a change in date or in scope. You may be asked to do something that is not reasonable.

The first thing I recommend is empathy. Put yourself in your boss's shoes and try to understand what she is going through. This will help you make sense of why things are changing. 

Offer alternatives to the approach that you find unreasonable. If you are following an agile process like Scrum or Kanban, it is easier to do this. Adding a story to an in-progress sprint or exceeding a Kanban constraint breaks major agile rules. Pull in the Scrum Master and ask for his help.

If this does not change your boss's mind, explain the consequences of executing her request. Get her to agree to the risks before you execute the changed plan. 

Your job is to explain the situation as you see it to your boss, but, in the end, the decision is hers. You may have to salute her and implement what she says even if you do not agree with it. It is very important that you do your very best to succeed in your execution of the new plan. Do not gripe or complain to others on the team. 

Your boss knows in her heart when she is asking for unreasonable things. If you behave in the manner I recommend, she will remember what you did and be grateful. 

In Conclusion

The bottom line is, you cannot control the events around you, but you can control how you react to them. Once you figure this out, you will be more successful and have a much happier life. If I was able to do it, so can you.