Zero-Downtime Deployment with a Database – Introduction

sqldevops.png

This is the first part of a two-part article that describes how to implement a Blue/Green deployment of an application when a schema change to a database is made. This schema change will break existing code. I also assume that the amount of data in the database is very large.

We have a Staging and a Production instance of the application. After a production release is performed, we want to be able to roll back to the staging instance.

The application, called Sports Store, is a Microsoft MVC 5 application using a SQL server database. The application uses Azure as the execution environment. Azure Deployment Slots are used to facilitate rollbacks. We also use Azure DevOps as the CI/CD tool, and all aspects of the process are automated.

This is an oversimplified example, but my hope is that it touches all the points a real application would.

Our Challenge

We use a simple Customers table, as shown below. We will be renaming the LastName field to SurName. Remember, this application has been running in production using the LastName field. We also assume that there are 1 million records in the Customer Table.

Image1.png

Our application has a simple way to browse the data and perform CRUD Operations. LastName will be changed to SurName.

Image2.png

Here is the data entry screen. Again, LastName will be changed to SurName.

Image3.png

My implementation is based on the excellent article Zero-Downtime Deployment With a Database.

Design

The biggest design decision I had to make involved the database. I could have a separate database for Production and Staging. This would make the deployments easier, since each environment would have its own isolated database.

However, because of the large amount of data involved, I decided to have a single database shared by both environments. This makes deployments trickier but would not require the movement of vast amounts of data between environments. The diagram below depicts the design I chose.

Image4.png

I also decided to associate a version number with the web application and the database. There is also a configuration setting for the web application to identify the database version it is compatible with.

Image5.png

In the above diagram, we see the database version is 2.0.3. The current production version is 2.2.6, and the stage version is 2.2.5. Both Production and Stage are compatible with database version 2.0.3. As long as Production and Stage are compatible with the same database, we can safely roll back a release. A release is rolled back by changing where the Production IP points to.

Deployment Slots

We use Azure Deployment Slots to implement our Blue/Green Environment. The screenshot below shows our single Sport Store Database. We also see the production instance of the Web application, OcambSSBlueGreenWeb, and the staging instance, Staging (OcambSSBlueGreenWeb/Staging).

Image6.png

Continuous Deployment Pipeline Diagram

We use Azure Devops to deploy artifacts to our Blue/Green Environment. The artifacts are created by the build process and use the codebase in the Master branch.

There is a deployment pipeline, Deploy – BlueGreen Master Database, with a single stage, Production DB, that deploys the database to production. This is a shared database, so great care must be taken when deploying database changes.

Image7.png

The deployment pipeline for the website, Deploy – BlueGreen Master Website, has two stages.

Staging is used to deploy the website to the Staging Slot. Final testing is done to ensure the deployment is acceptable. It is important to understand that prior to this, extensive testing was completed as sprints unfolded.

Production is used to swap the Production IP Address. Once this is completed, the new changes are available to production users. There is also a pipeline to roll back a release.

Continuous Deployment Pipeline – Azure DevOps

The screenshot below shows the release pipeline for the database. The artifacts were built from the Master branch. An approval is required before the artifact is deployed.

Image8.png

The screenshot below shows the release pipeline for the website. The artifacts were built from the Master branch. There are two stages to the deployment. Each stage requires an approval.

The first stage deploys artifacts to the Staging environment. We perform a set of regression tests to ensure we are ready to move the release to Production. During this time, Production is using the current release. Both Production and Staging are compatible with the database, so they can operate side by side.

When we are comfortable with the release, we change the Production IP Pointer to point to the staging environment by approving the Production stage. No code is deployed here; we only change the Production IP Pointer. If something goes wrong in production, we can easily roll back.

Image9.png

The screenshot below shows the release pipeline for a rollback. This only switches the Production IP Pointer to point back to the previous release. There are no artifacts involved in this case.

image10.png

In Conclusion

This article discussed the approach we are taking to implement a zero-downtime deployment of an application when a breaking schema change is made to the database. We discussed the challenges we faced and our design.

We showed how we set up deployment slots in Azure and how we use Azure DevOps to implement a Blue/Green environment with rollback capability.

The next article will dive into the details of how this was implemented.

Agile Transformation Contracts

046327647-contract-signature.jpeg

It can be challenging to get team members new to Agile all on the same page because, often, there are no clear responsibilities defined for the Agile coach and the team.

This Agile Transformation Contract can be used by teams and an Agile coach to define these responsibilities. It should be signed by the coach and each team member. Feel free to download the contract and modify it to your specific needs.

Agile Transformation Contract – Coach

Agile Transformation Contract – Team Member

Anxiety is an Impediment to Success

The definition of anxiety is:

071929285-business-man-stress-and-anxiet.jpeg
"a feeling of worry, nervousness, or unease, typically about an imminent event or something with an uncertain outcome."

When we work under High Anxiety, it becomes very difficult for us to deliver a high-quality and timely work product.

The situation is worse when our bosses feel the pressure from above and become stressed and anxious.

As an Agile Coach and consultant, I have found myself in situations where I have felt the anxiety building. This article offers some tips I use to help myself when I feel anxious and describes how I work with people around me who are anxious.

High Anxiety is Evil

It is normal to feel stress. Our professions are stressful; we have deliverables we need to complete. We attend daily standups and report our progress to everyone. This can definitely cause anxiety. 

When we spend much of our time worrying and find it hard to focus on doing our work, we are experiencing High Anxiety. This can really inhibit our ability to perform our work and be successful.

Head it off

The first priority is being able to tell when it is coming. A huge component of anxiety is fear. When you find it difficult to concentrate on your task because you are fearful of succeeding, that is a warning sign. If you spend more time worrying than working, you are in a High-Anxiety situation. 

Tackle the Problem

It is easy for anxiety to take over your mind and cloud your thinking. As you feel the anxiety building, take time off and tackle the problem. Write down some notes about what you will do to be successful. 

There are times when I lie in bed, worrying about something. When this happens, I spend a few minutes updating my notes. I have my Chromebook by my bed ready. Having opened up my notes and updated them, I can often fall asleep again.

This does not need to be a long and drawn out affair, spend a few minutes and take formal notes. Then execute on the notes you took. When the notes are no longer correct, change them and try again. This inspect and adapt approach causes you to focus on the problem. As you begin to succeed, the anxiety will fade.

Manage Your Anxiety

Once, I was often in a state of High Anxiety. If a situation arose that caused me anxiety, I would move right into the dysfunctional state of High Anxiety. I would wish that the anxiety would go away.

Then I had an epiphany. I realized that anxiety was a part of life. I had no control over many things, and anxiety was like gravity. I learned that, by heading off the anxiety and tackling the problem, I could lessen the time that I spent in a High-Anxiety state. 

At first, there was not much improvement. Something would happen and bam, there I was in a High-Anxiety state, sometimes for days. But, eventually, the days became hours and sometimes even less than that. Now I use this approach multiple times each week.

High Anxiety Happens to Your Boss Too

Everything I have said about anxiety happens to your boss too. She is human and experiences stress just like you do, maybe more. When your boss is under High Anxiety, her decision-making may be affected.

The most likely effect of her anxiety on you is an erratic change in plans—either a change in date or in scope. You may be asked to do something that is not reasonable.

The first thing I recommend is empathy. Put yourself in your boss's shoes and try to understand what she is going through. This will help you make sense of why things are changing. 

Offer alternatives to the approach that you find unreasonable. If you are following an agile process like Scrum or Kanban, it is easier to do this. Adding a story to an in-progress sprint or exceeding a Kanban constraint breaks major agile rules. Pull in the Scrum Master and ask for his help.

If this does not change your boss's mind, explain the consequences of executing her request. Get her to agree to the risks before you execute the changed plan. 

Your job is to explain the situation as you see it to your boss, but, in the end, the decision is hers. You may have to salute her and implement what she says even if you do not agree with it. It is very important that you do your very best to succeed in your execution of the new plan. Do not gripe or complain to others on the team. 

Your boss knows in her heart when she is asking for unreasonable things. If you behave in the manner I recommend, she will remember what you did and be grateful. 

In Conclusion

The bottom line is, you cannot control the events around you, but you can control how you react to them. Once you figure this out, you will be more successful and have a much happier life. If I was able to do it, so can you.
 

 


 


 


 

What Is Your Mindset?

047253707-new-mindset-new-results-sticky.jpeg

Whenever I start coaching a new team and promoting an Agile and DevOps approach to software delivery, it is common for some teams to procrastinate or even resist an automated delivery pipeline. Some teams feel it is better to get features manually deployed into production as soon as possible and that the time to develop an automated pipeline will minimize the number of features that can be delivered. 

Of course, this is true. Time is a limited resource, and the time to develop automated release pipelines takes away from delivering “true” features.

Some teams do not think this way. They see the value in investing the time in automation with the belief that it pays off in the long run.

I conclude that a team’s position on this depends on what a team mindset is and what they consider value to be.

Linear Mindset

Some teams view the delivery process as linier manner. Here we manually create our Dev, QA and Production environments. 

Imag01.png

We use version control and develop our product in the Dev Environment. When we are ready to test, we deploy the codebase to the QA Environment where the QA Team formally tests the product features. When a release has passed our testing, it is deployed to the Production Environment. 

Automation is not necessarily ignored, but the value comes from getting features into production as fast as possible. Time spent on automation takes away from delivering features into production. 

Teams who follow this emphasize the value of deploying features into production and minimizing that of automation. Their mindset causes them to have this perspective.

Cyclical Mindset

Image02.png

In reality, delivery is not linear—it is cyclical. We want to move through the development and test cycle and into production as fast and as easily as we can. The way to do this is to follow DevOps principles and automate everything if possible. Things that are not automated should be carefully documented. 

A full discussion on DevOps is beyond the scope of this article; however, some general points are highlighted below:

  • Infrastructure as Code – We can define each of our environments in code and store it in version control. This makes it easy to manage changes and deployments. 
  • Continuous Integration – As code is committed to version control, it is then compiled and undergoes unit tests. An artifact is then created for deployment to our other environments.
  • Continuous Deployment – The artifact created by Continuous Integration is deployed to other environments for testing. Integration Tests and Automated QA Tests are executed at this time. Manual testing is also performed.
  • Production Deployment – The same artifact we have been using is deployed to production. If zero down time is needed, we use Blue/Green deployment techniques.

If we expand this over the life of our product, the process looks like the diagram below. This allows us to deliver our product to our customer quickly and with high quality.

Imag03.png

It is very hard to do all this automation, and it does take time and money. If you have read this and still think it is not a fit for your situation, consider the following points:

  • It is a very competitive world. If you cannot deliver software to your customer quickly, your competition will.
  • You do not know what your customer needs until it is in production and used. If you cannot deliver software quickly to your customer, you cannot respond to your customer’s feedback on time.
  • When defects are found, you need to fix it fast without affecting the process development.

Teams who have accepted this balance the value of deploying features into production and the value we get by automation. They understand the investment in automation has high value.

Learn More

Some links to more information on DevOps is included below: