Automated QA Tests

It is important to have a set of automated QA tests for your product. These tests are executed before any code is committed to version control by the development team. Automated QA tests are also executed nightly by the build environment release.

We use Selenium for automated QA testing. Selenium is a free, open-source solution that supports multiple browsers. There is even a simplified IDE plugin for Firefox.

Selenium also provides a way to develop tests using a development language such as C# or Java, which provides better control over the tests developed. This code is then consumed by a unit test framework. This is the approach we use instead of the IDE available in the Firefox plugin. We use MSTest, like we do for unit tests and integration tests.


The login page being tested is shown above.


We do not actually code all the automated tests directly in MSTest. We develop a framework for our product that encapsulates the connection to browsers as well as details for each area of the product we wish to test. In the diagram above, the product framework performs this task.
The unit test framework is where we implement the actual test. It is shown below.


We set TestCategory to Nightly_Build so that it runs only in the nightly release to the build environment.

The test is written in an easy-to-use manner and is self-documenting. Here we test to see if the user Admin can log in with the password (“TopSecret”).
If we successfully log in, the products appear under the heading of “All Products” To validate the test, we assert that “All Products” appears.


The above code exists within the product framework. It uses Selenium Webdriver logic to locate the username and password on the login page. The admin-submit button is then clicked.

This code is implemented as part of the Sports Store solution. Automated QA tests are run nightly. They are executed as part of the scheduled nightly release in the build environment.


The above diagram depicts the process of the artifact from the current sprint being released into the build environment.


When we run the test on a developer’s machine, we see that it passes.

Test Pass Release.png

Each night, the codebase in the sprint branch is released into the build environment. We can see that all our tests have passed.


Here, we change the password. This will make the test fail.


Test Explorer in Visual Studio now shows that the test has failed.

It is best practice for the development team to run all tests before committing any changes to version control and performing a pull request.

Even if the team does not follow this practice, the code review should catch the failed test. Let’s see what happens if this change makes it into the sprint branch when these best practices are not followed.


We can see that our test fails when the release to the build environment is run. This demonstrates a key aspect of DevOps: we want to know about errors as early as possible before the codebase hits production.

The release to the build environment runs every night, occurring many times during a typical sprint.
Next, we will discuss how we can manage our environments using infrastructure as code.

Learn More

For details on the topics covered in this article, please visit the links below. You may also download the source code for the DevOps Roadmap Lab on Git HubContact me to set up a free meeting where we can dive deeply into the lab material  and discuss your individual needs.

  • Selenium's website is available at
  • The article “Get Started with Selenium Testing in a CI Pipeline” discusses everything you need to set up Selenium for continuous integration and deployment. Be sure to follow all of the steps carefully.
  • A basic article on private agents is available here.
  • You will probably need to set up WinRM. This article teaches you how to do just that.
  • There is an excellent course on setting up an automation framework on PluralSight here. There is a cost associated with it, but a free version is available on YouTube here.