Pages

Monday, February 6, 2017

TFS Continuous Integration Walk Through Part 4a - Filtering Tests

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 3 - Notifications




References
Test: Visual Studio Test
Part 2: Using Traits with different test frameworks
Part 3: Unit testing with Traits and filtering... <-Note that the documentation on mapping "Category" is wrong, see below!
Running selective unit tests in VS 2012 RC using TestCaseFilter
VSTS/TFS Visual Studio Test Task - Filter Criteria
Running unit tests with Test Explorer: Group and Filter
How to: Group and Run Automated Tests Using Test Categories
Run Tests using Visual Studio task

In some environments, not all unit tests should be run on the CI server. For example, there might be tests with external dependencies that are expensive to run, such as a fee for verifying data was saved to an external cloud-based site. Or legacy tests that require manual setup or interaction. Or, the most likely, a test that takes a very long time to run. While these should probably be refactored or replaced, the fact is sometimes we don't want all our tests running automatically.

One way to restrict tests running is to use the "Test filter criteria" setting. A simple example is using test categories to isolate the manual tests. "TestCategory", "Priority", "Name", etc are all generically referred to as traits.

Before showing the example, there are two rules for using traits with the Microsoft test console, which is the executable that runs tests regardless of framework.

Test Filter Criteria Rule
If you set test filter criteria, at least one of your unit tests in every assembly must match that criteria. Otherwise, an error will occur.

In practice, this means that if your filter is looking for TestCategory=weekly, any project that uses the build definition, and has tests, must have at least one test with a trait named "TestCategory". (For more detail, see Part 4b - Problems With Traits.)

This will make more sense as you go through the code below.

Add the following class to Program.cs. This is simulating a cloud-based retrieval of a greeting. For some fictitious reason, it's very expensive to do this.


    public class CloudMan
    {
        //This takes ten minutes to run!
        public string GetLastGreeting()
        {
            //Set up connection, pass credentials, etc.
            return "";
        }
    }

Add the following test methods:

MSTest

            [TestMethod]
            [TestCategory("manual")]
            public void SaveGreetingToCloud()
            {
                //arrange
                string value = "Cloud Atlas";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                CloudMan cloudMan = new CloudMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.AreEqual(expectedResult, cloudMan.GetLastGreeting());
            }

xUnit

            [Fact]
            [Trait("TestCategory", "manual")]
            public void SaveGreetingToCloud()
            {
                //arrange
                string value = "Cloud Atlas";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                CloudMan cloudMan = new CloudMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.Equal(expectedResult, cloudMan.GetLastGreeting());
            }

MSTest uses a TestCategory attribute for categories. xUnit uses its Trait attribute to set a "TestCategory" trait (traits are also the way to set "Priority").

Note You must set the xUnit trait name to TestCategory in this example, or the test won't be discovered. See Part 4b - Problems With Traits for more information.

If you run the tests locally right now, the new tests will still fail because the category is ignored. Likewise, if you check in the code, the tests will fail on the CI server because there's no filter, yet.

Applying the Filters

It's possible to filter tests locally. One way is from the command line.

https://msdn.microsoft.com/en-us/library/dd286683.aspx

But you can also group by traits in the Text Explorer, and selectively run the tests. (This doesn't work so well if you're using multiple test frameworks).

2017-02-02_102530

We want our CI server to only run the non-manual tests. Let's apply the filter in TFS Build. Edit the build definition as shown above, and select the Visual Studio Test step. Add the "Test Filter criteria" TestCategory!=manual and Save.

2017-02-02_103011

Click "Queue build" to rerun the build. If you haven't checked in the code with the test categories, do that instead. The result should be just two tests run, and both pass. Here are the Test Results detailed report. Be sure to change the Outcome to show all the tests.

2017-02-02_112051

Next Part: TFS Continuous Integration Walk Through Part 4b - Problems With Traits

No comments:

Post a Comment