Pages

Wednesday, February 22, 2017

TFS Continuous Integration Walk Through Part 5b - Multiple Solutions: Simple Project References

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.

https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 5a - Multiple Solutions - Overview




A New Beginning

One of my goals for the Multiple Solutions walkthroughs is to put the TFS repository in a, let's say, less than stellar organizational state, and then clean up. To do that, I'll start with a new, clean TFS collection.

Open TFS Administration Console, select Team Project Collections, and click Create Collection.

2017-02-20_145443

Choose a silly name like "CICollection2". Next.

2017-02-20_145927

Verify the SQL server instance and create a new database. Next, Verify, Create.

2017-02-20_150035

It'll take a few minute to create the collection. Click Complete, then Close. You can close the Admin Console, too.

2017-02-20_150517

Connect Visual Studio to the New Collection

Complaint
This was stupidly difficult to figure out. Google searches yielded nothing.

Open Visual Studio and the Team Explorer, then open Manage Connections.

2017-02-20_161007

Drop down "Manage Connections" and choose Connect to Team Project.

2017-02-20_161132

Select CICollection2, then click Connect.

2017-02-20_161318

This is the first time I'm using this collection, so I need to map my TFS Workspaces.

Complaint
I hate TFS workspaces.

2017-02-20_161528

Pick one of the "map workspaces" links, take the defaults, click Map & Get.

2017-02-20_161942

Now I'm connected to the new collection.

2017-02-20_162046

Create a new Team Project

In Visual Studio Team Manager, create a new Team Project by choosing Home > Projects & My Teams > Create Team Project.

2017-02-20_162400

Remember, one of my intentions is a messy collection. So, I'll name my team project Main. (I'm tempted to name it "Turing" and make life even worse, but this will do.) Click Next.

2017-02-20_162607

The default Agile process is fine. Next.

2017-02-20_162652

We're sticking with Team Foundation Version Control. Click Finish.

2017-02-20_162805

After a few seconds our team project is created.

2017-02-20_162930

Add the TuringDesktop Solution

In Visual Studio, create a new Console project. In the dialog, check the Add to Source Control box.

2017-02-20_163059

In the source control dialog, accept the defaults. We're adding our solution folder to the root.

2017-02-20_163216

So, right now our TFS collection structure is:

CICollection2
|_$/Main
  |_TuringDesktop

If you remember from the previous part, our desktop app is going to have two dependencies: the Magic8Engine, and TextColor. Right now, I don't know Magic8 is going to be a shared NuGet package, so I just add it as a new Class Library project. The same with TextColor.

I'm also going to need a unit test project, so I'll add an MSTest project now as well. I'm going to name it TuringDesktop.Tests, even though it's going to contain tests for all my projects.

2017-02-20_164651

In the end, this is my solution's folder structure.

2017-02-20_164939

Our First Suite

References

Assign TextWriter to MemoryWriter
Mocking System Console Behaviour
Magic 8-Ball

Below is all the code and tests for the solution. This is my super-pre-release versions, and has some (mostly intentional) problems.

The Solution Projects

Right now I've got everything in one solution. I'll explain each project in turn.

TuringDesktop

This is a Console application. It references the Magic8Engine and TextColor projects.

2017-02-22_102302

I'm simulating a database with a class that returns a list of answers, and is initialized with a connection string. For this simple sample, the connection string is just a name like "production". If the connection string is named something other than "production", then the name is prepended to each answer. For example, if the connection string were "stage", an answer might be "STAGE: It is certain."

using System;
using System.Collections.Generic;
using System.Linq;

namespace TuringDesktop
{
    public static class AnswerDatabase
    {
        /// <summary>
        /// Any dbName other than "prod" prepends answers with dbName.
        /// Ex. dbName = "stage", then answer is "STAGE: It could be so."
        /// </summary>
        /// <param name="connectionString"></param>
        public static IEnumerable<string> GetAnswers(string connectionString)
        {
            if (String.IsNullOrWhiteSpace(connectionString))
            {
                throw new ArgumentException("dbName cannot be null or empty");
            }
            if (connectionString == "production")
            {
                return DefaultAnswers;
            }
            else
            {
                return DefaultAnswers.Select(a => connectionString.ToUpper() + ": " + a);
            }
        }

        private static IEnumerable<string> DefaultAnswers
        {
            get
            {
                yield return "It is certain";
                yield return "It is decidedly so";
                yield return "Without a doubt";
                yield return "Yes, definitely";
                yield return "You may rely on it";
                yield return "As I see it, yes";
                yield return "Most likely";
                yield return "Outlook good";
                yield return "Yes";
                yield return "Signs point to yes";
                yield return "Reply hazy try again";
                yield return "Ask again later";
                yield return "Better not tell you now";
                yield return "Cannot predict now";
                yield return "Concentrate and ask again";
                yield return "Don't count on it";
                yield return "My reply is no";
                yield return "My sources say no";
                yield return "Outlook not so good";
                yield return "Very doubtful";
            }
        }
    }
}

Here's the Program class with my Main code. There are a few things to note:

  • I removed the args parameter from the Main method. I don't need it.
  • I made the Main method public, so it can be call from unit tests.
  • I'm using Dependency Injection for the answer engine (Oracle) and the console text colorizer (ConsoleColorizer).
  • There are three settings using a string literal: _connectionString, questionColor and answerColor.
using System;
using TextColor;
using Magic8Engine;

namespace TuringDesktop
{
    public class Program
    {
        static string _connectionString = "production";
        static IConsoleColorizer _colorizer = new ConsoleColorizer();
        static IOracle _oracle = new Oracle(AnswerDatabase.GetAnswers(_connectionString));

        public Program() { }
        public Program(IConsoleColorizer colorizer, IOracle oracle)
        {
            _colorizer = colorizer;
            _oracle = oracle;
        }

        public static void Main()
        {
            string name = "";
            string answer = "";
            string question = "";
            string questionColor = "Red";
            string answerColor = "green";

            Console.Write("Welcome! I'm AT. Please tell me your name. >> ");
            name = Console.ReadLine();
            Console.WriteLine("Good to meet you, " + name + ". "
                + "Ask me a yes-or-no question and I'll give you an answer. "
                + "When you're finished, say 'bye'.");
            do
            {
                try
                {
                    _colorizer.ColorizeWriteLine("Question?", questionColor);
                    question = Console.ReadLine();
                    if (question.ToLower() == "bye") { break; }
                    answer = _oracle.GetAnswer();
                    _colorizer.ColorizeWriteLine(answer, answerColor);
                    Console.WriteLine();
                }
                catch (Exception ex)
                {
                    Console.WriteLine("Error: " + ex.GetBaseException().Message);
                    Console.Write("Press any key to quit.");
                    Console.Read();
                    return;
                }
            }
            while (true);
        }
    }
}

The program's output is simple enough. It greets the user, asks for a name, then starts answering questions. Text coloring is used to make things clear.

2017-02-22_105715

Magic8Engine

I have an IOracle inteface, so that it's easier to use in unit tests.

namespace Magic8Engine
{
    public interface IOracle
    {
        string GetAnswer();
    }
}

The concrete class is initialized with a list of answers (the "database") from which it randomly selects.

using System;
using System.Collections.Generic;
using System.Linq;

namespace Magic8Engine
{
    public class Oracle : IOracle
    {
        Random _random = new Random();
        private IEnumerable<string> _answers = new List<string>();

        public Oracle(IEnumerable<string> answers)
        {
            _answers = answers;
        }

        public string GetAnswer()
        {
            if (_answers.Count() == 0)
            {
                return "";
            }
            int index = _random.Next(_answers.Count());
            return _answers.ToArray()[index];
        }
    }
}

TextColor

The ConsoleColorizer class also implements an interface to improve unit testing.

using System;

namespace TextColor
{
    public interface IConsoleColorizer
    {
        void ColorizeWriteLine(string text, string colorName, bool resetColor = true);
        ConsoleColor GetConsoleColor(string colorName);
        void ResetConsoleColor();
        void SetConsoleColor(string colorName);
    }
}

The concrete class sets properties on the Console object.

using System;

namespace TextColor
{
    public class ConsoleColorizer : IConsoleColorizer
    {
        public void ColorizeWriteLine(string text, string colorName, bool resetColor = true)
        {
            SetConsoleColor(colorName);
            Console.WriteLine(text);
            if (resetColor) { Console.ResetColor(); }
        }

        public ConsoleColor GetConsoleColor(string colorName)
        {
            ConsoleColor color;
            if (Enum.TryParse<ConsoleColor>(colorName, true, out color))
            {
                return color;
            }
            else
            {
                throw new ArgumentException("Invalid ConsoleColor: " + colorName);
            }
        }

        public void SetConsoleColor(string colorName)
        {
            Console.ForegroundColor = GetConsoleColor(colorName);
        }

        public void ResetConsoleColor()
        {
            Console.ResetColor();
        }
    }
}

This should raise a red flag. System.Console is a dependency. Why didn't I abstract that out? I could have. For example, I could have created an IConsole interface with just the features I'm using, then an explicit SystemConsole wrapper class that implented the interface. But it turned out I could test Console successfully without needing to make it entirely replacable.

Good question though. Glad you're thinking.

Tests

Finally, our test project has a class for each project we're testing.

2017-02-22_112309

Here are the unit tests for each project. I'm using nested classes to keep the tests readable.

Magic8 Tests

Note how these test make use of passing in a custom "answer database".

using System.Collections.Generic;
using System.Linq;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Magic8Engine;

namespace TuringDesktop.Tests
{
    [TestClass]
    public class OracleClass
    {
        [TestClass]
        public class GetAnswer_Should: OracleClass
        {
            static List<string> _answers;

            [TestInitialize]
            public void TestInitialize()
            {
                //Start each test with empty list;
                _answers = new List<string>();
            }

            [TestMethod]
            public void ReturnARandomAnswerEachTimeItIsCalled()
            {
                //arrange
                _answers.AddRange(new string[] { "a", "b", "c" });
                var oracle = new Oracle(_answers);
                //act
                List<string> answers = new List<string>();
                for (int i = 0; i < 10; i++)
                {
                    answers.Add(oracle.GetAnswer());
                }
                int uniqueAnswers = answers.Distinct().Count();
                //assert
                Assert.IsTrue(uniqueAnswers > 1);
            }

            [TestMethod]
            public void ReturnAnEmptyAnswerIfAnswerListIsEmpty()
            {
                //arrange
                var oracle = new Oracle(_answers);
                //act
                string actual = oracle.GetAnswer();
                //assert
                Assert.AreEqual("", actual, "Answer list count: " + _answers.Count());
            }
        }
    }
}

TextColor Tests

Neither of these tests confirms the correct color gets set. That's really something best left to a human to verify. But notice in WriteTheUserEnteredString how I'm using the Console.SetOut property to let me verify that user input actually gets written to the console.

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.IO;
using TextColor;

namespace TuringDesktop.Tests
{
    [TestClass]
    public class ConsoleColorizerClass
    {
        [TestClass]
        public class GetConsoleColor_Should
        {
            [TestMethod]          
            public void ReturnTheEnumUsingCaseInsensitive()
            {
                //arrange
                var tc = new ConsoleColorizer();
                string colorName = "gReEn";
                Exception actualEx = null;
                string errorMsg = "";
                //Will change to Green below.
                ConsoleColor color = ConsoleColor.Black;
                //act
                try
                {
                    color = tc.GetConsoleColor(colorName);
                }
                catch (Exception ex) { actualEx = ex; errorMsg = "Threw error " + ex.GetBaseException().Message; }
                Assert.IsNull(actualEx, errorMsg);
                Assert.AreEqual(ConsoleColor.Green, color);
            }

            [TestMethod]
            public void WriteTheUserEnteredString()
            {
                //arrange
                //Store the Console output in a stringwriter.
                StringWriter sw = new StringWriter();
                Console.SetOut(sw);
                var tc = new ConsoleColorizer();
                //act
                tc.ColorizeWriteLine("blamo", "Red");
                //assert
                Assert.AreEqual("blamo\r\n", sw.ToString());
            }
        }
    }
}

Console App Tests

If you're thinking the Main method should be refactored, you're right. I'll do that later. For now, it's interesting to see how we can test a console app.

First, I'll create my mock objects. I'm not even checking MockOracle's GetAnswer method, but that's OK. What I'm guaranteeing is my unit tests have no external dependencies.

MockOracle

    public class MockOracle : Magic8Engine.IOracle
    {
        public string GetAnswer()
        {
            return "This is a fake answer";
        }
    }

MockColorizer

In this mock, I store the simulated user's input in a list, and I always return a black console color. I don't set or reset console colors.

Important
Remember, I'm not testing if the colorizer works. There are already unit tests for that. I'm testing if the Main method works. I just need my mock objects to return consistent results quickly.

    public class MockColorizer : TextColor.IConsoleColorizer
    {
        public List<string> ConsoleLines = new List<string>();


        public void ColorizeWriteLine(string text, string colorName, bool resetColor = true)
        {
            ConsoleLines.Add(text);
        }

        public ConsoleColor GetConsoleColor(string colorName)
        {
            return ConsoleColor.Black;
        }

        public void ResetConsoleColor() { }

        public void SetConsoleColor(string colorName) { }
    }

Here are the tests for the Main method. Notice how I'm using StringWriters for both Console.In and Console.Out. This lets me buffer all the responses a user would make, and capture the console's output. I use a couple of helper methods to make this work.

What I'm not trying to do is test if Console works. I'm testing if I'm my code that writes to the console works. But, unfortunately, I do have a dependency on Console.

So, are these unit tests, or integration tests? Short answer: integration. I'll make some improvements later to isolate code that doesn't depend on Console.

There's a real danger here that I'll forget to use SendUserInputs, leading to the application hanging. I know this danger exists because I did it. The tests are brittle.

    [TestClass]
    public class ProgramClass
    {
        [TestClass]
        public class Main_Should
        {
            StringWriter _consoleOut = new StringWriter();
            string UserInputs = "";

            [TestInitialize]
            public void TestInitialize()
            {
                //Store the Console output in a stringwriter.
                Console.SetOut(_consoleOut);
            }

            [TestMethod]
            public void DisplayInitialGreetingMessage()
            {
                //arrange
                AddUserInput("Charles");
                AddUserInput("bye");
                SendUserInputs();
                var mockColorizer = new MockColorizer();
                var mockOracle = new MockOracle();
                var program = new Program(mockColorizer, mockOracle);
                string expected = "Welcome! I'm AT. Please tell me your name.";
                //act
                Program.Main();
                //assert
                string output = _consoleOut.ToString();
                Assert.IsTrue(output.IndexOf(expected) >= 0);
            }

            [TestMethod]
            public void DisplayWelcomeWithName()
            {
                //arrange
                AddUserInput("Charles");
                AddUserInput("bye");
                SendUserInputs();
                var mockColorizer = new MockColorizer();
                var mockOracle = new MockOracle();
                var program = new Program(mockColorizer, mockOracle);
                string expected = "Good to meet you, Charles.";
                //act
                Program.Main();
                //
                //assert
                string output = _consoleOut.ToString();
                Assert.IsTrue(output.IndexOf(expected) >= 0, "Output was: " + output);
            }

            #region "Test Helpers"
            private void AddUserInput(string value)
            {
                UserInputs += value + Environment.NewLine;
            }

            private void SendUserInputs()
            {
                //Send all the inputs needed for the Read and ReadLine statements
                StringReader consoleIn = new StringReader(UserInputs);
                Console.SetIn(consoleIn);
            }
            #endregion
        }
    }

Running my tests, I get nice, readable output.

2017-02-22_115207

Initial Continuous Integration Build Definition

In Visual Studio Team Explorer, click Builds.

2017-02-22_115629

Click New Build Definition.

2017-02-22_115709

This will open the web page and prompt for a definition. This is nice because we're taken directly to our team project builds.

2017-02-22_115934

Choose the Visual Studio template, check the box that says Continuous Integration and accept the defaults. Save the definition, naming it TuringDesktop Suite Build.

If you've been following along, you'll realize that we've never committed any of our source code! Do so now and the build should happen automatically and successfully. Be sure to verify that all the tests ran!

2017-02-22_122709

2017-02-22_122936

Next Up

We have a working application that's automatically built in our CI server. But we've also got some problems:

  • Refactoring for the main application's Console dependency.
  • String literals for settings.
  • Our application's data source is the same in development, CI, and production.
  • No separate integration tests.

In short, we're about to go from nice, sunny sample development to real-world, why-does-this-have-to-be-so-hard programming.


Next Part: TFS Continuous Integration Walk Through Part 5c - Multiple Solutions: Build Settings

Monday, February 20, 2017

TFS Continuous Integration Walk Through Part 5a - Multiple Solutions: Overview

ci-logo This is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 4b - Problems With Traits




Where I'm Headed

For the next walkthroughs, I'm starting with new team projects and sample applications, building a suite of solutions with shared dependencies. My goal is to uncover the various problems that come from:
  • DLL dependencies
  • Project references
  • Web.config and app.config settings
  • Unit tests
  • Integration tests
  • BONUS: Migrating to Git
Along the way, I'm going to completely abuse the TFS team project structure and show how to clean up and organize a repository.

Relationship of Team Projects, Solutions and Build Definitions

Team Projects are intended to manage a software project.
But that doesn't mean there's just one Visual Studio solution for a project. There might be several. For example, our CI Sample projects might end up looking like this:
$/CI Sample (team project)
|_CIConsoleSample
|_CIRestSample
|_CIDocumentation

$/CI Shared (team project)
|_CINuGetSample
What does this mean for automated builds?
  • Maybe the Console now has a dependency on the REST solution, and we need Console to build any time REST changes.
  • Maybe we need the NuGet solution to deploy to a staging location on a successful build, then build dependent solutions pulling this NuGet dll from the staging source.
  • And maybe we don't want the Documentation to build at all.
Important If you're using a Continuous Integration trigger, and any trigger fires, all the solutions in the definition will be rebuilt. For example, let's say the definition builds all solutions in the CI Sample team project above (the default behavior). You have a trigger that says "only build when CIConsole changes." It would still build all three solutions.

This feature of a build definition means that much of the time we'll configure a definition to build just one solution. However, as pointed out above, for a solution that builds an assembly used by multiple projects, you might want to rebuild all the solutions that depend on it, to catch bugs.

As this diagram shows, a team project can include multiple solutions. A build definition can build multiple solutions. A build definition belongs to a single team project, but, it can (with effort) build solutions in other team projects.


team-projects-and-builds

The Projects - A Turing Oracle

My sample suite of applications is a pseudo Turing Test, and is made up of desktop and web apps. The user can ask questions, and get answers that seem to be from a human. I'll start simple, imitating a real-world project progression. In the end, this will be my structure in the TFS repository.
_$Shared
|_$/Magic8Engine

$/TuringDesktop
|_TuringDesktop.sln
  |_TuringConsole.csproj (Dependencies: Magic8Engine, TextColor)
  |_TextColor.csproj

$/TuringWeb
|_TuringWeb.sln
  |_TuringWeb.csproj (Dependencies: TuringSoap)
|_TuringSoap.sln
  |_TuringSoap.csproj (Dependencies: Magic8Engine)

Magic8Engine The Magic8Engine produces the answers, using the eerily accurate Magic 8-Ball Algorithm. It's a shared dll, and lives in a separate folder. It's published as a NuGet package.


TuringDesktop TuringDesktop is a console application for asking questions. The team project has one solution, which itself has two projects.

TuringConsole, the UI, has two dependencies. One is the Magic8Engine dll, the other is on TextColor, which is configured as a project reference.

TextColor is for changing the color of the text. It is only used by TuringConsole.

TuringWeb This is our web interface. The team project has two solutions.

TuringWeb Solution This solution has one project, TuringWeb, which is dependent on the TuringSoap web service.

TuringSoap Solution Our old-fashioned SOAP web service is dependent on the Magic8Engine dll. In the future, the web service could become shared by other applications.

Next Up

In the next part, I'll build my desktop application as if I didn't know anything about the future web application.

Next Part: TFS Continuous Integration Walk Through Part 5b - Multiple Solutions: Simple Project References

Monday, February 6, 2017

TFS Continuous Integration Walk Through Part 4b - Problems With Traits

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 4a - Problems With Traits



 


References
Test: Visual Studio Test
Part 2: Using Traits with different test frameworks
Part 3: Unit testing with Traits and filtering... <-Note that the documentation on mapping "Category" is wrong, see below!
Running selective unit tests in VS 2012 RC using TestCaseFilter
VSTS/TFS Visual Studio Test Task - Filter Criteria
Running unit tests with Test Explorer: Group and Filter
How to: Group and Run Automated Tests Using Test Categories
Run Tests using Visual Studio task

I initially couldn't get the category filtering to work. The documentation noted in the References said that an xUnit trait of "Category" would be automatically matched to the test explorer's "TestCategory".

2017-02-02_112605

But, that's not true. xUnit (and presumably Nunit and others) passes its traits directly. See this exchange on the subject (Brad Wilson developed xUnit).

https://github.com/xunit/xunit/issues/1052

If you use this attribute in xUnit:

[Trait("Category", "manual")]

and this filter:

TestCategory!=manual

the tests will fail. The MS test console will report an error, not run the test as expected. To find the error, you need to look in the build's logs.

2017-02-02_120035

Here's the error message. It's the same message you'd get if you used the command line to run the tests (which is how I first discovered the error).

2017-02-02T18:55:03.9288278Z ##[error]Error: [xUnit.net 00:00:00.3802419]
CIConsoleSample.xUnitTests: Exception discovering tests: No tests matched the filter 
because it contains one or more properties that are not valid (TestCategory). Specify
filter expression containing valid properties (Category, DisplayName, FullyQualifiedName)
and try again.

"Well, OK," I thought, "I'll just add 'Category' into the filter."

TestCategory!=manual | Category!=manual

Now I get two errors:

Error: No tests matched the filter because it contains one or more properties that 
are not valid (Category). Specify filter expression containing valid properties 
(TestCategory, Priority, FullyQualifiedName, Name) and try again.

Information: [xUnit.net 00:00:00.3791321]   Discovering: CIConsoleSample.xUnitTests

Error: [xUnit.net 00:00:00.4843279] CIConsoleSample.xUnitTests: Exception discovering
tests: No tests matched the filter because it contains one or more properties that 
are not valid (TestCategory). Specify filter expression containing valid properties
(Category, DisplayName, FullyQualifiedName) and try again.

Maybe I need to use a Boolean AND in the filter.

TestCategory!=manual & Category!=manual

Nope, I get the exact same error message. After further testing and research, I learned that if there's a filter, at least one test in the assembly must satisfy the filter. In other words, if you filter on TestCategory, at least one test must include that trait, regardless of whether it's needed.

In my case, I'm using a single Visual Studio Test build step to run tests using two frameworks. When I was using [TestCategory("manual")] for MSTest, and [Trait("Category","manual")] for xUnit, and a filter of TestCategory!=manual | Category!=manual, here's what happened.

  1. The MSTest framework was called. The filter was evaluated and no tests with a trait of "Category" were found. So an error was reported.
  2. The xUnit framework was called. The filter was evaluated and no tests with a trait of "TestCategory" were found. So an error was reported.

The first time I ran the tests, I only had a filter TestCategory!=manual. The MSTests succeeded because there was, indeed, a trait with that name.

I did try some other filtering, hoping that filters might evaluate conditions in a "short-circuit" way, but always got the same errors. For example, I tried this:

(FullyQualifiedName~MSTest && TestCategory!=manual) || (FullyQuallifiedName~xUnit && Category!=manual)

This as a problem for reusing build definitions. It would be better if invalid traits were simply ignored, and the tests get run. This is a known problem, as Brad Wilson notes at the end of this thread.

https://github.com/xunit/xunit/issues/610

bradwilson commented on Oct 18, 2016 Unfortunately, this exception is generated by Visual Studio, not by us. We provide the list of traits to them, and they do the filtering; they're the ones generating this message. When you Google for the phrase "No tests matched the filter because it contains one or more properties that are not valid" you'll see hits for all testing frameworks, not just xUnit.net.>

The only workaround is to always include the trait on at least one test in the assembly. Note the trait doesn't need a used value.

    [Fact]
    [Trait("Category","blamo")]
    public void SaveGreetingToCloud()

So, that's not ideal, but it's up to Microsoft to take care of the problem.

Can we mitigate the issue at all? Sort of. We can at least have better management of our trait filters for different test frameworks.

First, let's grant that this is an unusual situation, using multiple test frameworks. The better way to manage this would probably be to have two test runs, one for MSTest and the other for xUnit. Add another build step to our definition, and then to change the Test Assembly fields to search for specifically. In other words, we'd have one build step for MSTest, and one for xUnit, and then we could name the traits as desired. This would rely on developers sticking to some naming conventions.

Warning
Be sure to evaluate your test results when you change settings and look for sane expectations. For example, when I first tried my xUnit build step, I added an extra "l" to .dll. The build didn't fail, it just didn't find any xUnit tests! I needed to look at the log and check that it had found the tests I expected.

MSTest step
2017-02-02_125322

xUnit step
2017-02-02_125431

Now, when running the automated build, there are two test runs, each with its own filter.

Next Part: TFS Continuous Integration Walk Through Part 5 - Multiple Solutions in Team Project

TFS Continuous Integration Walk Through Part 4a - Filtering Tests

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 3 - Notifications




References
Test: Visual Studio Test
Part 2: Using Traits with different test frameworks
Part 3: Unit testing with Traits and filtering... <-Note that the documentation on mapping "Category" is wrong, see below!
Running selective unit tests in VS 2012 RC using TestCaseFilter
VSTS/TFS Visual Studio Test Task - Filter Criteria
Running unit tests with Test Explorer: Group and Filter
How to: Group and Run Automated Tests Using Test Categories
Run Tests using Visual Studio task

In some environments, not all unit tests should be run on the CI server. For example, there might be tests with external dependencies that are expensive to run, such as a fee for verifying data was saved to an external cloud-based site. Or legacy tests that require manual setup or interaction. Or, the most likely, a test that takes a very long time to run. While these should probably be refactored or replaced, the fact is sometimes we don't want all our tests running automatically.

One way to restrict tests running is to use the "Test filter criteria" setting. A simple example is using test categories to isolate the manual tests. "TestCategory", "Priority", "Name", etc are all generically referred to as traits.

Before showing the example, there are two rules for using traits with the Microsoft test console, which is the executable that runs tests regardless of framework.

Test Filter Criteria Rule
If you set test filter criteria, at least one of your unit tests in every assembly must match that criteria. Otherwise, an error will occur.

In practice, this means that if your filter is looking for TestCategory=weekly, any project that uses the build definition, and has tests, must have at least one test with a trait named "TestCategory". (For more detail, see Part 4b - Problems With Traits.)

This will make more sense as you go through the code below.

Add the following class to Program.cs. This is simulating a cloud-based retrieval of a greeting. For some fictitious reason, it's very expensive to do this.


    public class CloudMan
    {
        //This takes ten minutes to run!
        public string GetLastGreeting()
        {
            //Set up connection, pass credentials, etc.
            return "";
        }
    }

Add the following test methods:

MSTest

            [TestMethod]
            [TestCategory("manual")]
            public void SaveGreetingToCloud()
            {
                //arrange
                string value = "Cloud Atlas";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                CloudMan cloudMan = new CloudMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.AreEqual(expectedResult, cloudMan.GetLastGreeting());
            }

xUnit

            [Fact]
            [Trait("TestCategory", "manual")]
            public void SaveGreetingToCloud()
            {
                //arrange
                string value = "Cloud Atlas";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                CloudMan cloudMan = new CloudMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.Equal(expectedResult, cloudMan.GetLastGreeting());
            }

MSTest uses a TestCategory attribute for categories. xUnit uses its Trait attribute to set a "TestCategory" trait (traits are also the way to set "Priority").

Note You must set the xUnit trait name to TestCategory in this example, or the test won't be discovered. See Part 4b - Problems With Traits for more information.

If you run the tests locally right now, the new tests will still fail because the category is ignored. Likewise, if you check in the code, the tests will fail on the CI server because there's no filter, yet.

Applying the Filters

It's possible to filter tests locally. One way is from the command line.

https://msdn.microsoft.com/en-us/library/dd286683.aspx

But you can also group by traits in the Text Explorer, and selectively run the tests. (This doesn't work so well if you're using multiple test frameworks).

2017-02-02_102530

We want our CI server to only run the non-manual tests. Let's apply the filter in TFS Build. Edit the build definition as shown above, and select the Visual Studio Test step. Add the "Test Filter criteria" TestCategory!=manual and Save.

2017-02-02_103011

Click "Queue build" to rerun the build. If you haven't checked in the code with the test categories, do that instead. The result should be just two tests run, and both pass. Here are the Test Results detailed report. Be sure to change the Outcome to show all the tests.

2017-02-02_112051

Next Part: TFS Continuous Integration Walk Through Part 4b - Problems With Traits

Sunday, February 5, 2017

TFS Continuous Integration Walk Through Part 3 - Notifications

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 2 - Create an Automated Build




That's great, right? It worked. But it's not going to help me if I don't know the build failed, and I'm not going to keep a TFS page open, watching it like a hawk. I want email alerts.

Note There used to be another option for notifications: the Build Notification application. It would show build failure notifications in the status tray. But that only works for XAML builds. There appears to be a third party utility that does the same thing for the modern builds: CatLight.

Configure SMTP for Email

References
Configure an SMTP server and customize email for alerts and feedback requests in TFS
TFS Email via Gmail Account <- He enables POP3, but you don't have to.

These steps assume you have administrative access to the administration console.

Open the Team Foundation Server Administration Console, select Application Tier, scroll down to and open Alert Settings.

2017-01-31_114324

In the Settings window, check Enable Email Alerts and configure your SMTP server settings. In my case, I used my Gmail account for testing.

Note There are challenges getting Gmail SMTP to work in TFS that are beyond the scope of this article. That means, it would take too much space to document it all.

2017-01-31_114911

Send a test email.

2017-01-31_115054
2017-01-31_115205

Configure TFS Alerts

References
Receive build notifications

Alerts are configured per team project. You can create project alerts from the Team Explorer > Settings, but below I show how to do it from the web site.

Open the main TFS page and click the Settings gear icon.

2017-01-31_123851

Select the CI Console Sample project. Click "View the project administration page"

2017-01-31_124017

Open the Alerts tab. Click the link under Create New Alert When... "A build fails". In the dialog, enter the Send To email address and click OK.

2017-01-31_124245

Configure Team TFS Alerts

Here's a simple configuration for build notifications. It assumes

  • Not everyone on the team should be notified of build failures.
  • There are two developers on the team who should receive notifications if any build fails.

Open the Overview tab in the Control panel. Click "New team".

2017-03-07_120812

Name the team something like "Build Failures". In the Permissions, leave as Contributers. Click Create Team.

Option
You could also allow anyone on the team to be a Build Admin by choosing that group.

2017-03-07_121236

Edit the team and add any desired team members.

Open the Alerts tab. Click the link under Create New Alert When... "A build fails". Rename as desired. The only difference from a personal alert is the Subscriber. In this case, "Build Failures" is the team who will receive the alert. Click OK.

2017-03-07_121630

Test Notification

Make a change of some kind to the source code (but that doesn't pass the tests!) and check in your changes. That will start a build. When the build fails, you should get an email.

2017-01-31_124659

Test Success

Let's finally get our test passing. Update the Greeting code, confirm the tests pass locally, then check in. You should not get an email alert, and you should see a successful build.

    public class TextMan
    {
        public string Greeter(string value)
        {
            return "Hello, " + value;
        }
    }

2017-01-31_125046

Next Part: TFS Continuous Integration Walk Through Part 4a - Filtering Tests