A pleasant walk through computing

Comment for me? Send an email. I might even update the post!

TFS Continuous Integration Walk Through Part 2 - Create an Automated Build


ci-logo
This is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Previous Part: TFS Continuous Integration Walk Through Part 1 - Installing TFS and Checking In a Test Project

Add the Solution to a New Build Definition

References
Build your .NET app for Windows

You can get to the web-based "create new build" wizard a couple of ways.

  1. In the Visual Studio solution, switch to Team Explorer, click the Home icon, then drop down Home and choose Builds.
     2017-01-31_101042
  2. On the TFS web site (http://[tfs-site]:8080/tfs), select your team project if visible, or click Browse and navigate to it that way. On the project page, open the Build menu and click the plus sign.
     2017-01-31_102849

Either way gets to the "Create new build definition" wizard. Choose "Visual Studio" and click Next.

2017-01-31_102926

Choose the Repository source, in this case the CI Console Sample. We only have one agent queue, the default. Click Create.

2017-01-31_103317

This adds a set of build steps to the definition. Click Save and name the definition. Definitions can be used by multiple solutions, so something generic like "Default Visual Studio Build" will work.

Note The Visual Studio Test step will auto-discover many third-party test adapters, including xUnit. It's not necessary to set the path in the step's Advanced > Path to Custom Test Adapters field for this example.

2017-01-31_103740

Let's test the build with the default steps. Click "Queue build", the OK.

2017-01-31_104047

The build will run in a pretty cool web-based console window. As expected, the unit tests fail.

2017-01-31_104540

There's also a summary error message at the end.

2017-01-31_104648

BuildConfiguration Default = Release

References
Unit testing on a build server : Release or Debug code?

Notice, above, when we queued the build that we got a dialog letting us set some variables. These variables are part of the build definition, in the Variables tab.

2017-01-31_104700

At first, I thought this was a poor choice for a default. And, I do have a concern. But after reading the article referenced above, I realized...

CI Principle
Auto-build your application to the code your customers will be using.

When would building in Release mode be a problem? The first thing I thought of was web.config transforms. What if you have this appSetting in your default web.config...

<appSettings>
  <add key="SysopEmail" value="developers@example.com" />
</appSettings>

and this one in Web.Release.config...

<appSettings>
  <add key="SysopEmail" value="live-support@example.com" xdt:Transform="Replace" xdt:Locator="Match(key)" />
</appSettings>

And, further, you have a method that's being tested that looks like this:

public void SaveCustomer(ICustomer customer) 
{
  var service = new CustomerService();
  try 
  {
    service.Save(customer);
  }
  catch (Exception ex)
  {
    MailService.SendError(ConfigurationManager.AppSettings["SysopEmail"], ex);
  }
}

Now, CustomerService should be mocked, and that should lead to no errors being generated, but "should" has nothing to do with reality. Do we want our CI tests to email our production support on error? No, because they'll think there's a problem in the production system.

There's anything wrong with the method per se. You can argue that settings like this should be in a database. Fine. But until then, what to do?

I don't know. I'm just pointing out that in some cases, the CI build configuration might need to be something other that Release.

Enable and Test Continuous Integration

Let's test that a build will run the next time we commit source code changes. And, let's be a sloppy developer, too.

First, we need to configure the build definition to trigger when we check in. On the TFS site, in the team project, choose the Build menu, select the Default Visual Studio Build, and click Edit.

2017-01-31_110712

Open Triggers, check Continuous Integration, leave the defaults, and Save.

2017-01-31_110924

Back in our solution, change the TextMan.Greeter method as follows. Note that this will not pass.

    public class TextMan
    {
        public string Greeter(string value)
        {
            return value;
        }
    }

You can run the tests if you want to prove they fail.

2017-01-31_105424

Now for the fun. On the TFS web site, go to the team project's Build tab, select the Default Visual Studio Build, and open the Queued view. Keep this page visible if you can. We want to see the build automatically start.

2017-01-31_105936

In Team Explorer, click Pending Changes

2017-01-31_105614

Enter a comment and click Check In.

2017-01-31_110109

Back on the TFS site page, click Refresh and the build should appear.

2017-01-31_111112

You can open the build and wait for it to finish (i.e. "fail"). The summary page nicely tells you who authored the last changes.

2017-01-31_111405

Our build server is successfully automatically building on check in!

Next Part: TFS Continuous Integration Walk Through Part 3 - Notifications

TFS Continuous Integration Walk Through Part 1 - Installing TFS and Checking In a Test Project

ci-logoThis is part of a series of walk throughs exploring CI in TFS, starting from the ground up. The entire series and source code are maintained at this BitBucket repository.
https://bitbucket.org/bladewolf55/tfs-ci-samples

Bonus: Uninstall!

Here's how to uninstall TFS, if necessary.

  1. Open Windows Services.
  2. Stop the VSO Agent service(s).
  3. Copy the service name(s).
  4. Delete the service from admin command prompt: sc delete vsoagent.Nesbit.Agent-Nesbit
  5. Open SQL Management Studio or Visual Studio SQL Server Object Explorer.
  6. Delete the Tfs databases.
  7. Open Programs and Features.
  8. Select Team Foundation Server Express 2015, choose Change, then Uninstall.

Install

Requirements and compatibility

Note These instructions are for TFS 2015 Update 3

 

Note about Visual Studio and MS Build Technically, you don't need to install Visual Studio to build using TFS. But, practically, you do, certainly if you use the recommended Visual Studio task. If you want to stick with just MS Build, you might be able to get away with installing the  2015 Build Tools

 

This walkthrough installs TFS, a build agent, and Visual Studio on one machine to keep the demonstration easy.

  1. Download latest version (2015 Update 3 as of this writing). ISO

  2. If you're using SQL 2016, Download and install the C++ Runtime prerequisite. http://support.microsoft.com/kb/3138367

  3. Run installer
    2017-01-27_142237

    2017-01-27_142348

  4. After installation, the Configuration Center opens. Select Basic Server and click Start Wizard.

    2017-01-27_143404

  5. Choose whether to participate in the Visual Studio Experience Improvement Program and click Next.
    2017-01-27_143548

  6. Choose whether to install SQL Express or use an existing instance. In my case, I already have SQL 2016 Express installed. Click Next.
    2017-01-27_143631

  7. The default SQL instance is displayed, or enter a named instance. This is the instance in which the TFS databases will be created. Click Next.
    2017-01-27_143755

  8. The next screen is what threw me. "Install and configure a build and deployment agent" is unchecked by default. You can install an agent later (I show how), but it's just as easy to install one now. Check the box and accept the defaults. Note the installation folder for later use. Click Next.

    2017-01-27_144005
    2017-01-27_144159

  9. Click Next again at the review screen.
    2017-01-27_144341

  10. Readiness checks will run. If something fails, you'll need to fix it before continuing, then click "Click here to rerun Readiness Checks. Once everything passes, click Configure.
    2017-01-27_144511

  11. TFS and the agent will be configured. This creates databases, the agent service, and the website. When complete, click Next.
    2017-01-27_144541

  12. Note the web site location. It's a good idea to open the web site now. Click Close.
    2017-01-27_150639

  13. You're done with the wizard, so click Close. This will open the Server Administration Console.
    2017-01-27_150745

  14. Select Build and Release. This page has some instructions. You can go directly to the TFS Settings > Agent page by click on Download Build and Deployment Agent. (Note: We will not be using the legacy XAML Build Configuration.). You can also get to the Settings page via the gear icon on the TFS web page.

    2017-01-27_151017

    2017-01-27_151136
    2017-01-27_151306
    2017-01-27_151349

Console App with Tests

  1. Create a Console application
     2017-01-30_104608
  2. Add two Class Library projects to the solution. We'll show consuming different test frameworks.
    CIConsoleSample.MSTests
    CIConsoleSample.xUnitTests
     2017-01-30_105018
  3. In the MSTests class library, add an Extensions reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework
    2017-01-30_105737
    Also add a reference to the CIConsoleSample project.
     2017-01-30_111716
  4. In the xUnitTests class library, add the xUnit NuGet package, and the xunit.runner.visualstudio package.
    2017-01-30_113117
    And, as above, add the reference to the CIConsoleSample project.

Create MSUnit Tests

Rename Class1.cs to CIConsoleSampleMSTests, add the needed using statement and the following tests code. These tests use nested classes and a Class.Method_Should.Function naming style.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
//Added for tests
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace CIConsoleSample.MSTests
{
    [TestClass]
    public class CIConsoleSampleMSTests
    {
        [TestClass]
        public class TextMan_Greeter_Should {
            [TestMethod]
            public void AppendTextToTheGreeting()
            {
                //arrange
                string value = "Charles";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.AreEqual(expectedResult, actualResult);
            }
        }
    }
}

Create xUnit Tests

Reference: https://xunit.github.io/docs/getting-started-desktop.html
Add a class file named CIConsoleSamplexUnitTests.cs, then the following code. As above, these tests use nested classes and a Class.Method_Should.Function naming style.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
//added for tests
using Xunit;

namespace CIConsoleSample.xUnitTests
{
    public class CIConsoleSamplexUnitTests
    {
        public class TextMan_Greeter_Should
        {
            [Fact]
            public void AppendTextToTheGreeting()
            {
                //arrange
                string value = "Charles";
                string expectedResult = "Hello, Charles";
                TextMan textMan = new TextMan();
                //act
                string actualResult = textMan.Greeter(value);
                //assert
                Assert.Equal(expectedResult, actualResult);
            }
        }
    }
}

Run Failing Tests

First, here's how I like to display my tests, taking advantage of the nested class structure.

  • Tear off the Test Explorer into its own window, so that you can easily resize and see all the tests.
    test-tearoff
  • Group the test results by class
    test-byclass
  • In your xUnitTests project, add an app.config file with the following setting. This causes the result to display only the method name instead of the fully qualified name.
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="xunit.methodDisplay" value="method"/>
      </appSettings>
    </configuration>
  1. Build the solution so the tests are displayed in the runner.
     2017-01-30_120234
  2. Run the tests. They should both fail.
     2017-01-30_120353

This is enough for the moment. Let's add this solution to TFS, then to our CI server.

Add Solution to TFS

References
Create a team project Choose a process
Open the Team Explorer. If this is the first time connecting to TFS, you may see something like this (including the commercial).
2017-01-30_145139
Close the commercial, if you want, drop down Manage Connections and choose Connect to Team Project.
2017-01-30_145415
Select your local server. It will search for collections. Click Connect.
2017-01-30_145611
For this walk through, we're using TFS workspaces, not Git, for version control.
You can click "Configure your workspace" in the big mustard prompt. Or, click Project > Configure Workspace
2017-01-30_145755
The dialog maps the remote collection's root folder to a folder on your local computer. You can accept the default (which I'll do for this example), or choose another folder (which I'd normally do). Click "Map and Get". This is our first solution, so there are none to get.
2017-01-30_150019
Switch to Solution Explorer, right-click the solution, and choose Add Solution to Source Control.
2017-01-30_150759
I got this error message.

Full Disclosure I don't like TFS. This is an example of why. So much ceremony! I know there's a good reason, but sometimes I just want to get to work.

2017-01-30_150946
Switch back to Team Explorer, and choose Home > Projects and My Teams > New Team Project.
2017-01-30_153823
Name your team project. Click Next.
2017-01-30_153932
Choose the Agile process. Click Next.
2017-01-30_154431
Choose Team Foundation Version Control. Click Next, then Finish.
2017-01-30_154535
2017-01-30_154613
Now switch to Solution Explorer and add the solution to the team project.
2017-01-30_154745
The solution files are added locally, but not yet committed (checked in) to the repository. Right-click the solution and choose Check In.
2017-01-30_155043
Enter a comment and click Check In.
2017-01-30_155135
2017-01-30_155242
Next Part: TFS Continuous Integration Walk Through Part 2 - Create an Automated Build

What Else Programmers Do: A Text Manipulation Example

Sometimes, in my "regular" (i.e. non-IT) life, I apply my job skills. Today, for example, I came across a Facebook post about how to contact and influence Congress members. The post suggested cutting and pasting to share with others. Unfortunately, it wasn't possible to select the text. Some sites do this to prevent copying their content. News sites, especially, for copyright reasons.

What follows is not a tutorial for regular (i.e. non-technical-programming-nerdy) people. It's an example of the kind of process I do in my job every day, and how I think about a problem. It requires tools, knowledge and experience.

Getting to the "Source" of the Matter

Now, the first problem is that I viewed the original FB post on my phone, and it turns out that's why I couldn't copy/paste the text. If I'd bothered to find it on my desktop, my problem would have been solved. Instead, I emailed myself a link to the page. That link still went to the mobile version. That's what the "m." in the URL is telling me.

2017-01-26 09_35_13

Here's some of the text:

2017-01-26 09_38_16

I can't copy the text, it's sort of like it's an image, but I'm pretty sure it's not an image. My next step is to view the page source HTML. I right-click in Chrome and choose "View page source".

2017-01-26 09_40_46

Which yields a huge mess of coded text, as befitting a complex web site. There are about three thousand more lines like this.

2017-01-26 09_42_55

I'm pretty sure buried in there is the text I want. And I'm also pretty sure I'll need to manipulate it. So, I open Notepad++, a sophisticated, free, open source text editor. Then I copy/paste the source from the browser into it.

2017-01-26 09_46_56

Now I'm ready to use my text editor.

Text Editing is Not the Same as Word Processing

In word processing, like you do with Microsoft Word, the concern is text formatting: making things bold, italic, different fonts, etc. The result of all that work is marked up text (or binary, but we'll stick with text) that tells the application how to display all of your words on the screen. Here's an example from a Word file that was saved in its standard .docx format. The document literally has one word: "hello". But Word needs all of this information to display that those five characters.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:document 
xmlns:wpc="http://schemas.microsoft.com/office/word/2010/wordprocessingCanvas" 
xmlns:cx="http://schemas.microsoft.com/office/drawing/2014/chartex" 
xmlns:cx1="http://schemas.microsoft.com/office/drawing/2015/9/8/chartex" 
xmlns:cx2="http://schemas.microsoft.com/office/drawing/2015/10/21/chartex" 
xmlns:cx3="http://schemas.microsoft.com/office/drawing/2016/5/9/chartex" 
xmlns:cx4="http://schemas.microsoft.com/office/drawing/2016/5/10/chartex" 
xmlns:cx5="http://schemas.microsoft.com/office/drawing/2016/5/11/chartex" 
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" 
xmlns:o="urn:schemas-microsoft-com:office:office" 
xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" 
xmlns:m="http://schemas.openxmlformats.org/officeDocument/2006/math" 
xmlns:v="urn:schemas-microsoft-com:vml" 
xmlns:wp14="http://schemas.microsoft.com/office/word/2010/wordprocessingDrawing" 
xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" 
xmlns:w10="urn:schemas-microsoft-com:office:word" 
xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" 
xmlns:w14="http://schemas.microsoft.com/office/word/2010/wordml" 
xmlns:w15="http://schemas.microsoft.com/office/word/2012/wordml" 
xmlns:w16se="http://schemas.microsoft.com/office/word/2015/wordml/symex" 
xmlns:wpg="http://schemas.microsoft.com/office/word/2010/wordprocessingGroup" 
xmlns:wpi="http://schemas.microsoft.com/office/word/2010/wordprocessingInk" 
xmlns:wne="http://schemas.microsoft.com/office/word/2006/wordml" 
xmlns:wps="http://schemas.microsoft.com/office/word/2010/wordprocessingShape" mc:Ignorable="w14 w15 w16se wp14">
  <w:body>
    <w:p w:rsidR="000E45DD" w:rsidRDefault="00FA6736">
      <w:r>
        <w:t>hello</w:t>
      </w:r>
      <w:bookmarkStart w:id="0" w:name="_GoBack"/>
      <w:bookmarkEnd w:id="0"/>
    </w:p>
    <w:sectPr w:rsidR="000E45DD">
      <w:pgSz w:w="12240" w:h="15840"/>
      <w:pgMar w:top="1440" w:right="1440" w:bottom="1440" w:left="1440" w:header="720" w:footer="720" w:gutter="0"/>
      <w:cols w:space="720"/>
      <w:docGrid w:linePitch="360"/>
    </w:sectPr>
  </w:body>
</w:document>

The concern of text editing is working with raw content like that above. A good text editor lets you perform complex Find & Replace, or do block editing (where you can select text not just in rows, but also in a columns).

Back to our FB post source. I know that somewhere in the text is the bit that I want. So, I search for some text that's very likely unique to the content I'm looking for. "Senator" is a good choice.

2017-01-26 10_00_34

TECH DIGRESSION!

I found, at this point, why I couldn't copy/paste the text in the first place. All of the content was inside a <script> tag. It was probably being processed by javascript, and displayed in some messed-up way.

There's other markup in there, but I'll take care of that in a little bit. First, I copy what looks like the text want into a new Notepad++ tab.

2017-01-26 10_06_45

I Need a Replacement

Now, I look at what's there, while thinking about what I want. Ideally, my end result is nicely formatted text, meaning it's got all the original line breaks. I could do that manually, but I'd prefer not to.

The first thing I notice is the markup after "Senator."

From a high-level staffer for a Senator:\u003C\/p>\u003Cp> There are two things 

That sequence, \u003C\/p>\u003Cp> appears throughout the text exactly where line breaks are in the original.

2017-01-26 10_13_55

So let's see what happens if I just replace all of those instances with a new line and carriage return.

TECH DIGRESSION!

Huh? How (you wonder) do I do that? How do I insert something that can't be seen? Well, even though you can't see them, line breaks (where you go down one line) and carriage returns (where you go back to the beginning of the line) are stored in text just like any other character. They just aren't displayed in the editor. But there are ways to represent them, and one long-standing way is with C programming-style expressions. I can use text to represent other, hidden text.

 

\n = newline
\r = carriage return
\t = tab stop

 

The back slash is an "escape" character. It tells the application "don't treat this sequence \n literally. Instead, look in the raw text for the hexadecimal byte sequence 0D." As an example, here are two lines of text:

life's
good

and its hex representation. I put the represented letters and linefeed/carriage return above the hex for clarity.

l  i  f  e  '  s  \n \r g  o  o  d
6c 69 66 65 27 73 0d 0a 67 6f 6f 64  

So, I select \u003C\/p>\u003Cp> and try to replace it with \r\n\r\n (two carriage return + new line). And it fails.

2017-01-26 10_32_08

Why? For the same reason the newline can be found: the escape character \. Notepad++ is looking at my "Find what" string and saying, "Oh, you've got a back slash in front of that u. That must mean something special to me, so I won't treat it as '\u'. I can't find your text, sorry."

What do we do if we want the back slash to be treated literally? Heh, heh. You escape it by putting another back slash in front of it. To find a literal occurrence of \r in text using C-style expressions, you search for \\r. Crazy logical, right?

My search express becomes this: \\u003C\\/p>\\u003Cp>

I try my Find & Replace again, and this time it works.

2017-01-26 10_39_04

I could live with this result if I weren't a perfectionist. But there's still lots of HTML markup in there (highlighted in yellow). So, I do my Find & Replace magic a bunch more times. By looking at the original text, I can figure out what characters the markup stood for (or I could look it up). There's also some non-standard markup.

&#039;  =   '
&quot;  =   "
&amp;   =   &
&gt;    =   >
\u2013  =   --
\/      =   /

The result is the text as I originally saw it on the post. Here's a partial example.

From a high-level staffer for a Senator:

 There are two things that all Democrats should be doing all the time right now, and they're by far the 
 most important things. [***If you want to share this, please copy and paste so it goes beyond our mutual
 friends***]

 --> You should NOT be bothering with online petitions or emailing.

 1. The best thing you can do to be heard and get your congressperson to pay attention is to have 
face-to-face time - if they have townhalls, go to them. Go to their local offices. If you're in DC, try
 to find a way to go to an event of theirs. Go to the "mobile offices" that their staff hold periodically
 (all these times are located on each congressperson's website). 

 When you go, ask questions. A lot of them. And push for answers. The louder and more vocal and present
 you can be at those the better.

 2. But, those in-person events don't happen every day. So, the absolute most important thing that people
 should be doing every day is calling. 

 You should make 6 calls a day: 2 each (DC office and your local office) to your 2 Senators & your 1 
Representative.

Wrap Up

It took a lot to extract the text from that mobile web page.

  1. Use the browser's View Source feature
  2. Use a specialized text editor
  3. Find the content I was looking for
  4. Recreate the line endings via Find & Replace using C programming language expressions
  5. Replace the remaining HTML markup

If you're wondering what kind of things that special programmer in your life is doing every day, this is it. It's not glamorous. In fact, it's pretty tedious. But, it's a living!