A pleasant walk through computing

Comment for me? Send an email. I might even update the post!

TFS Continuous Integration - Agent Installation and Visual Studio Licensing

The Summary

A build agent is what takes care of actually running a build definition. Agents can be installed on machines separate from the TFS server, allowing workload distribution.

The simple way to understand an agent is to imagine how you'd create continuous integration yourself.

  1. You'd have a machine that could build the software. That means you'd have to install anything needed to accomplish the build, such as Visual Studio, 3rd party controls, certificates, tools, etc.

  2. You'd write a script that could automate the build and report errors.

  3. You'd create a way of running that script on demand, such as developing a Windows service.

That's all TFS Build is doing. You configure the steps (the build script) on the TFS server as a build definition. You install an agent on a machine that can check out the source code and successfully build the application. TFS calls the agent on demand.

It was unclear to me if I needed a licensed version of Visual Studio, or VS installed at all. The answers are:

  • If you're not using the Visual Studio build step, and only the MS Build step, you might be able to get away with installing the 2015 Build Tools.
  • However, you'll probably need Visual Studio installed. It does not need to be licensed, assuming it's not also being used for development.

VS 2015 Licensing White Paper

Using Visual Studio on the Build Server: If you have one or more licensed users of Visual Studio Enterprise with MSDN, Visual Studio Professional with MSDN, or any Visual Studio cloud subscription then you may also install the Visual Studio software as part of Team Foundation Server 2017 Build Services. This way, you do not need to purchase a Visual Studio license to cover the running of Visual Studio on the build server for each person whose actions initiate a build.

References

Installing an agent is pretty simple. Really, just read Ben Day's post and you'll find out what you need. It's slightly outdated, but close enough. I've also listed the steps, below.

https://www.benday.com/2016/01/01/walkthrough-create-a-new-tfs2015-build-server/

The Installation

  1. Install everything needed to build the software. It's best to do this first.
  2. Download the agent from the TFS web. Manage Server (click the right corner gear) > click link "View collection administration page" > open Agent Queues tab > click "Download agent"
  3. Extract the zip into C:\TfsData\Agents[agent name]
  4. Run ConfigureAgent.cmd
  5. Mostly accept the defaults. The TFS server URL will be something like http://servername:8080/tfs. Answer Y to installing as a service.

After installation, you should see the agent in the Agent Queues.

Maintenance

Agent Versions

If you're using a local TFS installation, the agent version is tied to the TFS version. If you update TFS, be sure to update the agents. It's easy. In Agent Queues, right-click the queue and choose Update All Agents.

Adding Agent Capabilities

Normally, all you need to do is install the software with the capability, then restart the agent. However, here are a couple of articles related to capabilities.

How to Register Capabilities
Demands

The Wrap Up

Agents are just services that run build steps. An agent can be installed on almost any machine, letting you easily configure your build environment.

TFS Continuous Integration - ClickOnce Apps

The Summary

Oh, ClickOnce, you bane of development! You're always so attractive: easily created, self-updating installations. But, like a 21st century TV vampire, you end up sucking the life out of me when things get complicated.

In the case of continuous integration, we need to sign our application using a security certificate, to guarantee the publisher's identity. This makes sense, since the intent is that ClickOnce is installed and maintained from a web site.

http://stackoverflow.com/questions/8955332/what-is-signing-clickonce-manifests-for

So, there are two parts to manage in CI: the certificate and the signing.

There are several combinations for trying to build ClickOnce. Is your TFS on site, or are you using Visual Studio Online? Are you signing using a commercial, local-domain, or temporary certificate?

This document is for a specific circumstance:

  • Local TFS 2015
  • A temporary certificate

Note
This post will not deal with publishing a ClickOnce application via CI.

The Problems

So what happens when you try to build a ClickOnce app and on a separate CI server (without Visual Studio installed)?

It fails, that's what. At minimum, in the above scenario of using the default temporary certificate (which you shouldn't), it will fail because the signing utility, signtool.exe, isn't installed on the server.

How do you manage the signing process on a locally hosted machine?

Locally Installed TFS

Install SignTool.exe on the Server

When a developer creates a ClickOnce app, she must have the ClickOnce tools installed. In Visual Studio 2015, this is a feature checkbox during installation. If your forgot, you can open Programs and Features, right-click Visual Studio, and choose Change to rerun the installation.

https://social.msdn.microsoft.com/Forums/en-US/2d9414f5-1e78-4d33-b651-d09be74db80d/clickonce-publish-in-visual-studio-2015?forum=winformssetup

But we're not going to install Visual Studio on the server.

Just the Files?

Can we create the appropriate folder path on the server and just copy the needed files, instead of installing 1GB of utilities? Maybe. It looks like the required files are:

signtool.exe
mssign32.dll
wintrust.dll

And the paths are:

C:\Program Files (x86)\Windows Kits\8.1\bin\x86
C:\Program Files (x86)\Windows Kits\10\bin\x86

I used the first path on Windows 10 and it worked fine. This is definitely worth testing before installing the SDK.

Via the Windows SDK

We need to install the super-bloated SDK. Which one?

If you're on Windows 8.1/10/2012/2012R2, install the Windows 10 SDK.

https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk

If you're on Windows 7/8/2008R2, install the Windows 8 SDK.

https://developer.microsoft.com/en-us/windows/downloads/windows-8-1-sdk

During installation, choose the Windows Software Development Kit. Funny, in Windows 10 they moved the SDK to the bottom!

2017-03-03_151212

2017-03-03_151303

Note
On my Windows 10 machine, running Visual Studio 2015, when I installed the ClickOnce tools, signtool.exe was installed to the Windows 8.1 SDK folder instead of Windows 10. It works...but go figure.

The Wrap Up

For my specific case, it was relatively easy to get signing to work during the build. If a non-temporary certificate had been involved, I could have installed that to the server.

This doesn't answer what to do if using Visual Studio Online or some other continuous integration server. That will be an adventure for another day.

TFS Continuous Integration and Private NuGet Package Sources

The Summary

What if you

  1. Don't store your NuGet packages in source control.
  2. Have a NuGet package that's hosted in a private source.
  3. Need to locally test continuous integration.
  4. Or need offsite (i.e. cloud) continuous integration.

There are a few general solutions.

  1. Store the private packages in source control. I'd do this one.
  2. Set up public access to the private source. Not likely.
  3. For testing, set up a local private NuGet source (just point to the folder).

The Problems

NuGet packages are generally great. But there can be problems when it comes to source control and continuous integration.

Most sites' advice on storing NuGet packages in source control (regardless of git, Mercurial, TFS, SVN, etc) is: don't.

The reason is pretty simple. If package restore is enabled, they'll get downloaded and rebuilt anyway, so why store them and take up repository space? With the later NuGet installations, a cache is maintained in the user's profile, so a trip to the NuGet servers might not even be necessary.

The counter argument is also pretty simple. What if the NuGet source isn't available? Suddenly, you can't restore your packages, can't build, can't work. When would this happen?

All of these assume you've downloaded from source control (such as GitHub or TFS), but haven't built the code yet.

  1. You get on a plane and then try to build. If you have no Internet, you can't get the packages.
  2. For some other reason you don't have an Internet connection.
  3. You're connected, but the NuGet site is down.
  4. One of the packages uses a locally (i.e. corporate internal) hosted NuGet source. It's not on the internet, so you can't download the package.
  5. It's been a long time since anyone's built the source, and a package has been removed from NuGet.
  6. You use the cloud for continuous integration, which won't have access to your private NuGet source.

Most of the above can be solved by building the solution immediately after getting it. But here's a real-world example of number 4. I was working for a client, and had access to their Team Foundation Server. I got the source via a VPN connection, copied it to my laptop, and tried to build. This client has several NuGet packages they host locally. They're proprietary, so hosting them on a public site like NuGet would be wrong.

And I didn't have access to that package source, so I couldn't build.

Again, these problems could be solved, and maybe they point to some environment changes needing to be made. But wouldn't it be just as easy to include the package in source control?

Including the NuGet Package Folder in TFS

If you're using a version of TFS prior to 2012, I can't help you (and you should upgrade). Starting in TFS 2012, the tfignore file became available. The purpose of tfignore is just like gitignore and hgignore: tell source control which files to not display for adding/tracking.

But it can also be used to explicitly allow files. Why is this needed? Because, unlike git or Mercurial, TFS + Visual Studio ignores certain files by default, and I haven't found a way to change that or even find out what files those are. Dlls are ignored by default, for example.

  1. Create the .tfignore file in the root of the team project.
  2. Edit the file.
  3. Add the package fies to TFS.
  4. Add your .tfignore file to source control.

The manual way to create a .tfignore file is, in the project folder root

  1. Right-click > New > Text File
  2. Enter .tfignore.

See the trailing period? That's the magic sauce. Otherwise, Windows doesn't let you create a file/folder with a leading period.

Visual Studio shows package folders in Excluded Changes by default. But dlls, and sometimes lib folders, are not included, and we absolutely need those.

Here's how to include the entire packages folder. The leading ! means "don't ignore".

!packages\*

These don't work. Note, especially, that leading backslash doesn't work, even thought that's what VS itself will create via the GUI.

  • !\packages*.*
  • !packages
  • !packages\

To include a specific package, such as one that's hosted in a private NuGet source, ignore the packages folder, then specify the package path without the version number.

packages
!packages\My.Private.Utility.*

In Visual Studio Team Explorer, choose Pending Changes, find Excluded Changes, click Detected: x adds(s).

2017-03-03_101910

Deselect all files (they're selected by default, a bad choice), then check the ones you want and Promote them (which is the same as "add" in git or Mercurial).

2017-03-03_103218

Using a Private Repo for Testing

If you're testing CI on a separate machine, you can use or modify a nuget.config file to include that source. I tried this at the project level, but it didn't work, so I created the nuget.config file at the solution level. (But that ability is supposedly removed in later NuGet versions. So confusing!)

Here's an example file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageRestore>
    <add key="enabled" value="True" />
    <add key="automatic" value="True" />
  </packageRestore>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  <add key="local" value="C:\Users\charles\Documents\Testing\NuGet\" />
  </packageSources>
</configuration>

Again, this must not end up in the production source control repository!

The Wrap Up

I'm not weighing in on storing NuGet packages in source control, except to say that there are some situations where it's clear to me it's a good ideal. Having a private NuGet source involved is one.