One last thought first
This article uses the .NET framework and development for its examples. However, the principles described should apply to most languages and frameworks. For example, the issues and approach can be taken with Angular applications.
Contents
Credits
I wrote most of this, so I get a bunch of the credit. But it wouldn't be as good if my colleagues at Clear Measure hadn't reviewed my words and provided thoughtful questions and debate. They especially clarified to me that the message of the document should be positive toward packaging, that packaging is a critical and established technique for creating performant, maintainable, and scalable business applications.
So, thank you Trish Polay and Mike Sigsworth. You rock!
I don't consider this the last word on this subject. My views will evolve, and I want this article to evolve with them. Stay tuned!
Packages Are Good!
This document examines when it makes sense to publish shared dependencies as packages vs maintaining project references. A strong case is made that shared code doesn't have to become packages. However, this shouldn't be taken as affirming shared project references as desirable.
The question is less whether you'll be packaging reusable bits of your code base, and more when. If you're developing a large enterprise application, breaking it into smaller, independently developed, tested, and deployed services has important benefits. At minimum, this means implementing something like an Onion Architecture informed by Domain-Driven Design. For high-scalability, microservices can become a strong choice.
There's little doubt that developing independent--yet related--projects is more difficult than a monolith where all the code is available all at once. But difficulty isn't the only factor to consider. A loosely-coupled architecture that's easier to test, faster to deploy, and more resilient to change, pays for itself many times over, even if daily development takes more time or effort.
Beware coding by expedience. Technical debt is sure to quickly accrue.
The guidance below is when it makes sense to use packages given how your code is today. Maybe you're working in a monolith and packaging isn't a good near-term decision. But don't make that your long-term decision. Really.
Here are some examples of how you can think about it.
We're starting a new code base with shared code. What do we do?
Try putting some shared code in its own repo. Figure out how that works.
Maybe our new code base has code that might be shared, but isn't yet.
That's OK Likewise try putting the shared code in its own repo.
We have shared projects. But when we pull them out, developing the independently becomes a headache. What do we do?
Reexamine the code. Instead of assuming your projects as they are should become packages, try extracting code from those projects into good packages.
As you read below, you may be tempted to stick with your monolith and eschew packages. That's not what I want. I want you to build an environment where you can develop high quality, less complex, loosely-coupled applications quickly. You may not be there today, but that's where you're headed.
In all of the above, you won't get it right the first time. Your code will evolve along with your understanding. You'll need help and find out how other shops do it. Expect that.
The Developer's Essential Desire
Work on all parts of the application at the same time
The Practical Difference Between External and Internal Dependencies
- External dependency packages are intended for generic use cases
- Internal dependency projects are for specific applications
Both are dependencies, but who uses them are different. An external package like Json.Net doesn't know who's using it. An internal dependency like CustomerContractService is developed for specific applications.
The Fundamental Conflicts
- Loose coupling says projects can be developed, tested, and deployed independently.
- But related projects benefit from developing, testing, and deploying together.
External packages don't have specific parent dependencies, so they can be developed independently and reside in their own repository and solution workspace. Internal applications have shared dependencies that need to be developed simultaneously.
These two development models are at odds.
For external packages,
- The parent references the package dependency, which is downloaded from a store
- The package dependency's code isn't changed
- The package version is changed externally, and the parent chooses when to upgrade
For internal dependencies,
- The parent references the project dependency, which is in a relative path
- Changes to the parent and dependency are committed to source control together
- The package version is changed at the same time for the package and parent
The two critical concerns are:
- Internal dependencies have a fixed relative path stored in source control
- A parent expects the dependency to exist when submitted to continuous integration
Challenging Assumptions
There are several reasons a development shop may decide to separate dependencies into packages.
- Clarify the code base by explicitly decoupling the dependency
- Allow greater or easier consumption of the dependency
- Easily share reusable code between projects that live in separate repositories and are worked on by independent teams
- Allow independent projects to choose when to upgrade to new version of shared code
- A belief creating packages is how it "should" be done for separation of concerns, loose coupling, domain-driven design, microserves, etc.
Several of these reasons can be challenged, especially the last. Here are some questions to ask yourself and team.
- Imagine the dependency is a third arty open source project. How would that affect our application development?
- If we don't package our dependencies, is our coding easier or harder? Which parts?
- Does an application have to change the dependency code during feature development? If so, why? If not, how long can the application wait for the package to have the new changes?
- Is committing a relative path a hinderance? Is it real or reactive? Can our group establish a common structure for shared project dependencies?
Seriously, if you don't need packages, don't create them. You can still have loosely-coupled shared code.
A Simple Heuristic
Not all shared dependencies should be packages. That's a critical message to take away from this article. It's easy to think our development lives would be easier by packaging all our shared dependencies, but that isn't true.
If the dependency is usually developed in lockstep with the parent, don't make it a package.
For examples, these probably aren't good choices for packages
- A CustomerService assembly
- An Entity Framework repository assembly
Candidates for internal packages have these qualities
- Don't change major functionality frequently
- Are highly backward-compatible
- Who uses them doesn't have to be known
Here are some examples. Typically these are customized for the business and are application-agnostic.
- Security libraries. Great choice.
- Type manipulation libraries
- Networking libraries
- Libraries that work across domains
What do I mean by that last one? Let's say you're practicing Domain-Driven Design. You've segregated your domains and created separate repositories per domain. However, you find there are some domain behaviors that apply to all domains and need to be consistent. Those may be a candidate for a package.
Building Blocks
These are the building blocks we're dealing with.
- Source control
- Project vs package references
- Relative paths for project references
- Automated build dependencies
- Dependency versioning
A Common Folder Structure
Since we're assuming software development inside an organization, we can take advantage of that by sharing some conventions. For all the solutions below I recommend the same folder layout. It's not the only way, but is consistent and allows a mono-style to be broken up more easily later.
Use this general structure
|_repos
|_Repo
|_docs
|_src
|_Project1
|_Project1.csproj
|_Project2
|_Project2.csproj
|_Solution1.sln
|_build.ps1
|_pipeline.yml
A repo with shared project references doesn't need (but could have) a solution file. It's name is prepended with an underscore so that shared projects are at the top of the development folder.
|_repos
|__SharedRepo
|_docs
|src
|_Project1
|_Project1
|_Project2
The key to this or other layouts is the predictability of relative paths.
Repositories
As long as we stick to the layout, our project references will resolve correctly.
Here's a example of four Git repositories. Notice how, as long as the convention is followed, the repos don't conflict with each other.
repos
|__SharedRepo1
|_.git
|_docs
|_src
|__SharedRepo2
|_.git
|_docs
|_src
|_AppRepo1
|_.git
|_docs
|_src
|_Project1
-> ProjectReference ../../SharedRepo1/src/Project1/Project1.csproj
-> PackageReference PackageProject1.2.1.5
|_build.ps1
|_pipeline.ps1
|_AppRepo2
|_.git
|_docs
|_src
|_Project1
-> ProjectReference ../../SharedRepo1/src/Project1/Project1.csproj
-> ProjectReference ../../SharedRepo2/src/Project3/Project3.csproj
-> ProjectReference ../../PackageRepo1/src/PackageProject1/PackageProject1.csproj
|_build.ps1
|_pipeline.ps1
|_PackageRepo1
|_.git
|_docs
|_src
|_PackageProject1
|_PackageProject2
|_Packages.sln
|_build.ps1
|_pipeline.ps1
AppRepo2 Project 1 represents what I'd prefer not to do: reference a NuGet package project. This is discussed later.
For convenience, developers may want to have their own solution files that combine other solutions. However, most of these should not be stored in source.
If you put them in the root of the development folder (e.g. repos), there's no source control issue. But it may make more sense for them to be in a repository's folder, yet still not version controlled. This is accomplished by adding a line in .gitignore
# Ignore root-level solution files
*.sln
|_repos
|_Repo
|_.git
|_src
|_Solution1
|_Solution2
|_.gitignore
|_One-Solution.sln
|_All-Solutions.sln
|_CommonApps.sln
If there were a few solution files that were convenient to maintain, then a naming convention could be used instead.
# Ignore root-level solution files with this naming
*[Ss]olution.sln
In this case, One-Solution.sln and All-Solutions.sln would be ignored but CommonApps.sln would be tracked by Git.
Simultaneous Development With Packages, If You Must
If you have to develop a package along with its parent, you need to handle a few things.
- (Temporarily or permanently) set a project reference to the package, OR
- Publish the package locally for a code-publish-install development loop (don't do this, it's dumb!)
- Local build, and continuous integration, must either
- Remove the project reference, then deploy the package before building the parent, OR
- Maintain the project reference so the parent builds with the new dependency, and simultaneously build/deploy the package
.NET Core project files can have both a PackageReference and ProjectReference entry for the same assembly, but it's not good practice and will lead to headaches.
You should not be bouncing back and forth between project references and package references. Solve the problem that's leading to this anti-pattern
Let's assume you have an application that depends on a packaged project, but you're always updating that dependency in lockstep. The real problem is build and deployment.
- The parent project has a project reference to the dependency
- The build script builds the parent app using the project reference, but generates a package for the dependency
- The deployment server deploys the new package independently of the parent
Here's some PowerShell pseudocode. Your actual implementation will be different!
# MyApp build.ps1
# Run the dependency solution's build script, assuming it's independent
# This should prompt for a new version number, see explanation below
cd ..\..\_SharedRepo\src\CompanySecurityHelpers
\build.ps1
# Build and publish the solution locally, which will build the dependency dll
# with the new version number
cd ..\..\MyRepo\src
dotnet build -o Release MyApp.sln
dotnet publish -o Release --no-build MyApp.sln --output package
# if the dependency's script doesn't create the package, package it here
dotnet pack -o Release --no-build ..\..\_SharedRepo\src\CompanySecurityHelpers --output package
The Versioning Puzzle - In General
- Versioning is hard to fully automate. I think it's a mistake to try.
- Fully automating Patch versioning is probably OK, though.
- Reduce the friction to allowing developers to commit a new version number.
The idea is simple. Most of the time, you only need to increment the version number when you're ready to generate a pull request. At that time, you should be rebasing your changes on top of the latest mainline release, so you should have access to the latest package version.
- The local build script updates the package version
- The automated build/deployment pipeline fails if the package version is lower than what's in the store
Here's pseudo code for asking the developer if the version should be increased. It's the idea, not production code. This all works best when developers are committing and synchronizing code frequently.
$currentVersion = GetCurrentVersion
Write-Message "Current version is $currentVersion"
$reply = Read-Host -Prompt 'Do you want to increment the version to publish the package? (y/N)'
$bumpVersion = $reply -eq 'y'
if ($bumpVersion) {
$level = Read-Host -Prompt 'Which level? (major, minor, patch, [pre])'
# IncrementVersion takes care of updating the .csproj properties
$newVersion = IncrementVersion($currentVersion)
Write-Message "New version will be $newVersion"
}
A PowerShell script could also read the package store, find the latest package, read its version number, and compare to the package being built. This could run locally to catch a problem before the PR, and would definitely run in the CI/CD server.
The Versioning Puzzle - NuGet Packages In Particular
NuGet packages have a history of dependency resolution challenges. It looks like .NET Core has fixed most issues. To be sure, dependency resolution is hard because a top-level .NET project can only have one version of a dll when it builds. So, what should happen in the following environment?
- MyApp installs and uses Json.Net 9.1.0. It depends on features in that version.
- MyApp installs CoolAutoConfigurator, which itself depends on Json.Net 8.3.7.
MyApp
|_Json.Net 9.1.0
|_CoolAutoConfigurator
|_Json.Net 8.3.7
Which version of Json.Net should MyApp include in the bin folder? The latest? What if there were changes that break CoolAutoConfigurator?
Generally, in Core, the latest version will be used. It's up to the developer to deal with a breaking change by coding around it or contacting the package developer to request they upgrade.
In .NET Framework, including a package with a specific assembly version would often cause conflicts that were resolved using special files that would, in effect, say "If the assembly version is this, it's OK to use this other assembly version."
Package maintainers found the better way to manage Framework packages was to use specific package versions, but change assembly versions only on major releases. So, Json.Net package 9.2.3 and 9.1.0 would both generate a dll with assembly version 9.0.0. The job of the package developer was to ensure no breaking changes in major versions.
This matters. .NET Framework and Core resolve which NuGet package to install based on package version, but assemlies (exes and dlls) resolve which other assemblies to use by assembly version, and assemblies with different versions are not equal. If myapp.exe and mylibrary1.dll both use mylibrary2.dll, but expect different assembly versions, the program will fail.
Core improves on Framework by being more lenient in accepting assembly versions.
A further complication is which version numbers appear in the NuGet package store versus which are embedded in the compiled assembly (DLL). Here's a handy map for all you people who want to right click myfile.dll > Properties > Details.
I'm focusing on .NET Core here.
| .NET Core .csproj property |
.csproj value |
NuGet |
File Property |
| Package version |
1.0.2-alpha1 |
1.0.2-alpha1 |
Product version |
| Assembly version |
1.0.0.0 |
- |
- |
| Assembly file version |
1.0.0.123 |
- |
File version |
Weird, isn't it? The assembly version isn't visible in file properties, but it's critical for our applications and for the Global Assembly Cache (GAC).
Wrap Up
In general, organizations need to avoid making the assumption that all their shared project references should become packages. Instead, they should extract into packages libraries that can be developed independently, and maintain project references to the other shared code.
Importantly, they should then reduce the shared project references by ruthlessly reevaulating and refactoring the code.
Having a convention for code folder layouts makes it much easier to separate code into independent repositories.
Versioning is more difficult when it's highly automated. Try keeping this in the developer's hands, and create safeguards to accidentally deploying the wrong thing.
Finally, packages do not equal loose coupling, though they can aid in a decoupling effort. Loose coupling has more to do with architecture, modeling, and development practices. An Onion Architecture, Domain-Drive Design, and Test-Driven Devlopment will go a long way to lowering your code's coupling and complexity.
