SemVer, Team Development and Visual Studio Solutions
Controlling dependency chains between libraries and services in complex software projects isn’t easy, though you can, to a certain point, get away with very simple solutions. I guess most teams end up applying some sort of versioning strategy to the software they develop, even if it’s saying that it’s a 1.0 at deployment time and stick with that. Like I said, you can get away with a lot, depending on your context. A lot of the projects I have worked in my career where single client and single installation applications. That pretty much meant the version of the software installed was whatever was built based on some source control revision. Match the deployed package to whatever revision it was created with and you had your version. This generally meant a single version across components that integrated the software.
As simple as this method is, it works and takes a load off in terms of the amount of things a small team has to focus on to release software. Still, not all projects can live with such a simple solution. I’d say that you can live with this solution as long as you can contain the whole installation as a single package or build result. Once you start mixing and matching and thinking about plugable architectures, or have large package dependency graphs with more than a couple nodes, things can get complicated…. and break easy due to incompatibility.
Being this far into the article, a disclaimer may come as something somewhat strange. I do want to warn you, though, that as I write this, I don’t have a specific solution to this problem. This is something I’m looking into at the moment and there are still a set of doubts in my mind as to how to proceed (to resolve my own problem). I do want to present some of my thoughts, though, in case others are coming up on the same questions I’m having and maybe allow for a discussion upon this matter. Comments and suggestions are very welcome.
When can you get away with a very simple solutions to versioning libraries? Very simple meaning “version numbers just don’t matter to me”:
Versioning software and libraries isn’t a problem everyone has. Sometimes you just don’t need to loose your sleep over it.
(Un)fortunately, not every software project can stay that simple. More than one person starts working on the code base. You get a new client with special needs. Clients ask for new features. Clients ask to change existing features. Monoliths are split up into services. Libraries get shared between projects with different development and release life cycles. Code goes public and into the while and some people expect it to stay the same and stable.
You now have a problem that needs to be solved.
Just as we want software to be scalable, distributed and concurrency capable, our development process needs to support similar characteristics. We need to be able to scale or even distribute the dev team. Clients need new features at different rates, and we need to be able to support those needs. The code base isn’t the only piece of the puzzle we will need to look at. The process with which we build our software needs to be reviewed to handle these, especially for source control, artifact repositories, and continuous integration and deployment workflows.
I feel it would be simpler to review many of these aspects with an example project reference to look at, with a somewhat complex dependency chain. So, let us say that you have some web app A that your developing. this web app depends on a couple of external services that it requests data from or delegates processing activity to, and we’ll call them services A and B. Services A and B have their own teams and independent life cycles, but they evolve along as C gets developed (they are all part of the same system, and developed by the same company). Also, services A and B share a set of common utility libraries and framework libraries developed in the company that we’ll call Shared library A and B (SL-A, SL-B).Something like this could be contained in a single Visual Studio solution, where each component would be one or more projects. If so, you probably wouldn’t have versioning problems. But lets say that they’re not. Lets assume that the shared libraries are in a different solution (maybe so that they could be shared to the community in an open-source initiative) and the services and apps each live in they’re own solution also. That’s when things require control, because you can’t guarantee a single build across all of the projects. The services in this scenario are probably fulfilling references to the shared libraries via NuGet, for instance, meaning they depend on something previously built (and tested). The components have independent life cycles. You can change SL-A without redeploying service A or B. They might not be broken or not need a new feature added to the libraries. Services A and B will only require an update of SL-A if the change fixes something that was broken, or if the new features in the library are required by the service. Same goes for the seam between App A and the services. You can update the services’ code and not redeploy if it doesn’t affect your current installation, and even if you do deploy the services, you probably won’t need to update the App if there was no change to the services’ API.
In a scenario like this one, an automated integration and build process associated to a versioning process will help a lot. Test automation will help a lot, too. Let’s use SL-A as a first example, since it’s the least dependent. Our CI server will pull the code for the library, build the project and run tests. If everything passes, it can package the library up and pass it to some artifact repository (lets use NuGet).
This process will somehow need to update the packages version number, to distinguish it from the previous packages. It needs to be clear that some change has made this package a different and newer package than the ones created in previous builds. This process does not change downstream services and apps. It only affects the shared library. How the version number is changed is something I’ll talk about later (since at this point it isn’t completely clear). What is important here is that code comes out of a repository and an artifact / package is pushed to a repository, with an identifiable version number.
The same should be done to Service A – when some change to source code is pushed to the code repository, a build process should be triggered. Code is pulled from the repository by the build server along with dependent packages from the artifact repository. In this case, the newer version of SL-A should be pulled and used in the build. Tests should be run, including some integration tests that validate that the new package integrates well. If everything passes, a version number can be incremented, the service packaged and the new package pushed to the artifact repository.
The same process could be replicated to the other components in the diagram. Each Component gets built and packaged independently, pulling whatever dependencies are considered valid for it. The build and test process will validate the integration. A couple of notes:
The CI process becomes a key player in this strategy allowing automated builds and checks to occur. Still, if we rely solely on the CI server for verification, this might create a long feedback loop for developers. All committed code should go through this loop. Yet, if a developer is responsible for changes to a pair of components, it will be very useful to have a local packaging system to work with (NuGet can use a local folder to store packages), and not have to wait for the CI server’s feedback. If he is working with feature branches, for instance, he may not even have access to the default CI build processes, or might not be able to commit his code, yet need to check and develop changes in both packages. The dev should be able to make the changes and validate that they work (the software does need to evolve, after all), preferably locally, before committing and merging his branch. Merging before a coordinated fix could make the main default branch (or trunk, if you like) to become temporarily broken.
One of the steps of the CI process is to version the code after the build and tests. We want to distinguish a new build’s output as a new package with an independent version identifier. How do we change the version number in a meaningful way? There are many possible versioning strategies available. SemVer is a versioning standard for code that exposes a public API (which is pretty much anything that can get integrated and invoked). In it, version numbers have semantics associated to them. It is composed of at least three digits separated by dots:
..-
The prerelase bit is optional. Changes are made to the components based on what changes where made from the previous version to the new one:
The strategy is rather simple and clearly adds meaning to the parts in the version number. Its easy to understand the difference between a 1.0.0 and a 2.0.0 (changes API) and a 1.1.0 and a 1.3.0 (added functionality). It is a recommended standard for open source projects, and, well, any code project in general. Anything with more than one consumer will benefit greatly.
One thing to consider is when does the number change. We discussed why it changes (based on changes to it’s code / API) but not exactly when. I would say that the when is associated to the release. The version number is important especially for released software. Until it in the wild, the version number isn’t that important (though some aspects might be important to the development process). I mean, let’s say that you make multiple changes to the APIs in the library. Do you increment the major per change or join the changes in a single increment. If you go with the multiple increment in development, you might go from releasing 1.0.0 to releasing a 3.0.0 instead of a 2.0.0 because two major changes were made befor the release. Wait for release time and you’d release a 2.0.0 where you would have joined the major changes in to a single release.
Note that the component versions do not necessarily match a marketing version. Your 2.0 version for an application can be composed of a 1.0 library of library A, 3.0 service of Service A and a 12.0 of some other library. Marketing versions and component versions are not aligned, and don’t really need to be. This does mean that you should manage, list and store the changes and component to product version relations for future reference and maintenance.
SemVer isn’t the only route to go, though. Other routes exist but might not have such a clear semantics associated to it. For instance, you could align your version numbers with with your product releases. You might have to version all of your components in the same way, though, as your component versions would end tied up to release cycles and not the components change cycle. This also means you might have a component that suffered no changes but saw it’s version incremented. A library in two releases could have the same binary content, yet different version numbers. The version number here doesn’t version the binary content – it acts like a timestamp for a snapshot of the component at a release time. An example of this could be:
...
The first two digits would define the product’s major and minor version, associated to it’s roadmap. The third and forth digit aid developers in a way, where the third is a sprint number (the sprint number at which the component / product was developed and released) and the forth an incremental build number. At the beginning of a sprint, the sprint number could be incremented across all components, and any build versioned to the new number set.
The whole product would be built and versioned using the same number, whether or not it suffered changes. Some processes could benefit from something like this, and it would possibly be a simple step forward in relation to “everything is a 1.0” I described at the beginning of the post. It’s just not as powerful as SemVer in my opinion. “State at sprint” is something you’d probably get more effectively in your version control system.
Clearly, a version number strategy is not a immediate choice nor an exact science. It really depends on what you need. SemVer is probably the best route to go IF you can handle the management overhead. The benefit comes with some cost (whether it is a lot is up for debate).
One thing that I at the moment haven’t figured out is the best way to trigger the change. Version numbers don’t magically change (although an * in the version number does increment the build number in MSBuild). If you are using SemVer, the change in number will have a meaning, and that meaning is not one that a CI process can clearly decide upon.
Developers make changes to code, so a developer will somehow have to trigger that change. If he made the change, he should surely understand what type of change was made – bug fix, feature addition, or incompatible API change. Somewhere, in some element that is part of the build process, there will have to be an indication of an intent to change the version number.
Once again, this could be a very simple thing to do – go into an AssemblyInfo.cs file and change the number before committing code. It is a bit manual (requires a file edit, save, and build) but is clear on intent. If you have dozens, or a hundred projects in a solution (and therefore the same amount of assembly version files) a manual process will quickly drive you mad.
There are some tools and automation methods that can help out, depending on how you have your solution configured. Single API in the solution? Something like Zero29 would be more than enough, since it’ll run through every AsssemblyInfo in the directory and change the version semantically. The GitHub Page for the project even mentions how you can split change control between devs (who can manage major and minor values) and the CI server (that could assign patch and build numbers). You could also use it even if you have multiple API-exposing libraries. You would just need to be careful to split the different components into different directories (tree defined by components) so that the change to one won’t affect the other. Another solution is described on the “The imaginary road” blog, where a custom SemanticVersion assembly attribute is used and a MSBuild task sets the value of it based on some configuration file. Even if you don’t use the solution presented, it is well worth a read, since it can open up your thought to how you can apply the version number consistently.
A third way to make the change could be to depend on tagging and comment support and hook mechanisms in your version control system to make the update to the version number. Obviously some syntax will need to be defined that can be parsed and the version number extracted from. What is interesting about this approach is that the intention to update the version (major and minors) is clear when the developer is committing the code. The VCS becomes responsible for the update instead of a file change by the developer. If development is based on feature branching, the change could be assigned to merge steps, maybe. GitVersion uses this process to controll changes on pull requests. Another page describes how you can use git branches to control the termination (prerelease part) of the version number. This GitFlow document by DataSift describes a set of commands to use when working with Git to control versions.
One last thing to consider to consider in terms of making the change is the ability to distinguish a prerelease or development version of a library and a release version. Devs need to use prerelease versions, naturally. Code isn’t always stable. You do make changes that break builds. There is always the intermediate step from when we start to make a change to when it is completed. When we consider the need to work with changes cross components, a developer must be able to use a new package that isn’t yet in a releasable state. This is a bit more obvious if you are using feature branching. You develop the change in one component on a separate branch, but before merging to your main line, you need to use the change in a dependent project (also in a feature branch). The release package, built by the CI server, just won’t exist. This is the situation where you need local repositories for dev versions of a package.
One thing that is clear to me is that this change requires some effort. The team needs to buy in to it to use it correctly. More tools become part of the process. It’s become clear to me that new tools should come into play into the process, or at least you probably need to alter your use of some existing tools, like MSbuild, where you might need to create new tasks, or extra efforts required when using version control commands.
It does provoke a change, and as usual that may be a problem. If you can take this component into consideration right from the start, that would be so much better. Unfortunately we can’t always determine how big projects will become or fast they will evolve, so this change more often than not comes sometime after the project starts. If it can get decided and inserted before a 1.0 or at 1.0, so much the better.
Personally, SemVer strikes me as being the best approach and I am considering it’s use after this analysis. Yet, and unfortunately, a lot of change has to be made on the target project for it to work correctly. I have too many libraries and solutions that need to version independently. Change can’t and won’t be immediate, but change should have a positive impact in component separation and management. There is value in it.
I don’t yet know how exactly it will be implemented, but I understand that it is needed and desired, especially to correctly take advantage of NuGet as an artifact repository for cross solution packages. In the end, I will most likely have a solution that is more compartmentalized – more repositories, more solutions, but each with less projects in them, and possibly one per public package that is created. We’ll see…
Hopefully, if your thinking about this type of problem, these notes might have helped. Suggestions are always welcome, too.
