Everything about Nuget packages
This article discusses a way how to create and use a NuGet Package for Libraries which are not yet stable and which are still under active development.
Especially, the core idea is to develop such an library and a project consuming this library in an efficient way.
As we develop on Windows with Visual Studio, NuGet is a powerful tool to manage library dependencies in Projects, for external 3rd Party libraries, as well as for our own libraries. Using Git our alternatives would be:
- Git Submodules – with their known problems (detached head, multiple commits, etc.)
- cmake – vs. Visual Studio solutions
- manual paths / zip packages – manual work
Using NuGet seems the way to go for Windows-Only projects. Even for native c++ components. (Side note: coapp is dead and is not missed.)
However: if we have a library component and a consuming project, which we have to actively develop both at the same time, the NuGet process gets prohibitively tiring. For each change in the library, we need to build the lib, repack the NuGet package, increasing its (build) version number, update the consuming project’s NuGet reference, and build the project… For every tiny typo.
The core idea is to prepare the NuGet package in such a way, that a local override of the package’s content is possible. Think of an environment variable holding the file system path to the library’s development directory. If this environment variable is not set, the content of the NuGet package is use. If the environment variable is set, then include paths and library paths are set to directly use the files found in that development directory (obviously, resulting in an unclean in-development version of the project).
Since native NuGet packages simply use an MSBuild script to integrate themselves into a project, we have (almost) all the possibilities MSBuild has to offer.
Set up the NuGet Package
Creating a native C++ library NuGet package without coapp is not very difficult, but out-of-focus of this article. For the sake of simplicity I only discuss the (second) easiest task: a static library. We need some header file and some static library files. All in all the whole file looks like this:
<?xml version="1.0" encoding="utf-8"?> <package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd"> <!-- Create package: cd solution directory nuget pack NugetDevPackageTest.testLib.nuspec -Version 0.2.0-build001 --> <!-- package metadata --> <metadata> <id>NugetDevPackageTest.testLib</id> <version>$version$</version> <title>NugetDevPackageTest testLib</title> <authors>Sebastian Grottel</authors> <projectUrl>https://bitbucket.org/sgr_faro/nuget-dev-package-test/overview</projectUrl> <requireLicenseAcceptance>false</requireLicenseAcceptance> <description>Nuget package of `someLib` to test dev nuget package concept</description> <tags>nugetdevpackagetest testLib native</tags> </metadata> <!-- content --> <files> <!-- include header files --> <file src="DemoClass.h" target="build\native\include" /> <!-- static libraries --> <file src="lib\Win32\Debug\some.lib" target="build\native\lib\Win32\Debug" /> <file src="lib\Win32\Release\some.lib" target="build\native\lib\Win32\Release" /> <file src="lib\x64\Debug\some.lib" target="build\native\lib\x64\Debug" /> <file src="lib\x64\Release\some.lib" target="build\native\lib\x64\Release" /> <!-- build rules --> <file src="NugetDevPackageTest.testLib.targets" target="build\native" /> </files> </package>
There are two interesting aspects to highlight here:
- The static libraries can be collected from any directories. Important is that you specify the correct target directories. Follow the semantic rules NuGet specifies.
- The first build rule file, NugetDevPackageTest.testLib.targets, controls the integration of the NuGet Package’s content into the Visual Studio project. This file must have the same name as the package itself.
We set up the targets file as follows:
<?xml version="1.0" encoding="utf-8"?> <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <!-- If development directory is empty, use content from the package --> <PropertyGroup Condition="'$(NugetDevPackageTest_testLib_DevDir)' == ''"> <__NugetDevPackageTest_testLib_IncludeDir>$(MSBuildThisFileDirectory)include/</__NugetDevPackageTest_testLib_IncludeDir> <__NugetDevPackageTest_testLib_LibrarayPath>$(MSBuildThisFileDirectory)lib\$(Platform)\$(Configuration)\some.lib</__NugetDevPackageTest_testLib_LibrarayPath> </PropertyGroup> <!-- If development directory is set, use the content found there --> <PropertyGroup Condition="'$(NugetDevPackageTest_testLib_DevDir)' != ''"> <__NugetDevPackageTest_testLib_IncludeDir>$(NugetDevPackageTest_testLib_DevDir)/</__NugetDevPackageTest_testLib_IncludeDir> <__NugetDevPackageTest_testLib_LibrarayPath>$(NugetDevPackageTest_testLib_DevDir)\lib\$(Platform)\$(Configuration)\some.lib</__NugetDevPackageTest_testLib_LibrarayPath> </PropertyGroup> <!-- If development directory is set, define an additional macro --> <ItemDefinitionGroup Condition="'$(NugetDevPackageTest_testLib_DevDir)' != ''"> <ClCompile> <PreprocessorDefinitions>HAS_NUGETDEVPACKAGETEST_TESTLIB_DEVDIR;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> </ItemDefinitionGroup> <!-- Compiler options: package macro and include paths --> <ItemDefinitionGroup> <ClCompile> <PreprocessorDefinitions>HAS_NUGETDEVPACKAGETEST_TESTLIB;%(PreprocessorDefinitions)</PreprocessorDefinitions> <AdditionalIncludeDirectories>$(__NugetDevPackageTest_testLib_IncludeDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories> </ClCompile> <ResourceCompile> <AdditionalIncludeDirectories>$(__NugetDevPackageTest_testLib_IncludeDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories> </ResourceCompile> </ItemDefinitionGroup> <!-- Linker options: library (full) path --> <ItemDefinitionGroup> <Link> <AdditionalDependencies>$(__NugetDevPackageTest_testLib_LibrarayPath);%(AdditionalDependencies)</AdditionalDependencies> </Link> </ItemDefinitionGroup> </Project>
Create the NuGet package:
nuget pack NugetDevPackageTest.testLib.nuspec -Version 0.2.0-build002
(obviously, you adjust the version number as you like to.) and be happy that it already does work as you intended: if the environment variable NugetDevPackageTest_testLib_DevDir is set, the content of this folder is used instead of the NuGet package’s content.
Setting an environment variable and the need to restart Visual Studio any time you change it, this is obviously not great. We can improve on that by adding a GUI.
Visual Studio Property Page
We add an Visual Studio property page for your project to the NuGet package, allowing us to change the setting of the variable from within Visual Studio. For this we add an XML description of the values we want to change:
<?xml version="1.0" encoding="utf-8"?> <ProjectSchemaDefinitions xmlns="clr-namespace:Microsoft.Build.Framework.XamlTypes;assembly=Microsoft.Build.Framework"> <!-- Project property settings to override content of nuget development packages --> <Rule Name="ProjectSettings_NugetDevOverride" PageTemplate="tool" DisplayName="Nuget Development Override" SwitchPrefix="/" Order="1"> <!-- Category to identify this package --> <Rule.Categories> <Category Name="NugetDevPackageTest_testLib" DisplayName="NugetDevPackageTest.testLib" /> </Rule.Categories> <!-- Store override settings only in the user file --> <Rule.DataSource> <DataSource Persistence="UserFile" ItemType="" /> </Rule.DataSource> <!-- Override development path --> <StringProperty Name="NugetDevPackageTest_testLib_DevDir" DisplayName="Development Directory" Subtype="folder" Description="Override content of Nuget package with content of this directory. If this field is empty (default) the content of the nuget package is used. This value is persistent as user setting only." Category="NugetDevPackageTest_testLib" /> </Rule> </ProjectSchemaDefinitions>
Note that the StringProperty directly stores it’s value in the variable we are already using. Make sure not to use this variable as an Environment variable anymore!
Now, we reference this property page from the other files of our NuGet package:
... <!-- build rules --> <file src="NugetDevPackageTest.testLib.targets" target="build\native" /> <file src="overrideSettings.xml" target="build\native" /> </files> </package>
<?xml version="1.0" encoding="utf-8"?> <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <!-- include the gui property page to override the development directory --> <ItemGroup> <PropertyPageSchema Include="$(MSBuildThisFileDirectory)\overrideSettings.xml" /> </ItemGroup> <!-- If development directory is empty, use content from the package --> ...
Now create the new version of the package. Update the package reference in your consuming project. You might need to restart visual studio once. And you should find the new property page in the settings of your project.
Use the NuGet Package or the local Version
Be Aware of all the pitfalls
- Never commit the .user file of your projects to the repository!
- If your project code requires local changes within the code usually packed into NuGet, your project will not compile correctly on any other machine!
- Before you push, or at least before you create a pull request, you need to push and publish the changed NuGet project and you need to reference the NuGet package cleanly without override.
- You need to use two Visual Studios running, one for your project and one for the NuGet-ted project.
- If you run your project in the debugger, and you step into functions from the NuGet project, the corresponding source file is opened in the Visual Studio of your project. You can edit the files there, but of course you cannot compile the NuGet library here. Remember to switch to the right Visual Studio for that.
The Proposed Process
|testApp – Visual Studio #1||testLib – Visual Studio #2||Comment|
|Open testAppsolution in Visual Studio #1|
|Build testApp||This fetches the NuGet package of testLib and builds the App with the content of that package, e.g. version 1.0.0|
|Open testLib solution in Visual Studio #2|
|(Build testApp)||Nothing changes. Still the NuGet package is used. Build should not do anything (up to date)|
|Open Project settings and override the nuget settings.||Reference the directory where you cloned and built testLib|
|(Build testApp)||Known Issue: Visual Studio still believes the project would be up to date|
|Rebuild testApp||App is now built with the testLib built in your local directory!|
|(Build testApp)||The project is up to date (which is correct)|
|Change some code and build testLib|
|Build testApp||The project correctly detects that it needs to build (at least to link) again. The changes of the local testLib can directly be used.|
|You start testApp in the Debugger||You can even enter code of testLib, look at local / private variables, etc. The corresponding source files of testLib are opened in Visual Studio #1!|
|You edit source of testLib||WRONG Visual Studio!|
|You hit Build → testApp is up to date||It is the wrong Visual Studio. You cannot build testLib there. You need to switch to the other Visual Studio|
|Visual Studio might detect the changes at the source files, if they are opened.||If the files are not opened, everything is ok. If Visual Studio does ask you, select to keep changes made outside Visual Studio #2|
|Hit F5 (start Debugging)||Binary and source code do not match! Visual Studio #1 should warn you about it if you try, e.g., to set a breakpoint in source code of testLib|
|Build testLib||The testLib is update with the changes to the source code you made in Visual Studio #1|
|Hit F5 (start Debugging)||Visual Studio #1 correctly detects the updates of testLib, builds again, and executes testApp with the newest version of testLib.|
Develop your changes in testApp and testLib. Now you are finished and want to push a stable version to your CI system
|Be sure testLib is correct in all configurations|
|Commit + Push testLib|
|Trigger built / update of new NuGet package (we call it Ver++)|
|Disable the override settings for the NuGet package|
|Update the testApp project to use the NuGet package Ver++||Ideally this package should be the result of your CI system.|
|Compile and testApp in all configurations||Since you had the identical code tested before locally, this should not show any problems. If it does, you likely missed something before.|
|Commit + Push your final update of testApp||This should be solely the increment of the NuGet package version|
|Build / Test your testApp on your CI system||All should work fine.|
Visual Studio Property Pages GUI
The Visual Studio GUI is sometimes not correctly updated. One such situation is created by the property page descriptions injected via NuGet packages.
If you initially add a NuGet package with such a property page, or if the NuGet package you are using changed its property page, Visual Studio might not show you the right proptery page (or might not show any property page).
If this happens, restart Visual Studio after installing / restoring the NuGet package and you should be fine.
Visual Studio’s Fast Up to Date Check misses Changes of the StringProperty
When in Visual Studio a build is triggered (e.g. by pressing F5 for a Debug start), Visual Studio does not invoke MSBuild to determine if the project is up to date. Instead is uses an internal mechanism to quickly decide if a rebuild is required. This internal mechanism, known as fast up to date check, does not perform all required checks. E.g., it misses changed to the paths defined by the variable defined within the property page GUI created by our NuGet package.
This means, whenever the value of this variable is changed, you should trigger a rebuild of the consuming project.
One workaround would be to disable the fast up to date check for this project. However, for large projects, like SCENE, this introduces more pain than the required rebuild.
There is an semantic problem: you reference a Nuget package version, e.g. 1.0.0. But you override the content with changed files. Those will eventually be published as a new version, e.g. 1.0.1. But until then, your consuming project will reference those as version 1.0.0.
Your build process must be aware of this issue.
You should introduce integration tests to make sure such information mismatch never enters release branches.
Deep Dependency Graph Version Conflict
Consuming the unstable library from the development directory, is a project setting, not a solution setting. If you have a deep dependency graph and you are consuming this library via multiple routes, the override setting only influences one single route! Especially, if you use another Nuget package which uses your lib nuget, you cannot do anything about it. You need to be carefully aware of potential binary compatibility issues in such a construct.
Workarounds could be based compile-time and run-time checks of version numbers and the “dirty-lib-flag” (cf. HAS_NUGETDEVPACKAGETEST_TESTLIB_DEVDIR).
Hi, I updated the NuGet package for GLFW once again.
The content is basically identical to before, but there are three important changes:
- The version number now uses four numbers: 184.108.40.206. The fourth number is a package-build number which I can increase, even though I pack the same version of GLFW. This finally allows me to get rid of the “pre-release” flags I previously used for the VS 2017 repackage.
- I no longer use CoApp. CoApp is dead and not really missed. At work I started writing .nuspec and .targets files manually, and this is neither very hard, nor very much work.
- Last but not least, I added some properties, available within the Visual Studio GUI as part of the properties pages. There you can now enforce configuration and toolset version of the library you want to use. This is especially useful if you have special configurations which cannot automatically mapped by NuGet. You can now map those manually, at least.
Of course, everything is available on bitbucket.
Now, I am waiting for the next version of GLFW to be published.
I updated my nuget packages to support Visual Studio 2017 (v141):
- GLFW – https://www.nuget.org/packages/glfw/3.2.1-v141to2015
- Lua – https://www.nuget.org/packages/lua/
- voro++ – https://www.nuget.org/packages/voroplusplus/
Using this opportunity, I dropped support for Visual Studio 2010.
And now there is Voro++ on Nuget. Voro++ is a library and utility, written by Chris Rycroft (with Applied Mathematics in Harvard SEAS) for compute Voronoi cells für 3D data. I use his lib actually to compute natural neighbors in my particle data sets. For me, Voro++ is better suited than other libs, like e.g. qhull, for several reasons: it is easy to use, streamlined for 3D, and natively support periodic boundary conditions. The single drawback I found, there are no official binary releases for Windows. And that is why I started this nuget package.
The library’s source code is rather clean and compiles without problem in Visual Studio. So all I did, was compiling the static library for the four latest Visual C++ versions, and I put the nuget package together. My Visual Studio solution and project files are also publicly available on bitbucket: https://bitbucket.org/sgrottel_nuget/voro
The package contains the static library, the header files, some files from the documentation, and the command line tool built with MSVC 2015. If you only want the command line tool and don’t care for the lib, simply download the nuget package, unzip it (yes, nuget packages are basically zip files), and you’ll find the exe in the tools subdirectory. For the full documentation of the library, go the Chris’ website or the original source code package.
What I didn’t include in the nuget package are any samples. But compiling them for yourself is as simple as it could be:
1. start a new C++ console application in visual studio.
Be sure to deactivate precompiled headers and to make the project empty.
2. drag-drop-add any cc-example.
Just use any file from Chris’ source code package into the project.
3. Add the voroplusplus nuget package.
4. And you’re done! Compile it and run it by a push of a button.
I adopted yet another nuget package: Lua
The reason, of course, is MegaMol, more precisely an upcoming addition to MegaMol enabling scripting of all internal management functions. After some thought, we opted for Lua as scripting language in our project. Building Lua is no big deal. But I also wanted the build process of MegaMol as a whole to be a simple as possible. On Windows this calls for nuget.
Of course, the nuget package already existed, but it was old. Instead of just starting another one, I contacted the original authors of the coapp team. They were happy to grant me access to the package, so that I could update to the new binaries. I will try my best to keep it up to date and to support many Visual Studio versions and settings.
I present you another NuGet package: GLFW. Ok. Originally redxdev created the package. And I only took the official prebuilt binaries from www.glfw.org and repacked them. All that fun for the support of Visual Studio 2015.
- More about GFLW on the Project website, and
- on Bitbucket you find the Package Project.
- Readable Perma-Link: http://go.sgrottel.de/nuget/glfw
NuGet is a handy package system for Visual Studio. Originally meant for Dotnet libraries it was extended some time to support native C++ projects as well. The only catch is, that the packages need to be made especially for NuGet. There are some, but by far not all which one (I) needs for the daily work. Most problematic is that many packages do not support Visual Studio 2013. And that is even today, as Visual Studio 2015 is almost here.
So there is only one thing to do: join the fray! Well, I could stay out of it and continue to grumble about it, but let’s just keep that as plan b. So, here it is; my first NuGet package:
(with kind support of the author)
And this will not be the last you’ve seen of me.