SGrottel.de

A lot of stuff was happening around Lua and its NuGet package.

Lua 5.4.5 has been releases. So, it was time for an update of my Lua NuGet package. And for a surprise. And for some pain.

In the end, all was resolved, and the new version of the Lua NuGet package is online.

Although, Lua 5.4.5 was removed again in favor of a hopefully quickly upcoming Lua 5.4.6.

Update 20230514: Lua 5.4.6 is here!

Surprise: The Lua NuGet package was unlisted!?

Some months ago, my notifications from Github stopped working. No idea why. But since I don’t have much traffic there, it was never a high priority for me to fix it. For that reason, however, I nearly missed this issue raised by Smurf-IV “[Question] Why does nuget state that the packages have been witdrawn ?”

What? I have absolutely no idea what had happened there. But I have a guess. The NuGet package was originally created by the CoApp organization. However, at some point they stopped updating it. When I was needing a newer version of Lua, I decided to create a NuGet package myself. I already had good experience in creating native C++ NuGet packages, so I gave it a try, and it worked nicely. At that point I reached out to the CoApp org., and asked if they would let me co-author the NuGet package for Lua. My idea was that I could publish this package using the nice and precise NuGet package id. They agreed, gave me co-authorship, and I started publishing NuGet packages for every release of Lua ever since.

And now the package was unlisted by “the author” and marked as being deprecated and having critical bugs. What? I can tell you that, it was not me!

The “critical bugs” thing worried me a bit. I my previous job I was also tasked to regularly screen our code base for CVEs. So, I do know a couple of things to look at. I did a quick check for Lua, and yes, there are CVEs, but nothing out of the ordinary. Nothing as severe requiring the package to be locked down that hard.

And another thing became apparent: the CoApp org no longer seems to own any NuGet packages at all. None. Which is quite surprising.

So, my best guess is, the CoApp org on NuGet was maybe disbanded, but surely stopped their commitment on maintaining any of the NuGet packages. As a precaution, they shut down all the NuGets, and maybe automatically marked all of them as being deprecated and legacy. I am not sure about the “critical bugs” thing. That is why I assume an automatic “shutdown,” which indiscriminately flagged and unlisted all packages.

Well, my commitment to maintain the Lua Nuget package stands!

I will continue to release version updates and similar. And therefore I (re-)listed the 5.4.x versions of the Lua package again and removed the warning flags about being legacy, deprecated or having critical bugs. I could only speculate about the reasoning of the CoApp group. But, I won’t.

Pain: Moving from AppVeyor to Github Actions

All that happened in the wake of the release of version 5.4.5 of Lua. One task which is on my ToDo list for a long time was to migrate and consolidate the different code and CI platforms. Up until now I built the different flavors of Lua for the NuGet package on AppVeyor. And I planned to move this task to Github Actions for a long time. So, I decided that now is finally the time to do that. One reason being an upcoming public holiday in Germany, which gave me the extra time to take care of that.

While Github actions are quite nice for straight forward projects, and while they are quite powerful, I immediately run into an issue: my NuGet package also contains built flavors from older Visual C++ toolset, currently down to v120. And the runner VM of the GitHub actions only comes with the most recent Visual C++ and toolsets installed.

And so, a long run of trial-and-error started for me. A lot of pain, headache, and “why, why why…” moments. In the end, these are the solutions I came up with:

How To: Adjust Visual Studio 2022 installation on Github Actions Runner

Let’s start easy. The Github runner comes with Visual Studio Enterprise 2022 installed. So, the most recent toolchain should work right away, namely Visual C++ v143, v142, and v141, and it did. Now, for the tool chain v140, which can be installed via the Visual Studio Installer. So, that should be easy. Well, in theory, yes. The Visual Studio Installer offers command line arguments to modify an installation and to add components, and that is what I need to do. The only issue is, the Installer is not a console application. When it is started, the calling console immediately returns, thus a job in the workflow Yaml won’t wait, at least not automatically. I need to invoke the installer with `Start-Process` and need to explicitly tell the command to wait for the spawned process to finish.

When you run the Installer locally within a command prompt, like a Powershell, you will see output messages. When running on Github CI, you don’t. As I said, the Installer is not a console application. As such, it likely performs a manual console host connection to push in its output messages. And on the Github CI, the console host likely does not exist, as the process output is directed from the CI agent process into a temporary file. As a result, the Installer does not find a console host, and thus does not print any output messages. I am flying blind. To have some info of whether or not the command succeeded, I use ` vs_installer.exe export` to dump the list of installed components. That’s not beautiful, but it is good enough for manual debugging.

So, this is what worked:

- name: Modify Visual Studio 2022
  if: matrix.toolConfig.vs2022addMod != ''
  shell: pwsh
  run: |
    $component = "${{matrix.toolConfig.vs2022addMod}}"
    Write-Host "Adding:" $component

    Start-Process -FilePath "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" -ArgumentList "modify","--add",$component,"--installPath","`"C:\Program Files\Microsoft Visual Studio\2022\Enterprise`"","--passive","--includeRecommended","--norestart","--noUpdateInstaller" -Wait

    Start-Process -FilePath "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" -ArgumentList "export","--config","info-post.txt","--installPath","`"C:\Program Files\Microsoft Visual Studio\2022\Enterprise`"","--passive","--noUpdateInstaller" -Wait
    Write-Host "VS2022 Components after modification:"
    Write-Host (gc info-post.txt)

    Write-Host "Added:"
    Write-Host (gc info-post.txt | Select-String $component)

How To: Visual Studio 2013 on Github Actions Runner

This leaves with the build with the oldest still supported tool chain: v120 aka Visual Studio 2013, and this one is a real pain! I cannot blame most people, as this is an over 10 year old software. I would not want to give support to something like that. But, with this NuGet package, I sort of do.

The current Visual Studio does not support this tool chain. So, I actually need to have Visual Studio 2013. Ask the Internet what to do, and it will tell you to install Visual Studio 2013, e.g. the free “Expression” Edition, via Chocolatey. I tried, and it failed with the most cryptic error message ever:

“Exit code indicates the following: Generic MSI Error. This is a local environment error, not an issue with a package or the MSI itself […]” bla bla bla.

It gives a lot of wrong hints to what to do. The actual reason is quite different: The package actually only downloads the bootstrap installer of Visual Studio, around 1.2 MB. The rest of it would have been downloaded during installation on demand. And there is the issue. Microsoft removed those old download sources, as they were vulnerable because they were signed with SHA-1 hashes. And with that all Chocolatey packages are now broken. … I might be wrong, but the effects and the sparse info I carved out of the log files really look like that.

Meaning, I need a full Visual Studio 2013. At this time, it is officially available through Microsoft’s download website my.visualstudio.com. After logging in and clicking through a dynamic website, that is. For my automation purposes not acceptable. So, I rehost the DVD iso on my OneDrive. I’d say it ok, but not ideal. And now, I need to install the full Visual Studio 2013 then. Sounds expensive. Sounds painful. Sounds slow. Sounds like a job for Docker. … Well, yes, but sadly, I would need a Windows docker image, which would need a Windows Docker host, and that seems not supported on Github, yet. So, yes, I need to install Visual Studio 2013 on the runner.

- name: Install VS 2013 tools
  if: matrix.toolConfig.toolset == 'v120'
  shell: pwsh
  working-directory: ${{env.GITHUB_WORKSPACE}}
  run: |
    Write-Host "Downloading VS 2013 Express for Windows Desktop (w Update 5) DVD ISO..."
    Invoke-WebRequest -Uri "${{ secrets.VS2013ISO_URL }}" -OutFile dvd.iso
    gci dvd.iso
    $dvdiso = (Resolve-Path .\dvd.iso).Path

    Write-Host "Mounting Disk..."
    $DiskImage = Mount-DiskImage -ImagePath $dvdiso -StorageType ISO -PassThru

    $volumeInfo = (Get-Volume -DiskImage $DiskImage)
    $volumeInfo

    cd ($volumeInfo.DriveLetter + ":\")
    gci

    Write-Host "Installing Visual Studio 2013 Express..."
    Start-Process -FilePath ($volumeInfo.DriveLetter + ":\wdexpress_full.exe") -ArgumentList "/Q","/NoRestart","/NoRefresh","/NoWeb","/Full" -Wait

    Write-Host "Unmounting Disk..."
    Dismount-DiskImage -DevicePath $DiskImage.DevicePath

And this works. And, as expected, is painfully slow. To limit the overhead of this installation step, it does make sense to reduce the width of the job matrix in the workflow: only have one job to perform this installation and afterwards perform all build steps sequentially. In the end, this is almost the factor x4 faster as you would expect, due to the massive installation overhead. Luckily for me, this workflow is not a pull-request blocker during active development, but only a release automation.

Final Thoughts

In the end, the migration to Github Actions works. Was it worth it? I mean, Lua releases like once every two years. Do I even need CI for that? And, anyway, isn’t native NuGet dead, replaced by other means, like vcpkg, or something?

Well, all this is true. I did not follow through with this undertaking because of having a CI for my Lua NuGet Package. My main reason is “because I can,” or, really, “because I can’t, yet.” It was a good opportunity to improve my Github-Actions-Fu. My recommendation to you: if you see such an opportunity to learn and improve your skills, with the benefit of getting something usable out of it, do it.

And, now I am waiting for Lua 5.4.6 and maybe new surprises…