Late last year, I inherited an in-progress Mono CSharp upgrade that was a little behind schedule. The reasons behind this weren't very clear so I wanted to find out where the upgrade stood in terms of completeness.
Talking to other developers or QA personnel, things like fixing issues with SSL certificates and CI/CD coordinations seemed to be the dominant themes.
Nevertheless, I was interested in taking this on and completing the job for the client.
The current version in use was Mono 5.18 and the client wanted to upgrade to Mono 6.12.
Looking through a couple of PRs on Github, it wasn't super obvious how the upgrade was supposed to work and how the work would be coordinated. The client had multiple production-like environments in AWS, along with development environments for engineers on laptops using Vagrant virtual machines.
The production-like environments are managed by Saltstack so versioning is controlled through configuration and an upgrade isn't as simple as just logging onto a machine and typing in a couple of commands. There is a QA, Staging, and Production "environment" where each environment has at least 2 machines running Mono CSharp.
There is also a machine responsible for building artififacts of all of the clients internally development software. This machine runs its own version of Mono CSharp and needs to stay in sync with the other production-like environments. This was one area where the previous engineer doing the initial work failed to take into consideration.
Since the existing PRs left something to be desired in their current state, there was some work to be done in order to upgrade all environments and do it in a way that matched the clients release process.
Luckily, the configuration on the development Vagrant machine was straightforward and just required making a version change to a pillar file.
Basically, a new feature, bug fix, or upgrade proceeds as software that is built and sequentially promoted through development, QA, staging, and finally production servers when it is officially released. Due to the complexity of this upgrade, it was determined it was better off done on its own along with no other code changes.
As far as the work involved, I would need to:
The client does a major software release every 2 weeks, and normally does a hotfix or minor release once or twice between every major release. This presented a problem since once the builder machine was upgraded, it was not possible to create builds for environments still on the older versions. The setup here really only dictated an all-at-once deployment model, which obviously wouldn't allow for any testing outside of a development environment, and would prevent any release in the meantime.
After studying the PRs extensively, it was discovered that upgrading the Salt config for the builder machine was not possible and not accounted for. I determined that any upgrade to the production-like environments would not benefit from a build done with the upgraded version of Mono CSharp.
Getting the changes deployed would several coordination steps and cool heads. While performing the upgrade, the group would not be able to perform any releases.
After following all of the carefully crafted steps, the upgrade was deployed and enabled with no errors. The release resulted in zero downtime for the business and there was no need to interrupt the upgrade process for a hotfix or minor release.