My attention has turned recently to how to automate the installation of security updates on various Linux distributions.
As Van Patten Media runs more servers, the effort and time needed to apply critical security updates promptly grows. Waiting several days to get security fixes just isn’t acceptable in a post-Shellshock era, yet there is always a risk of a completely automated update breaking important functionality.
One of the projects I will be investigating over these few weeks is how we might build an automated test environment that could apply the updates quickly to a test VM, run a test suite to verify none of our critical client functionality breaks, then push those updates to the live servers.
There are various solutions for truly automatic updates; I focus on Debian’s unattended-upgrades package here. What seems to be more difficult, however, is being able to push that list of ‘approved’ packages and just install those when we are ready.
My current train of thought on how to proceed on this is as follows:
- Test box installs the day’s upgrades
- Runs the test suite automatically
- If the test suite passes, it determines which packages were most recently installed
- Pushes that/those package names to the unattended-upgrades whitelist on the clients
- Clients, on next unattended-upgrades run, will install those upgrades
- Upon successful upgrade, we reset the whitelist
I am in the very early stages of looking at this, so that is a very rough sketch of where my thoughts are currently. There are missing pieces, but I was looking at Watir for the browser test suite component.
I would be interested to hear from anyone who has looked at this before, or if anyone knows of any interesting similar projects I haven’t found!