Running a bleeding-edge rolling release Linux distribution like Arch Linux has its challenges and risks, but there is perhaps no greater feeling of absolute control over your own operating system! It also leads to the opportunity to have “adventures” like the one described here. While “adventure” might be tongue-in-cheek, the truth is that there is great educational value in breaking something and thus being forced to fix it!

I recently made the mistake of seeing that a pacman upgrade to icu (International Components for Unicode) would break a few Electron packages, which I needed for the open source build of VS Code, among a few other options. (I generally like to avoid Electron apps, but VS Code is a rare exception!)

I was notified by pacman that there was a dependency issue here that could not be solved, so that the entire transaction was not possible.

My mistake was thinking “oh, I’ll just exclude icu from the upgrade, as it’s causing the dependency issue”. Usually these issues are fixed in a day or two, so I’d upgrade it later.

This was indeed a mistake.

Somehow, apps were now referencing the new icu that wasn’t there already. I rebooted, to discover that the GUI wouldn’t start, and I couldn’t even use pacman in single-user mode to roll back the previous transaction, as pacman itself depended upon icu.

I’d broken the very package manager that I needed to roll back. Oops!

Install media to the rescue

I booted into the Arch install media, and set about to address the issue.

chrooting into the target filesystem and using pacman wasn’t an option, as running the package manager from the target filesystem failed with the same dependency issue.

I had a few false starts with trying to run pacman from the install medium, pointing it at the mounted OS partition with --sysroot.

While this didn’t pan out, I remembered that pacstrap, while designed for install-time, presumably could install packages in the target filesystem without needing pacman itself. I was concerned that this might reinstall some base packages, but it turns out you can specify which packages to install manually on its command line.

So, I gathered the list of packages that needed upgrading by using pacman -Syu --sysroot /mnt against the target filesystem, and then supplied this package list to pacstrap:
pacstrap -G -i -M /mnt icu brltty electron27 electron28 electron29 electron30 freerdp freerdp2 harfbuzz-icu raptor
And… we’re back!

Lessons learned

• The manual isn’t joking about “Partial upgrades are unsupported”.
• Considerable caution is required when using pacman --exclude
• “The OS is ephemeral and I can rebuild it” is a bit too relaxed an attitude when you don’t really want to have to rebuild the OS at short notice. Back up the OS packages and libraries too.
• Really understanding the package manager and install process gives you the tools to pull yourself out of the holes you dig for yourself!