Skip to content

Blog

Missing CNAMEs? Certification Authority Authorization (CAA) records forbid the CA from issuing a certificate

The configuration for the Let’s Encrypt TLS certificate for this site includes a number of additional domains, mostly with my name in them, which redirect to my main domain for this site, peter.upfold.org.uk.

Some of these additional Subject Alternative Names listed in the cert are www. CNAMEs on these domains, e.g. www.peterupfold.com. It turns out that some of these www CNAMEs didn’t exist in my DNS records.

Recently, a change to Let’s Encrypt means that they appear to use Unbound 1.18 internally, where the behaviour has changed in some way, and those www CNAMEs not existing will cause this error on certificate renewal:

Problem for www.peterupfold.com: urn:ietf:params:acme:error:caa :: Certification Authority Authorization (CAA) records forbid the CA from issuing a certificate :: Error finalizing order :: While processing CAA for www.peterupfold.com: DNS problem: SERVFAIL looking up CAA for www.peterupfold.com - the domain's nameservers may be malfunctioning

It’s unclear to me how this was working before, given I was missing these www. CNAMEs entirely!

My domain registrar and DNS provider doesn’t appear to yet support adding CAA records, but that’s fine — as long as the DNS request returns NOERROR, CAA records aren’t mandatory yet.

Somewhere in this config change to Unbound 1.18 on Let’s Encrypt’s side means that the failure to resolve these www. CNAMEs means that we are not considered to be returning NOERROR for the CAA records. This causes this error above, and the subsequent refusal to issue the renewed cert.

Anyway, I added www. CNAMEs in my DNS management panel for each domain that was failing, re-issued the renewal request and now all is well.

Smartcard login — the RDP client needs to be able to access the CRL

The revocation status of the domain controller certificate used for smartcard authentication could not be determined. There is additional information in the system event log. Please contact your system administrator.

The revocation status of the domain controller certificate used for smartcard authentication could not be determined. There is additional information in the system event log. Please contact your system administrator.

If you have smartcard authentication set up for logging into certain Active Directory systems, and also a restrictive web proxy on the machine acting as the RDP client, you may run into this issue.

My mistake was checking that the RDP server had access to the CRL mentioned in the certificate.

Yes, the RDP server might be quite happy in terms of checking the certificate revocation, but if the RDP client can’t access the CRL URL (perhaps through the configured proxy), you will receive this same error.

Check connectivity to the stated CRL distribution point from the RDP client and RDP server!

X11 Xorg.log amdgpu “no screens found” when a non-graphics card is in the primary PCI Express slot

I bought a used LTO4 tape drive with a 8088 SAS connection. Why?

For fun, for backups that feel like they might be more resilient than the shingled magnetic recording hard drives I accidentally bought (thanks Seagate for disclosing that), and for the enjoyment of something so wonderfully mechanical in a world that is very “solid state”.

This necessitated a SAS card purchase, to give myself the ports necessary to actually plug in the tape drive. It seemed unhappy with one of my PCI Express slots, so I moved it up to the primary PCI Express slot — the one you’d usually use for a graphics card.

Now this Arch Linux machine has no need for fancy graphics. The APU integrated graphics on the Ryzen 7 5700G are perfectly adequate.

However, once the SAS card was in the primary PCI Express slot, X11 would no longer start. My SAS card showed up beautifully with lspci, as did the tape drive with lsscsi, but I had to sacrifice the GUI for it. Seems a little extreme, even for me.

X11 would fail with “no screens found” when the amdgpu driver was enumerating screens.

The integrated graphics moved PCI ID

What had happened is that once something is in that primary PCI Express slot, the integrated graphics moved their PCI bus ID.

I first identified where the “VGA controller” had gone with lspci:

08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c8)

Then I edited /etc/X11/xorg.conf.d/amdgpu.conf to point the BusID at that new identifier.

For me, it had moved from PCI:7:0:0 to PCI:8:0:0.

And now, I have the delight of a GUI and a SAS card, and a tape drive.

Binary release of RPC Investigator

Screenshot of RPC Investigator
RPC Investigator

Happy 2023!

I am intrigued by Trail of Bits’ new tool RPC Investigator. Exploring Windows internals is of ongoing interest, and this seems like a very interesting tool to shed light on some of that internal complexity and learn more about how the OS works.

Trail of Bits is releasing a new tool for exploring RPC clients and servers on Windows. RPC Investigator is a .NET application that builds on the NtApiDotNet platform for enumerating, decompiling/parsing and communicating with arbitrary RPC servers. We’ve added visualization and additional features that offer a new way to explore RPC.

RPC is an important communication mechanism in Windows, not only because of the flexibility and convenience it provides software developers but also because of the renowned attack surface its implementers afford to exploit developers. While there has been extensive research published related to RPC servers, interfaces, and protocols, we feel there’s always room for additional tooling to make it easier for security practitioners to explore and understand this prolific communication technology.

I could not find a binary release of the code on GitHub, just instructions on how to build it yourself.

In case others want to play with RPC Investigator without needing to build it, I publish this binary release that you can download and just run.

I have done nothing to the original repo’s code except open and build in Visual Studio 2022. I am sharing this binary build in case others want to avoid having to build the code themselves.

Binary releases here may be kept up-to-date, or may not. It is on a best effort basis. 🙂

Have fun in RPC-land!

Surface Pro Type Covers not typing after waking and the Oblitum Interception driver (Veyon)

Following teachers’ return for this academic year in September, we suddenly found ourselves with a frequent issue. After waking the Surface Pro devices from sleep, the Type Cover would often not respond to keystrokes. The on screen keyboard was not affected, but USB keyboards also stopped working. The Type Cover trackpad would continue working fine.

A full Windows restart would always bring back the keyboard functionality.

This triggered a challenging investigation to determine what was wrong. The fact that we had made no significant software changes that should affect this over the summer made me look, with guidance from Microsoft Surface Business support, to Windows Updates as a possible issue. Rolling back both September and August’s Windows Updates did not seem to have any effect.

Clearly this wasn’t a wide enough issue to be affecting everyone, or many more customers would be up in arms about having to restart 6 or 7 times in a working day!

With the issue affecting a wide range of different Surface Pro devices and different Type Covers, it looked more likely to be a software issue than hardware. Predictably, perhaps, I was unable to reproduce the issue on a device with nothing but a stock Windows install on it… it’s got to be software. Right?

The build of Windows we run is kept as simple and close-to-stock as possible, for exactly the reason that it saves you from this type of issue! Of the software we do run, the prime suspects seemed to be:

I dug a little deeper into what Veyon brings along to do its magic. Its ability to remotely control other systems for classroom management purposes, including remotely inducing the Secure Attention Sequence (Ctrl-Alt-Delete to normal folks!) means that it must have some kind of driver installed that permits this functionality. Eventually, it dawned on me that this interacts with the keyboard, making it a good candidate for the culprit for, you know, keyboard problems.

» Read the rest of this post…

It is time to talk about software security in education

Weak Bridge sign
Photo by Elliott Brown. Licensed under CC-BY 2.0

Both software quality, and the mechanisms that support its improvement, are critical to the security of people’s personal data.

In education, protecting sensitive personal data is an integral part of safeguarding those for whom we are responsible.

It isn’t good enough to shrug our shoulders if sensitive data about the children in our care could easily be compromised and leaked.

It isn’t good enough to idly preside over a plethora of vulnerable smart things sending who-knows-what to who-knows-where and say we are keeping people safe online.

Unfortunately, we know that software quality in a lot of sectors is… patchy. With the broadest brush strokes, we can separate software into these categories:

  • Really great work, made with great care;
  • Work that will need ongoing extrinsic motivation to deliver and maintain quality;
  • Software that is so badly designed it should not be out there.

The particular challenge is that it is very difficult, if not impossible, from marketing materials, to determine which category a given product is in and make an informed decision about whether to invest in it or not. Proprietary code, licence agreements that forbid investigating how things actually work, software supply chains that are opaque even to the vendor… It is even less likely you will be successful at that assessment when it’s software as a service, a.k.a “in the cloud”, because you can’t see any of it.

Throughout the software industry, there exists this problem: without regulation and enforcement of professional standards (where are the professional standards?), and because customers can’t accurately assess quality for the reasons I’ve just stated, many get away with delivering inadequate quality. Or, they could and would do the right thing, but don’t have the expertise or the extrinsic motivators that help to identify problems and incentivise improvement. Because competitors aren’t held to a higher standard either, there is a race to the bottom of the barrel for software quality in order to compete.

The best tools today we have to address critical security issues include vulnerability disclosure programmes (bug bounties), actively soliciting the support of others to identify and fix vulnerabilities. Even if a rewards programme isn’t part of the picture, the Enlightened Vendor does have a process and responds appropriately to good-faith security researchers.

However, education is an area that often suffers from a lack of computer security expertise, and certainly doesn’t have enterprise budgets. Today, education vendors generally do not fit into the “Enlightened Vendor” category, because people are not yet asking the questions.

“No-one’s ever asked us that before”… well, a lot of the time people should have asked that before.

Where I see myself fitting into this equation: I would like to be someone who can help drive this improvement in education software. My interest and experience with computer security, and being in the position to influence this as a school IT Manager puts me at the crossroads of safeguarding in education and computer security.

So, I will be asking the difficult questions that “no-one has ever asked before”. I will indeed be observing how software actually behaves in practice when trialling software. I will be asking SaaS vendors why they don’t have a vulnerability disclosure policy and making sure they are thinking about emerging threats.

This isn’t going to be particularly easy.

But, if we say we care about keeping those in our care safe from online threats to their safety, growth and development, computer security is an area we should no longer ignore.

AutoPilot hanging at “checking connection to Microsoft”

I ran into some trouble recently with a machine that had previously been registered for Intune/Microsoft Endpoint Manager AutoPilot deployment hanging on a Windows reinstall (the SSD had been replaced).

The machine would sit at “checking connection to Microsoft. This might take a while”.

Take a while it did — a spell overnight on this very screen would not help. I used Shift+F10 to get some diagnostics tooling on the system. I could see references to a /join HTTPS endpoint being accessed that seemed to be Intune-related, but it was neither obviously succeeding nor failing.

Some perusal of logfiles suggested to me that UEFI variables are involved in the AutoPilot process.

Fortunately, the machines in question are desktop PCs. A very simple way to (destructively!) clear out UEFI variables was to remove the CMOS battery for a period of time. Upon trying again, we jump right past “checking connection to Microsoft” and can move forward with the install.

On systems where a battery pull is not effective, it may be worth getting yourself into a UEFI shell and using dmpstore to identify UEFI variables in NVRAM that may be related to AutoPilot and deleting them. Sorry I can’t be more specific!

Block Mounting of ISO Images with Microsoft Intune (Endpoint Manager)

Today’s malware-loader-du-jour, Bumblebee, has been seen achieving initial access through phishing sites that convince a user to mount a downloaded ISO image. This may be a reaction to Microsoft’s recent improvements to macro-enabled document security.

Adversaries push ISO files through compromised email (reply) chains, known as thread hijacked emails, to deploy the Bumblebee loader. ISO files contain a byte-to-byte copy of low-level data stored on a disk. The malicious ISO files are delivered through Google Cloud links or password protected zip folders. The ISO files contain a hidden DLL with random names and an LNK file. DLL (Dynamic Link Library) is a library that contains codes and data which can be used by more than one program at a time. LNK is a filename extension in Microsoft Windows for shortcuts to local files.

https://cloudsek.com/technical-analysis-of-bumblebee-malware-loader/

One of the things that we can do to help our users avoid this new initial execution foothold is by blocking the mounting of ISO images, as long as you can be confident this will not break anything they actually need to do! I am fortunate enough to be able to do this.

(Djordje Atlialp shows us how to achieve this with classic GPOs, and also a more comprehensive neutering of ISO files.)

Here is what I have rolled out as an Intune PowerShell Script to block the mounting of ISOs. No reboot is required. Users will see the Mount option disappear from the context menu of an ISO file within File Explorer and will be unable to double-click to mount a malicious ISO. Or, indeed, any ISO. 😉

We will head to Microsoft Endpoint Manager admin center, go to Devices > Scripts and create a new Windows 10 and later PowerShell script.

Restrict mounting of ISOs in File Explorer

The Intune Script

UPDATE: I have made some improvements — namely, the previous one liner will cause failures to be reported in Intune on subsequent runs. We will now only add the value where it does not exist, and we will add support for Windows.VhdFile as well. It’s no longer a one-liner!

$items = @(
    @{
        path = "HKLM:\SOFTWARE\Classes\Windows.IsoFile\shell\mount"
        valueName = "ProgrammaticAccessOnly"   
    }
    @{
        path = "HKLM:\SOFTWARE\Classes\Windows.VhdFile\shell\mount"
        valueName = "ProgrammaticAccessOnly"
    }
)

foreach($item in $items) {
    if ($null -eq (Get-Item -Path $item.path).GetValue($item.valueName)) {
        New-ItemProperty -Path $item.path -Name $item.valueName -Value ""
    }
}

The body of the script can be as follows:

New-ItemProperty -Path "HKLM:\SOFTWARE\Classes\Windows.IsoFile\shell\mount" -Name ProgrammaticAccessOnly -Value ""

(This REG_SZ value need only exist, with a blank string as its Data, for this to work.)

Assign it to the device group and you are all set.

Removal

To undo this change, we can reverse what we’re doing and Remove-ItemProperty on the items we added:

$items = @(
    @{
        path = "HKLM:\SOFTWARE\Classes\Windows.IsoFile\shell\mount"
        valueName = "ProgrammaticAccessOnly"   
    }
    @{
        path = "HKLM:\SOFTWARE\Classes\Windows.VhdFile\shell\mount"
        valueName = "ProgrammaticAccessOnly"
    }
)

foreach($item in $items) {
    if ($null -ne (Get-Item -Path $item.path).GetValue($item.valueName)) {
        Remove-ItemProperty -Path $item.path -Name $item.valueName
    }
}

Conclusion

This doesn’t make you bulletproof, but will, if tolerated by your users, provide a substantial degree of protection, at the time of writing, from any number of current malware loaders that are using the ISO image technique to achieve initial code execution. The nature of the separate filesystem within the ISO presently prevents it from being marked as being from the Wild Wild West World Wide Web.

ShutdownBuddy — save resources while saving resources

In a continuation of my desire to write really lightweight software that doesn’t add to the undesirable background bloat running on computers, I set about in June-ish to write something to improve upon a VBScript-Scheduled-Task-and-shutdown.exe gaffer tape of a solution to forcing a full shutdown when a computer is idle that I had previously cobbled together.

Power management in Windows is mature and capable, for sure, but what is less obvious is how to, on shared fixed desktop computers, actually trigger a proper shutdown and not just put idle machines to sleep. Hibernation is an option, of course, but the relentless increase in complexity of Windows brings to mind the other, stability-related, benefits of regular proper restarts.

So, then, we want something that:

  • identifies when no-one is interactively signed in
  • waits a configurable amount of time
  • if still no-one has signed in in that time, shut down properly

Additionally, because this unavoidably must run with high permissions and regularly assess signed in users in the background, it should be a Windows service that is as lightweight and simple as possible. Reduced resource usage (RAM, CPU time in background) so we can shut down and have reduced resource usage (of electrical power). I can see the beauty of it already!

So I wrote ShutdownBuddy.

It is configurable through the registry:

HKLM\SOFTWARE\upfold.org.uk\ShutdownBuddy

EvaluationIntervalSeconds — DWORD. How frequently, in seconds, to evaluate for interactive sessions.

ShutdownAfterIdleForSeconds — DWORD. How many seconds of idle computer (i.e. no interactive sessions) before issuing a shutdown. This is periodically evaluated as above.

Like all my lightweight, C(++) Win32 projects, it is officially experimental as I am using these projects to learn how to write this kind of code properly. Any suggestions and improvements are gratefully received.

Microsoft Azure Backup Server “Invalid vault credentials provided” on brand new vault

Despite the inexorable march towards running all workloads in the cloud, I see some specific advantages in maintaining some on-premises servers where this makes sense. Especially in a small scale environment like the one I am responsible for, this lets me do Interesting Things with my skill set and at small scale, only because I have full control over on-prem kit.

One of the options for backing up Hyper-V workloads I am looking at is Microsoft Azure Backup Server (MABS) v3. On a brand new deployment (Windows Server 2019), following Microsoft’s guide, I ran into issues with the MABS installer.

Vault Identification -- selecting vault credentials dialogue box
From https://docs.microsoft.com/en-us/azure/backup/backup-azure-microsoft-azure-backup#software-package, (C) Microsoft. Believed to be licensed under CC-BY 4.0. https://github.com/MicrosoftDocs/azure-docs

At the point of connecting to the Vault above, it would take a long time to “validate credentials” and then:

Invalid vault credentials provided. The file is either corrupt or does not have the latest credentials associated with recovery service. ID 34513

I found a way to work around this — we need to slipstream an updated version of the Microsoft Azure Recovery Service (MARS) Agent into the MABS installer before we run it.

Tidy Up First

If you have a messy server after some failed install attempts, roll back the VM if possible, or uninstall all MABS and MARS components with Add/Remove Programs.

Also, use the MMC certificates snap-in in Local Computer mode to remove any stale vault credential certificates.

To do, this launch mmc.

MMC with blank console

We will Add/Remove Snap-ins, and choose Certificates, Local Computer.

Remove any CB_ certificates from previous runs to avoid any possibility of confusion when the registration process runs again.

Removing stale MABS certificates from the Certificates snap-in

Take care to ensure you understand what you are doing before deleting certificates and keys. Do not delete anything you are not certain is unused and related to MABS. I cannot take responsibility for your loss if you follow this guide and have issues.

In the Azure Portal, go to your backup vault and check Backup Management Servers and Protected Servers, removing any registrations from previous failed installs, so we are installing into the vault with a clean state. Alternatively, create a new vault. I unfortunately ended up with my MABS server in Protected Servers and had to delete it. Ultimately it will be registered as a Backup Management Server, not a Protected Server.

Azure Portal -- Backup Infrastructure, Backup Management Servers interface

The Workaround — Slipstream Updated MARS into MABS Installer

We are following Microsoft’s documentation on installing MABS, diverging only to slipstream in an updated MARS installer.

Download MABS from https://www.microsoft.com/en-us/download/details.aspx?id=57520

We will have the .exe and 7 .bin files. Run the .exe to extract the bin files. The install “media” in this case has now been extracted to C:\System_Center_Microsoft_Azure_Backup_Server_v3\System Center Microsoft Azure Backup Server v3.

Before we run the MABS setup wizard, we need to slipstream in the updated MARSAgentInstaller. Do not run the MARSAgentInstaller separately – we must have it invoked by the MABS wizard at the right time and in the correct context. If MARS is already installed when we try to install MABS, it will be unhappy and refuse.

Download the updated version of the MARSAgentInstaller https://download.microsoft.com/download/b/7/b/b7b5d2c9-d345-4d96-b48a-fe98a50f4c11/MARSAgentInstaller.exe

Copying MARSAgentInstaller to replace the copy in MARSAgent in the MABS installer.

Copy and replace the version in C:\System_Center_Microsoft_Azure_Backup_Server_v3\System Center Microsoft Azure Backup Server v3\MARSAgent\MARSAgentInstaller.exe before running either setup program.

Now run C:\System_Center_Microsoft_Azure_Backup_Server_v3\System Center Microsoft Azure Backup Server v3\Setup.exe (the MABS installer, not the MARSAgent installer) and proceed as normal.

Provide the vault credentials we downloaded from the Azure portal to register the server as normal.

References

This references issue raised on TechNet forums. I cannot reply as the thread is locked.

https://social.technet.microsoft.com/Forums/en-US/bdcd9f72-ccbd-4b40-b984-5f3c09cf71f6/invalid-vault-credentials-provided