The configuration for the Let’s Encrypt TLS certificate for this site includes a number of additional domains, mostly with my name in them, which redirect to my main domain for this site, peter.upfold.org.uk.

Some of these additional Subject Alternative Names listed in the cert are www. CNAMEs on these domains, e.g. www.peterupfold.com. It turns out that some of these www CNAMEs didn’t exist in my DNS records.

Recently, a change to Let’s Encrypt means that they appear to use Unbound 1.18 internally, where the behaviour has changed in some way, and those www CNAMEs not existing will cause this error on certificate renewal:

Problem for www.peterupfold.com: urn:ietf:params:acme:error:caa :: Certification Authority Authorization (CAA) records forbid the CA from issuing a certificate :: Error finalizing order :: While processing CAA for www.peterupfold.com: DNS problem: SERVFAIL looking up CAA for www.peterupfold.com - the domain's nameservers may be malfunctioning

It’s unclear to me how this was working before, given I was missing these www. CNAMEs entirely!

My domain registrar and DNS provider doesn’t appear to yet support adding CAA records, but that’s fine — as long as the DNS request returns NOERROR, CAA records aren’t mandatory yet.

Somewhere in this config change to Unbound 1.18 on Let’s Encrypt’s side means that the failure to resolve these www. CNAMEs means that we are not considered to be returning NOERROR for the CAA records. This causes this error above, and the subsequent refusal to issue the renewed cert.

Anyway, I added www. CNAMEs in my DNS management panel for each domain that was failing, re-issued the renewal request and now all is well.

With a relatively recent Azure AD hybrid directory under our belts, we decided at work not to use the older Azure AD Connect tool and instead use the newer Azure AD Connect Cloud Sync. It’s lighter weight, doesn’t require a SQL database — lots of reasons to love it.

It does appear that, juuuust in time for our deployment, password writeback is supported, in preview.

However, I struggled to follow the official instructions to enable it, as the cmdlet did not seem to be available after importing the DLL.

Import-Module 'C:\\Program Files\\Microsoft Azure AD Connect Provisioning Agent\\Microsoft.CloudSync.Powershell.dll' 
Set-AADCloudSyncPasswordWritebackConfiguration -Enable $true -Credential $(Get-Credential)

Set-AADCloudSyncPasswordWritebackConfiguration : The term 'Set-AADCloudSyncPasswordWritebackConfiguration' is not
recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if
a path was included, verify that the path is correct and try again.
At line:1 char:1
+ Set-AADCloudSyncPasswordWritebackConfiguration
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Set-AADCloudSyn...ckConfiguration:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

A little bit of investigating revealed that the DLL does indeed export the cmdlet, so what is going on?

I tried to run the above in PowerShell 7. It imported the cmdlet, but hit an issue with running it when importing its required libraries.

My Workaround

My workaround is to use PowerShell 7, but import the module with the -UseWindowsPowershell compatibility switch.

Import-Module "C:\Program Files\Microsoft Azure AD Connect Provisioning Agent\Microsoft.CloudSync.Powershell.dll" -UseWindowsPowerShell 
Set-AADCloudSyncPasswordWritebackConfiguration -Enable $true -Credential $(Get-Credential) 

And with that, password writeback is working on Azure AD Cloud Sync!

Following applying the 2020-05 Cumulative Update for Windows Server 2016 (1607), a PFSense 2.4.5 Hyper-V virtual machine on this host OS suddenly became very unstable.

The machine appears to experience high load averages immediately following boot, with a large percentage of the vCPU usage (>99%) being shown as “system” in top.

Additionally, sometimes it is seen that the pfctl process is also using close to 100% of a CPU. The system is unresponsive, or intermittently responsive throughout this time — both in terms of the console, the web UI, and packet forwarding that the machine should be doing.

After a few minutes, Hyper-V appears to restart the guest OS. This isn’t a PFSense kernel panic and no dump is made by PFSense.

I spun up a new VM of the unstable daily snapshot of PFSense, a Generation 2 UEFI-based VM. This completely eliminated these problems for us in this new VM — it works fine, routes just fine for what we’re doing and is responsive.

I don’t have a solution — I just want to throw this issue out there in case others have experienced it, in case we might work together to figure out what’s going on here.

In my recent trials and tribulations with ADFS 3.0, I came up against an issue where we were unable to host ADFS 3.0 with Nginx as one of the layers of reverse proxy (the closest layer to ADFS).

When a direct connection, or a cURL request, was made to the ADFS 3.0 endpoints from the machine running Nginx, all seemed well, but as soon as you actually tried to ferry requests through a proxy_pass statement, users were greeted with HTTP 502 or 503 errors.

The machine running ADFS was offering up no other web services — there was no IIS instance running, or anything like that. It had been configured correctly with a valid TLS certificate for the domain that was trusted by the certificate store on the Nginx machine.

It turns out that despite being the only HTTPS service offered on that machine through HTTP.sys, you need to explicitly configure which certificate to present by default. Apparently, requests that come via Nginx proxy_pass are missing something (the SNI negotiation?) that allows HTTP.sys to choose the correct certificate to present.

So, if and only if you are sure that ADFS is the only HTTPS service you are serving up on the inner machine, you can force the correct certificate to be presented by default, which resolves this issue and allows the Nginx reverse proxied requests to get through.

With that warning given, let’s jump in to what we need to do:

Retrieve the correct certificate hash and Application ID

netsh http show sslcert

You’ll need to note the appid and the certificate hash for your ADFS 3.0 service.

Set the certificate as the default for HTTP.sys

We’ll use netsh‘s interactive mode, as I wasn’t in the mood to figure out how to escape curly brackets on Windows’ command line!

You want the curly brackets literally around the appid, but not the certhash.

netsh
netsh> http
netsh http> add sslcert ipport=0.0.0.0:443 appid={appid-from-earlier} certhash=certhash-from-earlier

Verify the proxy_pass settings

Among other configuration parameters, we have the following in our Nginx server stanza for this service:

proxy_redirect off;
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_set_header X-MS-Proxy the-nginx-machine;
proxy_set_header Host the-hostname-in-question

And, with that, we were successfully reverse proxying ADFS 3.0 with Nginx. 🙂

The Pushover Simple Notification Service

I’m a huge fan of the Pushover Simple Notification Service for receiving critical alerts about the servers for which I am responsible. It’s beautifully simple — in short, it takes the ‘walled garden’ of Google or Apple’s push notification system and extends it, so that you can push any text notification you want through their API, and it’ll get to your phone.

In this post, I will share how I get critical alerts from a Microsoft System Center Operations Manager instance to Pushover.

» Read the rest of this post…

XDebug to the rescue…

My friend Niall Brady dropped me an email, saying that some of the users of his Windows-Noob forums were reporting getting redirected to a spammy-looking site (url4short dot info) when clicking on search results to the site.

The forums run the Invision Power Board (IP.Board) software. There had been some reports of vBulletin boards being hit with this kind of spammy redirect, but fewer suggestions that this was an IPB problem. There had been a patch for a critical IPB issue released in December, but that had, obviously, been applied to the site as part of normal good practice.

Nevertheless, I was concerned. Clicking on a search engine result should definitely not be redirect somewhere other than the result page!

Without evidence that the issue was not limited to one machine, or one connection, however, it could not be ruled out that it was just malware on that visitor’s machine.

» Read the rest of this post…

Three years ago, I made the slightly crazy decision to run this website from my own server. This page is brought to you by a four-year-old generic PC that sits under my desk and dutifully hands out the web pages of my site to anyone from anywhere on the internet that asks for them.

Over the last three years, running my own server has taught me a lot. It has given me complete freedom and control, as well as complete responsibility over my own website. The hardware, software and configuration are all my own thing — if I get it wrong, I have to fix it.

» Read the rest of this post…

Due to some work that is happening to our house (hot water here we come), the electricity supply will need to be turned off at some point. This means that there will be some downtime for this server during the period of Monday 11th – Wednesday 13th.

Unfortunately, I will not actually be here at the time and it is not yet clear when or for how long the downtime needs to occur. It should, however, be a relatively short downtime period; I am not expecting the server to be down the whole three days.

In any case, the machine will be safely powered down before the supply is turned off and will come back up as soon as possible.

UPDATE: In the end, it didn’t need to go down at all! Everything is still running smoothly.

One year ago today I announced that I would be moving my site here over to a new hosting provider. Me.

Therefore, today marks the one year anniversary of this hosting arrangement! I’m very happy I’ve had an opportunity to do this. I have learnt a lot about running a server and all the different things involved with it and doing everything myself gave me complete freedom and control over every aspect of my personal site.

There have been issues along the way, but most of those minor. A brief security scare (which I now am pretty sure was nothing, but it was all reinstalled just to be sure), a hard drive failure and one occasion where my database backups failed for a couple of months without me noticing, to mention a few!

Overall though, I’m really glad I did take the step to self-host. It is undoubtedly more work, but it is rewarding as you get complete control and I have gained very useful experience of doing things on the server end with Linux (or Unix in general).

Sadly, it’s not 365 days uptime, but as of writing the server reports “up 49 days, 3:41” which I think is pretty good.

Here’s to the server. Hopefully much more service from it to come.

Also before I go, I’d like to thank everyone back last year who updated their links over to the new domain at the time. Your prompt action in taking that step allowed me to climb back up Google with formidable pace!

My server (and subsequently this site) is going to be down tomorrow between 10:00 AM and 1:00 PM GMT, for scheduled maintenance. Basically a clear out the cobwebs (which is scarily literal) job, and the opportunity to double-check that everything hardware-wise is hunky-dory. Also I’ll be doing a couple of special backups so I have even higher levels of redundancy, while I get the opportunity of the server being down.

See you on the other side.

UPDATE: Completed successfully, and I didn’t break anything.