Skip to content


Reverse Proxying ADFS with Nginx

In my recent trials and tribulations with ADFS 3.0, I came up against an issue where we were unable to host ADFS 3.0 with Nginx as one of the layers of reverse proxy (the closest layer to ADFS).

When a direct connection, or a cURL request, was made to the ADFS 3.0 endpoints from the machine running Nginx, all seemed well, but as soon as you actually tried to ferry requests through a proxy_pass statement, users were greeted with HTTP 502 or 503 errors.

The machine running ADFS was offering up no other web services — there was no IIS instance running, or anything like that. It had been configured correctly with a valid TLS certificate for the domain that was trusted by the certificate store on the Nginx machine.

It turns out that despite being the only HTTPS service offered on that machine through HTTP.sys, you need to explicitly configure which certificate to present by default. Apparently, requests that come via Nginx proxy_pass are missing something (the SNI negotiation?) that allows HTTP.sys to choose the correct certificate to present.

So, if and only if you are sure that ADFS is the only HTTPS service you are serving up on the inner machine, you can force the correct certificate to be presented by default, which resolves this issue and allows the Nginx reverse proxied requests to get through.

With that warning given, let’s jump in to what we need to do:

Retrieve the correct certificate hash and Application ID

netsh http show sslcert

You’ll need to note the appid and the certificate hash for your ADFS 3.0 service.

Set the certificate as the default for HTTP.sys

We’ll use netsh‘s interactive mode, as I wasn’t in the mood to figure out how to escape curly brackets on Windows’ command line!

You want the curly brackets literally around the appid, but not the certhash.

netsh> http
netsh http> add sslcert ipport= appid={appid-from-earlier} certhash=certhash-from-earlier

Verify the proxy_pass settings

Among other configuration parameters, we have the following in our Nginx server stanza for this service:

proxy_redirect off;
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_set_header X-MS-Proxy the-nginx-machine;
proxy_set_header Host the-hostname-in-question

And, with that, we were successfully reverse proxying ADFS 3.0 with Nginx. 🙂

Forms-based ADFS 3.0 Endpoints Inexplicably Showing HTTP 503

Azure Active Directory logo

As with many other organisations, at my day job we are using the Office 365 service for email, contacts and calendars. There are a few ways to integrate 365 with your local Active Directory, and we have been using Active Directory Federation Services (ADFS) 3.0 for handling authentication: users don’t authenticate on an Office-branded page, but get redirected after entering their email addresses to enter their passwords on a page hosted at our organisation.

We also use the Azure AD Connect tool (formerly called Azure AD Sync, and called something else even before that) to sync the directory with the cloud, but this is only for syncing the directory information — we’re not functionally using password sync, which would allow people to authenticate at Microsoft’s end.

We recently experienced an issue where, suddenly, the endpoints for ADFS 3.0 that handle forms-based sign in (so, using a username and password, rather than Integrated Windows Authentication) were returning a HTTP 503 error. The day before, we had upgraded Azure AD Sync to the new Azure AD Connect, but our understanding was that this shouldn’t have a direct effect on ADFS.

On closer examination of the 503 issue, we would see errors such as this occurring in the AD FS logs:

There are no registered protocol handlers on path /adfs/ls/ to process the incoming request.

The way that the ADFS web service endpoints are exposed is through the HTTP.sys kernel-mode web serving component (yeah, it does sound rather crazy, doesn’t it) built into Windows.

One of the benefits of this rather odd approach is that multiple different HTTP serving applications (IIS, Web Application Proxy, etc.) can bind to the the same port and address, but be accessed via a URL prefix. It refers to these as ‘URL ACLs’.

To cut a very long story short, it emerged eventually that the URL ACLs that bind certain ADFS endpoints to HTTP.sys had become corrupted (perhaps in the process of uninstalling an even older version of Directory Sync). I’m not even sure they were corrupted in the purely technical sense of the word, but they certainly weren’t working right, as the error message above suggests!

Removing and re-adding the URL ACLs in HTTP.sys, granting permissions explicitly to the user account which is running the ‘Active Directory Federation Services’ Windows service allowed the endpoints to function again. Users would see our pretty login page again!

netsh http delete urlacl url=https://+:443/adfs/
netsh http add urlacl url=https://+:443/adfs/ user=DOMAINACCOUNT\thatisrunningadfs

We repeated this process for other endpoints that were not succeeding and restarted the Active Directory Federation Services service.

Hurrah! Users can log in to their email again without having to be on site!

This was quite an interesting problem that had me delving rather deeply into how Windows serves HTTP content!

One of the primary frustrations when addressing this issue was that a lot of the documentation and Q&A online is for the older release of ADFS, rather than for ADFS 3.0! I hope, therefore, that this post might help save some of that frustration for others who run into this problem.

Isn’t it funny that so frequently it comes back to “turn it off, and turn it back on again”? 🙂

Staying Safe

I have written on this subject before, but as suspected, surveillance is back on Parliament’s agenda again.

Is the Investigatory Powers Bill the latest attempt at a “modernising” of existing laws and conventions, as is often claimed, or an unprecedented extension of surveillance powers?

I would argue strongly that the capability for your local council, tax enforcement authorities, and the myriad of other agencies that are proposed to have access to this data, to ‘see’ every thought you might have dared to research online is vastly more than would have been possible in human history. It’s also vastly more than any other country has sought the legal power to access.

Photo by Luz on Flickr. Licensed under CC-BY.

Photo by Luz on Flickr. Licensed under CC-BY.

Given what we know in a post-Snowden era, this proposed legislation is quite clearly not about ensuring a continued intelligence flow for the purposes of national security. That has been going on behind closed doors, away from any democratic process and meaningful oversight, for many years, and will no doubt continue. Whether or not the activities of military intelligence agencies have a strong legal foundation has apparently not stopped them from gathering what they need to do their job. It is important for me to note that I don’t doubt the hard work they do, and the success they have had over the last ten years in preventing violence in the UK. However, we know that overreach and abuse have occurred — at the kind of scale that undermines the very values our government and their agencies are there to protect.

It is clear to me that, given the secret and ‘shady’ nature of much of the activities of the security apparatus of perhaps every nation state, what we do not need to do as a democratic society is provide a strong legal protection for such morally ambiguous acts. If a tactic is invasive or aggressive, but genuinely necessary in a “lesser of two evils” sense, the fact that the actor has to take on the liability for it provides an inherent safeguard. If it is easy and low risk to employ that tactic, there is a stronger temptation for its abuse, or for its inappropriate extension into everyday investigations. When these laws are ‘sold’ to the people as being for national security and to keep us safe from violence, it cannot be acceptable that the powers are made available to other agencies for any other purposes, as the Bill proposes.

A nation state does not have the right to violate the sanctity of the boundary of someone’s home without strong justification — a warrant. A nation state similarly does not have the right to violate that boundary in the form of bulk data collection on an entire populace. The Internet connections we open and the data we transfer is something that we can keep private from our government, unless due process is followed to override that on an individual basis.

That must remain. That principle must be protected, or we’ve forgotten why we bother with this ‘free country’ thing.

It must be protected even when we face short- and medium-term risks to our safety. Why? Because it is not hyperbole to say that failing to do so lays the technical and legal foundations of a police state, which is a much more significant long-term risk.

Fortunately, there are many fighting against this Bill, which (even if you disagree with my arguments above) is widely regarded to be completely unfit for purpose.

I wholeheartedly support the Don’t Spy on Us campaign and its six principles, and I stand with them as a supporter of the Open Rights Group, one of the organisations behind the campaign.

Apple Event Brain Dump

Some very raw and unfiltered thoughts on today’s Apple announcement:

I thought after all this time there’d be more content deals for the new Apple TV — the “apps” focus suggests that they are having to concede their former approach entirely and acknowledge that they won’t funnel much TV content through iTMS at all.

Harry Potter photos!

4K video capture on a phone is pretty amazing.

The MLB Apple TV app demo with watching two games at once must have been a Back to the Future II reference for 2015 — “give me channels 5, 9…”. Right? Right?

I’m interested to see (hopefully non-fanboy/girlish) thoughts on how the iPad Pro will compare with the Surface range. It’s interesting to me actually that if MS get the touchy style apps done well (they have to do a better job than with Win8!), the Surfaces also having the flexibility to run classic Windows apps too might make them more competitive in that “pro tablet” area.

I want to go play with 3D Touch when I can! If they’ve done it well, it could be quite cool.

I’m no artist in drawing terms, but the Pencil looked pretty amazing. I was deriding it at first as a silly stylus. I was wrong.

So. Much. Stuff!

Adventures in SELinux

Security Enhanced Linux (SELinux) is a pretty powerful technology that adds another layer of access control to a Linux system. It helps significantly limit the ability for an attacker who has partly compromised a system to use their access to jump deeper into the system.

It has been standard in Red Hat Enterprise Linux and its derivatives for quite some time, and is often the cause of many a headache when something doesn’t work because it is being (apparently) silently blocked by SELinux’s security enhancements!

Its potential to cause breakage, especially when third-party bits and pieces are brought into the mix, means the advice from well-meaning individuals is often a cry of “just turn off SELinux!”, rendering a system without that extra layer of protection.

I will not pretend that my recent dealings with SELinux in CentOS 7 have been free of frustration, but a few simple tools have made it a surprisingly simple affair to get something up and running again if a particular behaviour (always of something a bit third-party in my experience!) is being erroneously blocked.

I think a big part of what makes SELinux get switched off in frustration is the perception that it is breaking things silently, and the psychological impact of its verbose ‘scariness’ when you do find those logs!


As long as you remember that SELinux can be the cause of potential unexplained weirdness, your first port of call can be audit2why:

audit2why -a

What is particularly nice about this tool is how quickly you get (semi-)human readable output, detailing which rules an application is breaking. If you do hit one of these ‘weird problems’, a quick trip to this tool usually makes it clear that SELinux is the cause of the failure!


I was surprised by how relatively quick it was to identify an issue with audit2why and make a custom module with audit2allow to get an application working again.

There are a good set of instructions in the Red Hat manual.

It sounds like a big deal, but the tools have made it almost completely automated — it really isn’t necessary to have a deep understanding of SELinux’s internal workings.


Finally, there are some flags that are disabled by default for SELinux protected applications that you might need. Again, audit2why will often make it clear what you need to toggle using this tool!

For example, a web server process that does legitimately send out emails might need the appropriate flag switched on. Without it, the web server doesn’t get the right to talk to sendmail.

Give SELinux Another Chance!

I suppose my point in this rambling is this: give SELinux a chance if you have given up on it in the past. If you have the time to set up your system properly (and you should!), taking a little extra to figure out how to grant only the permissions really needed does make a material difference to your security should one application be compromised. An attacker being able to get their foot in the door needs to be assumed to be possible, so making their life a lot harder at that point is worth making your life slightly harder on the odd occasion.

With a little patience and the use of the tools I have talked about, I think it is a lot easier to work with than it might seem at first glance, or when it first arrived in RHEL many moons ago!

My Alternative Christmas Wish

Conflict minerals 5 by Sasha Lezhnev of the Enough Project -

Conflict minerals 5 by Sasha Lezhnev of the Enough Project

Licensed under CC-BY-ND 2.0

I am always interested to know where the products I buy come from, and at this particularly consumer-focused time of the year, it highlights the issue further. It is interesting to me just how complex the chains of dependencies involved in making any non-trivial ‘thing’ really are.

The computer I am writing this on was assembled in the UK, but I would suggest that most, or all, of its components were not. What about the suppliers who provided sub-components for those components? What about the raw materials, including the traces of rare earth minerals needed to do its electronic magic?

Unfortunately, the result of this enormous complexity, and the fact that the retailers from which we buy care about little but the price they pay, means it is very difficult to verify really important aspects of how the goods we consume were made. It seems that nobody, not even the retailers at this end of the chain, has the depth of insight into their supply chain needed to affirmatively say “this is how our product was made”.

So, when cost is the primary concern, and nobody really digs deep to understand what is happening at each stage of a product’s life, how can consumers at this side of the transaction be empowered to make more ethical choices?

If I go to my local big-name supermarket to buy a kettle, for example, I cannot look at all the options and make an assessment about which product was made in a way that best aligns with my values. Did the manufacture of the cheapest £5 kettle involve the exploitation of somebody? Probably. Is the £30 ‘luxury’ choice any better? We just don’t know.

There are pockets of hope in this area — initiatives like Fairtrade have, for some product categories, encouraged supermakets to go ‘all Fairtrade’ for particular items, and for other companies (tea and coffee businesses being a good example) to take steps to at least appear to be sourcing more ethically.

I just wish there were a big push from somewhere to gather accurate intelligence about how our stuff is made, and begin labelling more products in a way that empowers consumers to make better choices. I think there is a good portion of the population who want to make better choices that support human rights, environmental protection and social progress, but without high quality, verifiable information about what goes into what we consume, we are in the dark. While we remain uninformed, we cannot exert pressure on the market to do better as a whole.

Serving Pi

I recently completed a physical migration of my server, the one that hosts this very page! It all went successfully, and without any noticeable downtime for this site, which I am pleased to be able to do.

There was, however, a period of time during which this server needed to be physically switched off and moved to the new location. To enable zero downtime, something would need to be able to host the server during that period.

Enter my Raspberry Pi!

Raspberry Pi in box

This amazing little thing is capable of running Raspbian, a modified version of Debian, which means I get access to the rich library of Debian packages that are available. I have a private Git repository containing a modular set of Puppet manifests. These describe the exact configuration of this server, so by applying the Puppet manifests, I can spin up a new instance of this particular server’s configuration on a whim.

So, I dusted off an SD card that was lying around, dropped Raspbian on it, and installed Puppet and Git and applied the manifests.

If I’m honest, there were a few components that weren’t quite so happy to run, despite packages being available. Varnish didn’t seem to like my VCL file, so I had to run the site here directly with Nginx pointing to PHP-FPM instead.

To cut a long story short, it worked! I was successfully serving up this site, from the Pi, using (almost) my existing configuration. Performance was not stellar, even compared to the modest hardware that normally serves this page, with page load times about 10 times slower than uncached page loads normally would be. The main blog page did take 1.5 seconds to render! For the short time I needed it though, I was very happy to have a very inexpensive and easy solution.

The Changing Face of Vulnerability News

Heartbleed logo  Shellshock Logo

The recent news about the bash vulnerability being called “ShellShock”, and the degree to which it is getting mainstream press has got me thinking about how software vulnerabilities are now being reported in the mainstream media.

Apparently, no vulnerability these days is complete without a catchy name and logo — see Heartbleed and Shellshock! Joking aside, though, the very fact that these vulnerabilities are making non-tech news headlines puts pressure on everyone running potentially vulnerable systems to do their duty — usually as straightforward as running a pre-packaged security update.

The Heartbleed and Shellshock stories are taking the place of what we used to see reserved for particularly influental computer worms, like Sasser and Mydoom. It’s most definitely positive that some vulnerabilities are getting attention — unfortunately it is still the case that for some companies and system administrators, only outside pressure will convince them to promptly, diligently and consistently apply security updates.

What I’d like to see, is some way for people interested in improving computer security, the “good guys” for lack of a better term, to leverage this media interest to send a message to system administrators that it’s always necessary to apply software updates promptly, even when they don’t get on the TV news!

The Curse of The Black Box

The other key issue that Shellshock highlights, as did Heartbleed, is the issue of embedded ‘black box’ systems that might be vulnerable. This kind of system is everywhere — and because in many cases they are ‘set it and forget it’ machines, they represent a particular risk. It’s often very difficult to convince vendors of these systems of the importance of pushing upstream software updates down to end users, particularly when there is a lack of understanding and a lack of financial incentive.

Something big and mainstream, like Shellshock and Heartbleed, might convince system administrators to badger vendors to release patches for this kind of product, but we need to extend this further, and make it a social (or even a legal) expectation on vendors to supply security updates for any product they ship, for a reasonable lifetime period for that product.

The security landscape is too complex, and everything too interconnected, for anyone to have the opinion that “I don’t need to patch that, because there’s nothing important on it”.

Leaving Yourself in the Loop

I want to part with a few bullet points, with some actions I try to take to stay up-to-date. Automatic updates are increasingly common, but not universal, and these simple things can help you not miss a known vulnerability.

  • Document and understand the whole software footprint of the systems for which you are responsible. (This means embedded systems, software libraries, and more!)
  • Subscribe to announce mailing lists, follow Twitter accounts of the software projects and systems you use. (It pays to be in the know about available updates, and not hear about them after it is too late!)
  • Look for useful vulnerability resources for particular projects you use. (For example, for WordPress, the recently launched WPScan Vulnerability Database.)

Personalised Search: Technologically Induced Confirmation Bias

DuckDuckGo filter bubble site

I can’t unfortunately remember what led me to this page (I think a retweet from someone), but I found myself perusing DuckDuckGo’s marketing site “Escape your Search Engine’s Filter Bubble” recently.

(I don’t have a relationship or particularly strong opinion about DuckDuckGo at this time, by the way, so this is no marketing astroturf.)

It shows you just how search engines deliver different results for the same query, based on the user’s habits in the past.

The profile the search engine has built up on the user through their cookies doesn’t just inform them about relevant advertising, it literally changes the search results.

This troubles me greatly.

Now, I don’t believe I am experiencing this when I search. I am borderline obsessed with clearing cookies and other browsing data to ‘reset’ my browser to the same state after each session. Assuming mainstream search engines aren’t using technology like Evercookie, then I get a generic set of results across different browsing sessions.

Most people don’t do that, which means that most people are becoming increasingly unlikely to come across viewpoints that differ from their own on the web. They have a technologically induced confirmation bias, where, unless they click through a number of pages of search results, they will rarely hear people who might (respectfully, thoughtfully) disagree with them.

Confirmation bias… is a tendency of people to favour information that confirms their beliefs or hypotheses. People display this bias when they gather or remember information selectively, or when they interpret it in a biased way.

I am, frankly, frightened at the idea of people never being exposed to a diversity and plurality of opinions. I am frightened of how easy it could be to not develop and nurture empathy. The consequences of that could very well be more profound than we might realise on the surface.

As well as the societal implications, it doesn’t seem the right decision to me either in terms of the technical role a search engine should play. What I previously liked about the ‘old days’ of Google Search was their strong commitment to put the most relevant result first. I’m not sure, though, that delivering the most personally relevant result is the same as delivering the most relevant result for the query.

So how do we address this? Well, I think that media literacy in general is something we need to make a priority. Trying to change the way the search engines work when everything ‘targeted’ is such big business is unlikely to be successful.

At the very least, we need to get the message out to people that this is happeningprivacy might not be the only reason you might want to clear your cookies.

iOS 7 and Obsolescence

iPhone 4 with iOS 7

This is my iPhone 4. I purchased it more than three years ago.

You don’t get into the technology world without, begrudgingly or otherwise, accepting that things move very fast. What is relevant today may be completely superseded in a matter of months.

A big reason why I have ended up a user of Apple’s iOS ecosystem is that, unlike some of its competitors, there seems to be a genuine focus on the relationship with the customer after you have purchased the device. I can run this old iPhone 4, using the latest operating system that was released this month.

From a security point of view, upon which I can’t resist to comment, the pace of mobile OS development is such that security fixes are not routinely backported to older OSes. You end up with the situation we have today with Android — scores of vulnerable devices out there in the wild.

Aside from some frustrations I do have — the original iPad that was released in the same year as my iPhone 4 is now stuck back on iOS 5 — Apple actually seem to think about device lifespan the least cynically of all the manufacturers. When they were developing the iPhone 4, they clearly thought about how it would run the next three operating systems yet to come.

It can’t be denied that the iPhone 4 isn’t quite as quick and responsive with iOS 7 as it was when it shipped with iOS 4. It doesn’t enable all the fancy features of the new OS. What it is, though, is in line with the performance you would expect from a device that is a little older now. It is definitely acceptable, and probably even good.

This is why I make the purchasing decisions I do. As long as you avoid first generation products(!), you can make an investment in a piece of Apple kit. It is so much more than just a product to shift off the shelf.