XDebug to the rescue…

My friend Niall Brady dropped me an email, saying that some of the users of his Windows-Noob forums were reporting getting redirected to a spammy-looking site (url4short dot info) when clicking on search results to the site.

The forums run the Invision Power Board (IP.Board) software. There had been some reports of vBulletin boards being hit with this kind of spammy redirect, but fewer suggestions that this was an IPB problem. There had been a patch for a critical IPB issue released in December, but that had, obviously, been applied to the site as part of normal good practice.

Nevertheless, I was concerned. Clicking on a search engine result should definitely not be redirect somewhere other than the result page!

Without evidence that the issue was not limited to one machine, or one connection, however, it could not be ruled out that it was just malware on that visitor’s machine.

Searching for Evidence

When there is a suggestion that a site has been subject to some kind of malicious code injection, which was what the report hinted at, the first port of call is to search the filesystem for things that look suspicious.

In the case of PHP injection, the usual suspects are odd-looking calls to base64_decode() and/or eval(). (There are plenty of legitimate uses of those functions, but, in combination with obviously obfuscated code, they should arouse suspicion.)

So, I logged on and started hunting through the website files for these strings, using grep:

grep -r base64 /where/the/websites/live/ | less
grep -r eval /where/the/websites/live/ | less

These commands produced some results, but… nothing that looked like it wasn’t obviously a legitimate part of IP.Board, which does make use of these PHP functions!

This was a little frustrating. It pointed to a much more sophisticated attack, or, at least, an attack that was more sophisticated at hiding itself.

The Code, or the Database?

IP.Board makes some interesting (and I would say unorthodox) use of file caching in the database. This meant that possibly, the injection had occurred inside the database, so a search of the filesystem would not find it.

So, I dumped the database:

mysqldump -u some_user -p some_database > database.sql

and then ran similar grep commands there, too.

grep -i base64 database.sql

Nothing odd!

Alternative base64 Hunting

My next thoughts were that perhaps this code was obfuscated and/or encoded, but that somehow it managed not to use the base64_decode() and eval() functions directly. Entirely possible.

So, I set about trying to search for base64 encoded code in a different way.

One hallmark that suggests such encoded information is the presence of two or more equals signs (searching for one would bring too many false positives!), which are ‘padding’ for when the input doesn’t fit into base64’s fixed block size:

If there was only one significant input byte, only the first two base64 digits are picked, and if there were two significant input bytes, the first three base64 digits are picked. ‘=’ characters might be added to make the last block contain four base64 characters.

grep -r == /where/the/websites/live/

Nothing but false positives there, but perhaps the encoded data had a deliberately perfect input size to avoid this tell-tale sign!

“Works for Me”

One of the issues so far, is that I had’t yet been able to reproduce the spammy redirect for myself. It happened only some of the time, and even though I cleared cookies and cache, and even swapped internet connections to use a fresh IP address, I couldn’t seem to get the spammy redirect code delivered to me. A very sophisticated attack that was able to hide from known administrators?

So, I was stuck, not being able to search for anything related to the output of the injected PHP — I hadn’t seen it! I was still stabbing in the dark looking for things that were obviously suspect, but I hadn’t found any.

Then, a breakthrough. Niall had the bad redirect happen to him, for the first time.

A little while later, and he had been able to reproduce it several times, and provide a Wireshark packet dump of the whole process, start to finish — from search engine result page to the redirect, and to the spammy page itself.

Analysing the Output

Armed with this new packet dump, I could now see actually what was happening to users, although not yet the code that was causing this to happen. But we were somewhere!

An additional JavaScript file was being referenced on the generated HTML page itself, which went to /forums/index.php?ipbv=[normal caching string]&g=js. That request itself looks (almost) quite normal, and something that is part of IPB.

However, the server’s response to this request for the extra JavaScript file was most definitely the thing we were looking for:

Yep, document.location = spammy spammy spam is asking the browser to immediately redirect to our not-so-friendly spam site!

This was, oddly, reassuring to me.

Yes, we knew that it was the site that was compromised with this code, which is deeply uncomfortable and not what we ever wanted to happen, but at least we did know for sure what was wrong.

I could now reproduce it by running:

curl -b "session_id=some_session_id" -A "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)" -e "http://www.windows-noob.com/forums/index.php" "http://www.windows-noob.com/forums/index.php?ipbv=[caching string]&g=js"

Getting a grep

So, back to grep it was. Still no obfuscated code in sight, I tried searching for what I now knew the output looked like, hoping that the code was unobfuscated, and that is why I had missed it.

grep -ri document.location /where/the/websites/live/

Nothing suspicious.

grep -ri "\$_GET['g']" /where/the/websites/live/

Nope.

I tried a few more searches, but to no avail. A different tactic was needed.

Unleashing the Debuggers

So, I could now reproduce the issue — if I supplied the right session cookie and requested that special g=js URL, I got the spammy redirect spat back at me. Perhaps I could use this to start tracing where this code was executing.

This was an issue on a live website, yes, but what I needed was a debugging tool that would normally be used in site development.

Fun with register_shutdown_function()

My first thought now was that this code must be the last, or nearly the last, thing that runs when I request this g=js page. So, there must be a way to trace what the last thing PHP was doing in that request.

So, I hooked up a shutdown function in one of the files IPB loads early in the process:

[php]function peters_shutdown_function() {
echo debug_backtrace();
}
register_shutdown_function( ‘peters_shutdown_function’ );
[/php]

This did print out a backtrace when I requested the g=js URL again, showing me what had immediately run after the JS redirect, but again, there was nothing suspicious. It clearly wasn’t able to trace back far enough, or trace in the right file.

I needed to turn to more heavy-duty tools.

XDebug to the Rescue!

XDebug is a PHP extension that vastly enhances the debugging capabilities of the runtime. It is very useful for providing more detailed error messages during development, or tracing through a program as it is running to see what is actually going on.

So, I installed and enabled it, and requested the same JavaScript page again, this time, enabling an XDebug trace file to be created.

This trace file would show me every function call that happened in order to generate our spammy JavaScript link page. That must reveal where the injected code is that is doing the deed!

File in hand, I ran the trusty old base64_decode search against it.

BINGO!

So, it was base64_encoded, and it was using eval() to execute itself once unpacked. It was just, until now, well hidden!

More importantly, it shows us that the code was injected into /forums/cache/skin_cache/cacheid_1/skin_global.php, on line 814.

I opened up that file, immediately found the injected code, and disabled it. No-one could then reproduce the spammy redirect issue. Win.

How Did it Hide?

The code, as obfuscated, lived in the cache file in this form (I commented it out as of taking this screenshot, but it was not commented out when the attack was ‘live’):

The $k variable was actually a ‘key’ that the attacker could use to come back, send it to the script and, oh, and run any code they wanted. Not good.

The rest of it was the encoded form of what we saw in the XDebug trace.

It seemed to be able to hide by not ever using base64_decode() or eval() directly. Instead, there is clever use of regular expressions to somehow unpack the encoded malicious stuff.

How Did It Get In?

Another important question is how the injection was possible in the first place. A week later, and after much further analysis of a number of sources of information, it cannot be absolutely determined with the information I have available to me.

I can tell you what I think is most likely.

The attack likely happened around Christmas 2012, when a number of other forum sites were hit, exploiting either VB.SEO plugin vulnerabilities in the case of vBulletin, or the critical vulnerability in IP.Board in our case.

It seems most likely that the site was compromised, and this PHP was injected, in the 24-hour latency between the patch being made available and it being applied, or perhaps before the patch was available (zero-day).

This should underline, if it is not already obvious, the important of prompt application of security updates.

Niall and myself have taken measures to ensure either of us can respond even faster to a critical security fix such as this one, and apply it quicker than 24 hours after release, to minimise that window. That would have closed the hole that likely allowed this in.

There is no evidence of the attacker being able to privilege escalate to do anything else on the system, and no evidence that anything else was done other than the injection of the redirect. To the greatest degree that it is possible to know, nothing else bad was done to the system. It continues to be monitored closely.

Moving On

It is uncomfortable and regrettable when any site gets compromised, but it is a fact of life that software has vulnerabilities, some known and some as yet unknown. Software that is exposed to the world via the internet therefore can be exploited, if the bad guy knows about a vulnerability that the good guys don’t.

Twenty-four hours of latency is plenty enough for the attacker, and everyone is a target, because exploits can be automated to require no human input.

On the positive side, I learned much about detecting and removing such exploits through this process. When things do go badly in one place, you get an opportunity to learn how attacks work, where their weaknesses are, and what security measures you can improve across the whole gamut of computer systems over which you have influence.

How Do I Get Rid of It?

On your IP.Board web server, try:

grep ‐ri \$mds /wherever/your/website/folder/is

to locate the injected code, and while file it resides in. You can then go into that file and remove it.

($mds is the unique name of one of the variables this particular variant used. You might also want to try searching for \$jsa, \$jsb, or for odd uses of preg_replace combined with strtr.)

Also try re-caching all the skins and languages in the Admin Control Panel. Make sure all IP.Board updates and patches are applied to prevent the compromise happening again.

Reset your passwords and keys. Take measures to detect and continue detecting other infiltrations.

---

Quite an adventure.