A recent version of iTunes dropped support for downloading and syncing apps to iOS devices — the only methods for this are now using the App Store, Apple Configurator(?), or having some other over-the-air MDM solution push the apps to devices.

This caused me to run into a little bit of an issue in my day job — we’d been having sporadic issues doing over-the-air app updates on our corporate network, but when devices were taken off site, apps would update perfectly. I had been sort-of ignoring, sort-of working around this by downloading the apps to a desktop running iTunes and syncing (a small(!) number of) devices by Lightning cable.

But, now my workaround feature was gone! What was I to do? The story here is true, but some names and specific technical details have been omitted for professional privacy!

I was curious, so I did a packet dump or two and discovered the iOS devices were talking happily to the iTunes Store as you’d expect they should, but that when it came to actually downloading the app’s bits, they were contacting an IP address in a private range!

A couple of examples:

Request URI [truncated]: http://10.xx.xx.xx:50726/apple-assets-us-std-000001/Purple128/[snip]/pre-thinned[snip].thinned.signed.dpkg.ipa?accessKey=
[truncated]HEAD http://10.yy.yy.yy:55639/apple-assets-us-std-000001/Purple128/[snip]/pre-thinned[snip].thinned.signed.dpkg.ipa?accessKey=

Neither of those 10. addresses were ours — in fact, we were not using anything in those /16 ranges.

So what could be happening?

It turns out that a feature of the MacOS Server package is Caching Server, which is (I thought) a relatively simple, but highly specific proxy server for caching downloadable content from Apple that would normally be delivered by their CDN (i.e. apps, software updates, etc.).

In order to make it work, ahem, “seamlessly”, it seems that there’s an option when setting it up to register with a centralised Apple service the public IP address/range that the caching server’s clients will be emerging from and connect that to the private IP address of the Caching Server.

When an iOS device connects to a network, it (among other steps) contacts lcdn-locator.apple.com and receives information about where its local caching servers should be. It must have been the case that another organisation connected to our wider corporate network was using this Caching Server feature, and it had ‘registered’ one of the IP addresses that our internet traffic also emerges from (a proxy server IP address that we might share from time to time). This had the unintentional effect of instructing our iOS devices to contact their Caching Server, even when we are not the same site, and not on the same LAN.

Of course, those other 10. addresses are in the ‘wider’ corporate network, but were not actually our “on site” addresses, so they were not routable from our site, or were firewalled off, or whatever. The consequence was that this request to get the app’s bits would time out, and apparently there is no fallback to using the normal Apple CDN if the local CDN fails!

I worked around this internally by blocking lcdn-locator.apple.com at our on-site proxy servers, and suddenly apps updated and downloaded again — this time from http://iosapps.itunes.apple.com/, rather than a private IP address!

Two things seem wrong here:

• The assumption that emerging from a given public IP address means that a client has the ‘right’ to assert, for that IP address, a given private IP address of a Caching Server. Could this affect dynamic ISP clients at home if users had a Caching Server in one of these environments?
• Why doesn’t iOS fall back to using the normal Apple CDN if the specified local CDN times out?

It was an interesting exercise in troubleshooting, if nothing else!