8192+ bit RSA keys in OS X
Recently, I purchased my own domain for personal use and as file/project dump. Being a good security-conscious netizen, I immediately turned to StartSSL to get some fresh TLS certificates for this domain, so people wouldn’t receive those pesky self-signed security errors when attempting to browse my pages.
Seeing as I usually generate a seperate certificate key per domain, it was time to create a new one. Because I don’t really want to change certificate domain keys a lot, I decided to make this one future-proof by generating a 8192-bit RSA keypair for use in the certificate. All was well: my keys were accepted and certificates were generated by StartSSL just fine, the site seemed to work just fine in Firefox on OS X (my own setup) and in a few friends’ various browsers on other operating systems as well.
That is, until a few days later someone started complaining about receiving the following error on Chromium:
The same issue turned out to be present in Chrome, Safari and basically every browser that used OS X’s native crypto stack as well. Seeing as the given error messages were less than ideal (Safari just pretended I didn’t actually want to load a page), I decided to investigate by starting Chromium in debug log mode:
[7137:1287:1112/010412:VERBOSE1:one_click_signin_helper.cc(1105)] OneClickSigninHelper::NavigateToPendingEntry: url=http://shiz.me/
[7137:25603:1112/010412:VERBOSE1:resource_loader.cc(223)] OnReceivedRedirect: http://shiz.me/
[7137:21259:1112/010414:WARNING:cert_verify_proc_mac.cc(137)] Unknown error mapped to CERT_STATUS_INVALID: (-2147409643)
An unknown error. Wonderful. At least we have the native error code, as emitted by Security.framework. After doing some digging around I found the appropriate error message that belongs to error code -2147409643, or perhaps better expressed in its unsigned form as 74005:
❯❯❯ ./get_cssm_error -2147409643
error: CSSMERR_TP_INVALID_CERTIFICATE
Still not very informative. After this unsuccessful endeavour, I tried importing the key into my system keychain and see what happened: it gave me, again, a rather cryptic error code of -67762. Luckily, searching for that code on Google led me to a certain StackOverflow question:
Security update 2006-007 apparently broke 8192-bit certificates on OS X and no one bothered to fix it (https://discussions.apple.com/message/3650856#3650856). However there is (or rather used to be) a solution for this bug (or feature?) and it is in https://discussions.apple.com/thread/2668985 You’ve just got to execute
sudo defaults write /Library/Preferences/com.apple.crypto RSAMaxKeySize -int 8192in a Terminal prompt.
Sadly, like the question states, this does not work in Mavericks and upwards. This is because when Apple upgraded libsecurity_apple_csp, the library that takes care of the basic crypto services for Security.framework, from r55003 to r55471 for Mavericks, they also changed the location where the preference should be written to. Let’s take a look in /lib/RSA_DSA_keys.h:
/* Those max RSA sizes can be overridden with these system preferences */
#define kRSAKeySizePrefsDomain "com.apple.crypto"
/* Those max RSA sizes can be overridden with these system preferences */
#define kRSAKeySizePrefsDomain "com.apple.security"
A-ha! We have to write to /Library/Preferences/com.apple.security now! Simply changing the file target in the aforementioned command gives us the desired result:
To be fully future proof and to (hopefully) never have to deal with this again, I set the maximum key size to 32768 bits:
sudo defaults write /Library/Preferences/com.apple.security RSAMaxKeySize -int 32768
Now, of course, the obvious question is: why? Why would you artificially limit RSA key sizes? Admittedly fairly clueless on the subject, I decided to ask my good friend Aaron Jones:
<@Shiz> Hey, Aaron. Are there any real reasons for artificially limiting the maximum RSA key size a crypto library can process?
<&Aaron> Shiz: mitigate denial of service? it does get exponentially harder for the higher security levels
(to clarify, with ‘exponentially’ he means non-linearly, not exponential growth.)
This seems to be backed by a post on StackOverflow and the general maths behind RSA. According to RSA’s own FAQ, on page 75, “public key operations take O(k^2) steps, private key operations take O(k^3) steps, and key generation takes O(k^4) steps”, in a regular implementation, with k being the key size.
Indeed, Mozilla, up until last year, used to restrict the Diffie-Hellman key size to 2236 bits in the crypto library used by Firefox, NSS, in a similar fashion and for similar reasons. There’s a reason people say TLS takes a heavier load on the server, and it definitely makes it more susceptible to denial-of-service attacks.
However, as hardware gets faster and more efficient, we should also take care in keeping those maximum key sizes sensible for modern uses: seeing as the 2006-007 update instated the maximum key size in RSA initially, I’d say it’s about time for an update and would like to implore Apple to explore the options it has: 8192+ bit RSA keys are getting increasingly (if even slightly) common, and they will only start appearing more over the following years. Especially since, you know, a lot of users are on a desktop and needn’t fear getting (D)DoS’d over TLS any time soon.
TL;DR:
If you’re on Mountain Lion or lower:
sudo defaults write /Library/Preferences/com.apple.crypto RSAMaxKeySize -int 8192
If you’re on Mavericks or higher:
sudo defaults write /Library/Preferences/com.apple.security RSAMaxKeySize -int 8192
Special thanks go to @duane_moody for figuring out that the newest version of libsecurity_apple_csp lives in the Security project instead of being its own project like it was before on the Apple Open Source repository. Thanks to /u/pigeon768 for pointing out a mistake in Aaron’s quote.
