We could use a more flexible crypto lib to underpin our decrypt operations in the clearkey GMP. This is largely motivated by bug 1516673 where we wish to support further encryption options in a more ergonomic fashion.

With the need for CBC support, OpenAES has some sharp edges: while it does support a CBC mode, some of the implementation details of OpenAES make using it more painful than I'd like to use. For example, OpenAES expects buffers it decrypts to be also encrypted by OpenAES. You can see this in that it writes its own custom header into encrypted buffers. It also expects buffers to be in this format when decrypting them -- this means we'd have to write dummy heads into our CBC data to get decryption to work.

We currently use OpenAES's encrypt functionality to do decryption in ECB mode to implement our CTR encryption. This works, but we could reduce the work we're doing in clearkey by using a lib with better CTR support -- we wouldn't have to manually manage the counter, nor would we need to drop OpenAES's custom header from the buffers we get back.

In bug 1044742 a clearkey GMP was implemented. On that bug we can see that a number of crypto libs were considered:

• OpenSSL: not usable due to license concerns.
• Crypto++: not usable due to compiler issues (likely no longer applies).
• OpenAES: what we went with in the end.

There is no discussion of NSS on that bug, but I gather that due to difficulty in statically linking NSS at that time (as the GMP must be self contained), as well as concerns with NSS not working inside the GMP sandbox, it was not used.

With recent work to get NSS built as a static lib (bug 1543545) it may be worth considering trying this again. If that doesn't bear fruit we may wish to consider vendoring in Rust code or reconsidering Crypto++.

Clearkey previously relied on OpenAES to do its encryption. In order to
facilitate future changes and the need for CBC support, switch to NSS, which
should be more flexible and actively maintained.

Pushed by rvandermeulen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b5b8a0153dbe
Update Clearkey to use NSS for decryption instead of OpenAES r=bryce,jld

https://hg.mozilla.org/mozilla-central/rev/b5b8a0153dbe