I've experimented with the attached patch, to see what's missing for this approach.

(1)
We still have several undefined references to symbols from the gcrypt and gpg-error libraries,
so there's more work left to be done to make the shim complete.

(2)
The current shim makes a shortcut regarding the secure allocation functions. We must not simply map the allocation functions to standard malloc/free, but it must be changed to use the more secure versions from NSS that properly erases key material from memory.

(3)
The use of NSS low level crypto functions like AES_Encrypt needs to be modified.
NSS doesn't make those functions directly available from the shared libraries, instead, they must be accessed indirectly through function pointers, that are obtained using the freebl export function FREEBL_GetVector, which returns a vector with function pointers to all exported functions.

(4)
The mpi functions (arbitrary precision integers) aren't exported by NSS at all, they're only available from inside the NSS freebl library.
We'd have to identify a way how to use them. Because the code is contained in the Mozilla source tree, we could potentially compile them locally as part of the shim, and link them statically into the otr library.
Or we could ask the NSS team to export them in some way.

It's possible there's a more complete patch here,
https://github.com/arlolra/ctypes-otr/commits/master/patches/gcrypt-shim.cpp

but I don't remember where things stood. Just FYI

After many more changes, I'm able to link and run, and the JS code loads the OTR lib.
However, we're running into an assertion immediately.

Apparently the shim code assumes that the gcrypt-mpi and the nss-freebl-mpi code are compatible.
The attached patch uses this shim code:

gcry_error_t gcry_mpi_scan(gcry_mpi_t *ret_mpi, enum gcry_mpi_format format,
const void *buffer, size_t buflen,
size_t *nscanned) {
mp_err err = 0;
err = mp_init((mp_int *)*ret_mpi);
...

It uses a typecast from the gcrypt type "gcry_mpi_t" to the NSS-freebl type "mp_int",
but it seems the types are different:

struct gcry_mpi
{
int alloced; /* Array size (# of allocated limbs). /
int nlimbs; / Number of valid limbs. /
int sign; / Indicates a negative number and is also used
for opaque MPIs to store the length. /
unsigned int flags; / Bit 0: Array to be allocated in secure memory space./
/ Bit 2: The limb is a pointer to some m_alloced data./
/ Bit 4: Immutable MPI - the MPI may not be modified. /
/ Bit 5: Constant MPI - the MPI will not be freed. */
mpi_limb_t d; / Array with the limbs */
};

vs.

typedef struct {
mp_sign sign; /* sign of this quantity /
mp_size alloc; / how many digits allocated /
mp_size used; / how many digits used */
mp_digit dp; / the digits themselves */
} mp_int;

The assertion was triggered by a NULL pointer given to mp_init, which it doesn't allow, so besides the type differences, the function call semantics are different, too.

Some NSS changes that were necessary in the Mozilla tree to make the above comm changes build.

This comment repeats comment 15, using markdown. (IMHO, the only bugzilla formatting, which simply kept whitespace indenting with a monospace font, was much simpler to use. From now on I'll have to worry about formatting every time I write a commnent, instead of simply focusing on the contents?)

After many more changes, I'm able to link and run, and the JS code loads the OTR lib.
However, we're running into an assertion immediately.

Apparently the shim code assumes that the gcrypt-mpi and the nss-freebl-mpi code are compatible.
The attached patch uses this shim code:

gcry_error_t gcry_mpi_scan(gcry_mpi_t *ret_mpi, enum gcry_mpi_format format,
                           const void *buffer, size_t buflen,
                           size_t *nscanned) {
  mp_err err = 0;
  err = mp_init((mp_int *)*ret_mpi);
  ...

It uses a typecast from the gcrypt type gcry_mpi_t to the NSS-freebl type mp_int,
but it seems the types are different:

struct gcry_mpi
{
  int alloced;         /* Array size (# of allocated limbs). */
  int nlimbs;          /* Number of valid limbs. */
  int sign;	       /* Indicates a negative number and is also used
		          for opaque MPIs to store the length.  */
  unsigned int flags; /* Bit 0: Array to be allocated in secure memory space.*/
                      /* Bit 2: The limb is a pointer to some m_alloced data.*/
                      /* Bit 4: Immutable MPI - the MPI may not be modified.  */
                      /* Bit 5: Constant MPI - the MPI will not be freed.  */
  mpi_limb_t *d;      /* Array with the limbs */
};

vs.

typedef struct {
    mp_sign sign;  /* sign of this quantity      */
    mp_size alloc; /* how many digits allocated  */
    mp_size used;  /* how many digits used       */
    mp_digit *dp;  /* the digits themselves      */
} mp_int;

The assertion was triggered by a NULL pointer given to mp_init, which it doesn't allow, so besides the type differences, the function call semantics are different, too.