Each rkv store has a 1MB size limit if not otherwise specified upon creation. Looks like we're using that default size for cert_storage, which could make the writes fail if the store grows beyond that limit.

We're looking to add the auto-resizing support for rkv. Before that lands, we can workaround it with following options:

• Estimate the store size and tell rkv when you open it (yes, you can change the size for the existing stores). Instead of Rkv::new(), you'd open it as follows,

  let builder = Rkv::environment_builder();
  builder.set_map_size(expected_size);
  let rkv = Rkv::from_env(path, &builder);
  

• Set the map size on demand, i.e. when a rkv write hits the "MapFull" error. We're going to add a new API for rkv (e.g. rkv.set_map_size()) to let the consumer resize the map if needed.

Looks like the first option is more reasonable for now, as the second one would be unnecessary once the auto-resizing is implemented in rkv.

:myk, what do you think?

The current disk page usage looks good on my nightly:

$ mdb_stat -afer security_state

Environment Info
  Map address: 0x0
  Map size: 1048576
  Page size: 4096
  Max pages: 256
  Number of pages used: 67
  Last transaction ID: 872
  Max readers: 126
  Number of readers used: 7
Reader Table Status
    pid     thread     txnid
       612 7fff8a1cb380 -
       612 70000349a000 -
Freelist Status
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 2
  Free pages: 10
Status of Main DB
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 1
Status of cert_storage
  Tree depth: 3
  Branch pages: 4
  Leaf pages: 49
  Overflow pages: 0
  Entries: 871

Eventually we're going to want to store a few hundred certificates using cert_storage (they're a few kB each) - presumably that could get near or exceed the current limit, right?

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #2)

Eventually we're going to want to store a few hundred certificates using cert_storage (they're a few kB each) - presumably that could get near or exceed the current limit, right?

It has consumed about 25% of disk space to accommodate the 871 entries on my machine, I'd assume the current setup can handle 3000 - 3500 entries if the new certificates have the similar size.

LMDB usually recommends leaving enough space for the potential increase (e.g. 150% of your estimate). It won't occupy any disk space (more like a placeholder) until the data gets inserted.

Again, this complexity should be gone once rkv supports auto-resizing.

(In reply to Nan Jiang [:nanj] from comment #0)

Looks like the first option is more reasonable for now, as the second one would be unnecessary once the auto-resizing is implemented in rkv.

Over in bug 1538093, I'm adding support for cert_storage to re-open its environment in read-only mode after initial creation and then re-open it in read-write mode temporarily when making changes. After that lands, it should be reasonably straightforward to re-open the environment with a larger map size and re-try a write if it initially results in a "MapFull" error.

Whether it makes sense to do that depends on how soon we expect to land auto-resizing, however. If auto-resizing is going to happen soon, then I agree that the first option is more reasonable. Nan, do you have a sense of how long it'll take to complete the auto-resizing work?

(In reply to Nan Jiang [:nanj] from comment #3)

LMDB usually recommends leaving enough space for the potential increase (e.g. 150% of your estimate). It won't occupy any disk space (more like a placeholder) until the data gets inserted.

That's true on Mac/Linux, but on Windows it creates a file at the map size, so we should take that into account when choosing the map size for an environment. See https://github.com/mozilla/lmdb-rs/issues/40 for more info on why it does this and what we could do to grow the file size incrementally on Windows.

(In reply to Myk Melez [:myk] [@mykmelez] from comment #4)

Whether it makes sense to do that depends on how soon we expect to land auto-resizing, however. If auto-resizing is going to happen soon, then I agree that the first option is more reasonable. Nan, do you have a sense of how long it'll take to complete the auto-resizing work?

Yep, I am on it. There is a WIP patch here: https://github.com/mozilla/rkv/pull/132

(In reply to Nan Jiang [:nanj] from comment #3)
That's true on Mac/Linux, but on Windows it creates a file at the map size, so we should take that into account when choosing the map size for an environment. See https://github.com/mozilla/lmdb-rs/issues/40 for more info on why it does this and we could do to grow the file size incrementally on Windows.

Aha, it behaves differently on Windows. Thanks for the tip! ;)

But 1530545 set the default size to 16MB (downloading all of the intermediates takes about 6.5MB, so 2x capacity would be 13MB, and I like powers of 2), so I think we're good for now (automatic growing would be nice, though).

FWIW, we've added various metadata APIs to rkv that could be used to better understand the disk usage and then resize the database if needed.

In particular, the Environment::load_ratio() and Environemnt::set_map_size(), which allow the consumers to resize the database before it hits the MAP_FULL error.