Closed Bug 1499550 Opened 6 years ago Closed 4 years ago

Iterating store with cursor value + continue() is slow

Categories

(Core :: Storage: IndexedDB, enhancement, P3)

Product:
Core
Component:
Storage: IndexedDB
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Project Flags:
Performance Impact none

People

(Reporter: leplatrem, Unassigned)

References

Details

In IndexedDB docs [0], it is recommended to use a cursor in order to step through all the values in the object store. In Bug 1486980 we figured out that doing so was extremely slow. I isolated the usage of the high-level lib [1] that we use in RemoteSettings in order to observe the phenomena: https://codepen.io/anon/pen/QZaRBe Iterating on less than 1000 entries using a cursor takes around 1 second on my Firefox. It is globally 7 times faster in Chromium (Note: other IDB operations are faster too but out of scope here). Some explanation was given by :asuth in https://bugzilla.mozilla.org/show_bug.cgi?id=1486980#c32 tldr: Bug 1168606 could help and switching to getAll() might be the workaround for some use-cases. Having a bug that tracks this precise performance deficiency can help. Especially now that more and more browser features rely on IndexedDB we could face this core issue indirectly. It might also be a good idea to be more insisting about `getAll()` in the docs, as well as mentioning its downsides regarding I/O, its usage with the `count` parameter etc. [0] https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB#Using_a_cursor [1] https://github.com/Kinto/kinto.js/blob/master/src/adapters/IDB.js
Priority: -- → P3
Whiteboard: [qf]
As I understand it, bug 1168606 will fix this, right? If so, I'll qf- this, since we're tagging bug 1168606 with a qf priority.
Whiteboard: [qf] → [qf-]
Just double-checking on my understanding of the relationships here with asuth.
Flags: needinfo?(bugmail)
Yes, bug 1168606 will fix this.
Flags: needinfo?(bugmail)

Now that Bug 1168606 was fixed, I tried to run this snippet again, and performance between Chromium and Firefox is now comparable:

With Chromium:

sync: 224.454833984375 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
list all: 6.830810546875 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
get by id: 3.3232421875 ms iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1
get by attribute: 13.05712890625 ms

With Firefox:

sync: 585ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
list all: 4ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
get by id: 2ms - timer ended iframeConsoleRunner-7f4d47902dc785f30dedcac9c996b9f31d4dfcc33567cc48f0431bc918c2bf05.js:1:4080
get by attribute: 9ms - timer ended

I'm therefore closing this bug! :tada:!

For the reference, here is the snippet:

      const db = new Kinto.default({remote: "https://settings.prod.mozaws.net/v1/", bucket: "main"});
      const collection = db.collection("tippytop");

      const btn = document.getElementById("run");
      btn.onclick = async function () {
        btn.disabled = true;

        console.time("sync");
        await collection.sync();
        console.timeEnd("sync");

        console.time("list all");
        await collection.list();
        console.timeEnd("list all");

        console.time("get by id");
        await collection.list({filters: {id: "7210b4bc-81f6-4f04-976e-1acf5faeb740"}});
        console.timeEnd("get by id");

        console.time("get by attribute");
        await collection.list({filters: {domain: "amazon.es"}});
        console.timeEnd("get by attribute");

        btn.disabled = false;
      }
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Performance Impact: --- → -
Whiteboard: [qf-]
You need to log in before you can comment on or make changes to this bug.