STR:

1. Open Library (Ctrl+Shift+B keyboard shortcut)
2. Go to some bookmark folder with nice amount of bookmarks
3. Play and move them some amount of time in various order
   after some time you will see that sometimes bookmarks are placed in wrong order like in attachment and you can't move them in place you want.

Nothing shows in Browser Console.

Known workarounds:

• restarting Library fixes it in most cases
• restarting Mozilla Firefox fixes it in all cases

Regression:
longstanding one, trying to find it for long time without results as issue isn't always reproducible

Places Database - Integrity results log:

Task: checkIntegrity

• The places.sqlite database is sane
• The favicons.sqlite database is sane

Task: invalidateCaches

• The caches have been invalidated

Task: checkCoherence

• The database is coherent

Task: expire

• Database cleaned up

Task: originFrecencyStats

• Recalculated origin frecency stats

Task: vacuum

• Initial database size is 71680KiB
• The database has been vacuumed
• Final database size is 71680KiB

Task: stats

• Places.sqlite size is 71680KiB
• Favicons.sqlite size is 52256KiB
• pragma_user_version is 52
• pragma_page_size is 32768
• pragma_cache_size is -2048
• pragma_journal_mode is wal
• pragma_synchronous is 1
• History can store a maximum of 128924 unique pages
• Table moz_places has 113959 records
• Table moz_historyvisits has 58684 records
• Table moz_inputhistory has 0 records
• Table moz_hosts has 0 records
• Table moz_bookmarks has 80644 records
• Table moz_bookmarks_deleted has 5 records
• Table moz_keywords has 0 records
• Table sqlite_sequence has 1 records
• Table moz_anno_attributes has 5 records
• Table moz_annos has 52574 records
• Table moz_items_annos has 33958 records
• Table sqlite_stat1 has 17 records
• Table moz_meta has 5 records
• Table moz_origins has 8233 records
• Index sqlite_autoindex_moz_inputhistory_1
• Index sqlite_autoindex_moz_hosts_1
• Index sqlite_autoindex_moz_bookmarks_deleted_1
• Index sqlite_autoindex_moz_keywords_1
• Index sqlite_autoindex_moz_anno_attributes_1
• Index sqlite_autoindex_moz_origins_1
• Index moz_places_url_hashindex
• Index moz_places_hostindex
• Index moz_places_visitcount
• Index moz_places_frecencyindex
• Index moz_places_lastvisitdateindex
• Index moz_places_guid_uniqueindex
• Index moz_historyvisits_placedateindex
• Index moz_historyvisits_fromindex
• Index moz_historyvisits_dateindex
• Index moz_bookmarks_itemindex
• Index moz_bookmarks_parentindex
• Index moz_bookmarks_itemlastmodifiedindex
• Index moz_bookmarks_guid_uniqueindex
• Index moz_keywords_placepostdata_uniqueindex
• Index moz_annos_placeattributeindex
• Index moz_items_annos_itemattributeindex
• Index moz_bookmarks_dateaddedindex
• Index moz_places_originidindex

Task: _refreshUI

I'm resetting the STR flags here, as the steps aren't good enough. We ideally need to know exactly how to get into this state.

Running the integrity check should reset all the indexes to be sequential (and this is also run in the background occasionally, so if a user gets into a bad state, after a day or so they should be "reset" again). It is worth restarting Firefox after doing that and then seeing if you can reproduce.

If you can then cause it to reproduce from a "good" state, then we need to know what actions caused it. Preferably with approximate indexes. For example, from my knowledge of the code I could guess that there may be something around dragging multiple bookmarks, but at the same time, I know the tests for that have reasonable coverage, so I don't think it is that.

It could also be Firefox Sync related, but that's just a guess.

I'm always manually verify integrity of places database daily by hand before I start long browsing session. Next I'm updating Mozilla Firefox Nightly and in end I restarting browser, so everything will be up-to-date.
Also doing manual verify integrity of places database when issue is starting occurring doesn't works as workaround.

This bug is reproducible for me and starting to occur either I move only one bookmark and more bookmarks.

I'm not using Firefox Sync, I disabled it in about:config.

I found that moved bookmarks are placed in incorrect and wrong order in Library, but when it first starting occurring and I close and reopen Library, everything is placed correct and proper order in Library.
So more I move and play with bookmarks after first bug symptoms and signs, more bookmarks will be placed in incorrect and wrong order due to this.

If you have more tips to me I would gladly do them.
Would catching profile with Gecko Profile helps? If so, with which settings should I do that?

Sorry for the delay here. I'll see if I can get a special build for you in the next few days which will allow logging of the actions and checking the moves.

No worries, this bug is kinda hard to reproduce, so it will also take some time for me too. :)

Also I want to add that this issue is also reproducible for new added bookmarks. Sometimes they aren't added to bottom of "Other Bookmarks" main folder, but some amount of bookmarks higher.

Have you been able to reproduce it yet, and did you try the extension I passed to you? (even if it hasn't got to the bad state, it'd be interesting to know if it shows that everything is ok currently).

(In reply to Mark Banner (:standard8) from comment #5)

Have you been able to reproduce it yet, and did you try the extension I
passed to you? (even if it hasn't got to the bad state, it'd be interesting
to know if it shows that everything is ok currently).

Yes to both, and I'm very sorry for the delay!

STR2 - newly added bookmark is placed in incorrect and wrong order in folder in Library

1. Start Mozilla Firefox
2. PlacesDebug 0.0.1 extension shows every bookmark folder as "good"
3. Open about:support
4. Press "Verify Integrity" button under "Places Database" section
5. Restart Mozilla Firefox
6. PlacesDebug 0.0.1 extension still shows every bookmark folder as "good"
7. Start long browsing session
8. Add many bookmarks (>100) in this long browsing session
9. Manage bookmarks (also some of newly added) by moving them by mouse to specific folders in Library
10. "wild 'issue' appeared!" - one of last added bookmark was placed in incorrect and wrong order in folder in Library
11. PlacesDebug 0.0.1 extension list whole bookmarks one by one from folder "Other Bookmarks" and their data,
    which I copied (only last part, because of length, I hope it's the most needed and useful information from there, if you will be all, I will try to redo test with folder with less bookmarks) and anonymized per privacy and security reason
12. All of next added bookmarks will be placed in incorrect and wrong order in folder in Library
13. Open about:support
14. Press "Verify Integrity" button under "Places Database" section
15. All of next added bookmarks will be placed in correct and good order in folder in Library (till next "wild 'issue' appeared!")

(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #6)

I will try to redo test with folder with less bookmarks

I done that. I moved all bookmarks from "Other Bookmarks" folder to another one. Next, I "Verify Integrity" of places database and restarted browser. I started long browsing session on helping at Bugzilla with music from YouTube in background. I managed some of bookmarks, which I added to "Other Bookmarks" folder in this browsing session and I encountered this issue again.

Result isn't anonymized this time.

Which method(s) are you using to create bookmarks? Is it just the Star UI, or are you using bookmark this link and related items?

I'm wondering, as the 1547102 has an earlier date added than the other three, probably an hour and a half apart. Do you recall which order they were added in?

Could the 1547102 be already somewhere else in your bookmarks and you added a duplicate?

I suspect these aren't the issues, but I'm asking anyway, as nothing much is standing out at me at the moment.

(In reply to Mark Banner (:standard8) from comment #8)

Which method(s) are you using to create bookmarks? Is it just the Star UI,
or are you using bookmark this link and related items?

I'm mainly using Star UI,
sometimes, but rarely, I'm using star from right mouse button context menu and Ctrl+D keyboard shortcut.

(In reply to Mark Banner (:standard8) from comment #8)

I'm wondering, as the 1547102 has an earlier date added than the other
three, probably an hour and a half apart. Do you recall which order they
were added in?

Explained in lower comment.

(In reply to Mark Banner (:standard8) from comment #8)

Could the 1547102 be already somewhere else in your bookmarks and you added
a duplicate?

No, no duplicates in this case,
but it could be bookmark which was removed and re-added later in same browsing session,
as sometimes I do that, when I want to see some of bookmarks in "Other Bookmarks",
because it's faster for me, to remove it and re-add it, than move in by "hand"/mouse in Library or by changing its folder in Star UI.

I also remember that this (STR2) could be in some way related to STR1,
as some of my bookmarks got outdated URLs and I added new bookmarks with updated URLs, like:

• open bookmarked http://bugzilla.mozilla.org/
• bookmark new redirected URL which is https://bugzilla.mozilla.org/home
• move new bookmark from "Other bookmarks" to bookmark folder with this old bookmark with obsolete URL

and sometimes this issue appear where newly added bookmarks are placed in incorrect and wrong order in folder in Library

Please could you try it with this build? Make sure you have the browser console open, and try and keep an eye on it. I've set it up to log each of the bookmark actions, and to verify the folder contents before and after.

We're really looking for the point where it goes from good to bad, I think (we shouldn't need the whole log).

It will only log guids/indexes/parents/dates, so no need to sanitise.

https://queue.taskcluster.net/v1/task/IsIpi60rQWu71iWYjwdOJQ/runs/0/artifacts/public/build/target.zip

Done, please have a look.

I enabled only "Logs" in Browser Console to avoid too much "spam".

STR:

1. Create new ~50 bookmarks.
2. Open about:support and press "Verify Integrity" button under "Places Database" section
3. Restart browser
4. Add new ~20 bookmarks
5. Open ~10 bookmarks from #1 and delete them
6. Readd them
   and see that they will be placed in incorrect and wrong order in folder in Library - in most cases before these ~10 bookmarks added in #5
   if not
7. Move recently added bookmarks
   and bug should be reproducible now

Thanks for that debugging, I've just traced this down to the removing action - we're not updating the positions correctly in the multiple delete at the same time case.

We're probably not seeing this reported more frequently because the integrity check that is running in the background will be fixing people's databases when it does occur.

Not sure why I can't always reproduce from the UI, but definitely seeing it in tests consistently, and I think I know what's up with the algorithm, so I'll go and fix it.

This was probably regressed by bug 1411891.

https://hg.mozilla.org/mozilla-central/rev/dd2f73926f7f

Is this something you wanted to nominate for Beta approval ahead of the next ESR or can this ride the trains?

I suspect that patch to fix this bug caused critical dataloss bug #1557853. So better wait some time and not land it now to Beta and ESR.

Unfortunately, I can still reproduce issue. I created follow-up which will be tracked in bug #1557853 with additional issue, so I'm marking this bug as VERIFIED.

(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #19)

I suspect that patch to fix this bug caused critical dataloss bug #1557853. So better wait some time and not land it now to Beta and ESR.

This patch is only changing indices, it is more likely that the original db conditions were a lot more complex than expected (not just wrong indices), or the UI is doing the wrong thing, thus when doing operations the ui doesn't reflect the db state.
anyway, I agree with the idea of not uplifting this, it is also our first usage of Sqlite window functions.