Let's find a single XUL test that has an intermittent failure to migrate to HTML in order to make sure the new test file gets tagged up properly in treeherder/bugzilla. From an earlier discussion about this:

Treeherder very crudely extracts search terms from each failure line [1] and then uses those to search against the bug summaries of bugs with the keyword intermittent-failure. If the rename affects the string that appears as the second to last token in the pipe-symbol delimited string (ie TEST-UNEXPECTED-FAIL | <test name> | <some message that isn't used for search term>), then it will affect the bug suggestions.

To prevent this, the bug summaries would need to be renamed, however older branches (eg mozilla-beta) will still be using the old test filename, so if preserving suggestions there was considered necessary (I'd ask the sheriffs), the bug summary will need to contain both the old and new test name, at least until the rename works it's way through the trains.

[1] https://github.com/mozilla/treeherder/blob/9778665a81d65b8eac0f5fd427a6059eece5afa0/treeherder/model/error_summary.py#L103-L160

This is the intermittent bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1410000

From https://mozilla.logbot.info/treeherder/20190411#c16203988 it sounds like we can update the summary of https://bugzilla.mozilla.org/show_bug.cgi?id=1410000 from:

Intermittent toolkit/content/tests/chrome/test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

To something like:

Intermittent toolkit/content/tests/chrome/test_largemenu.html or toolkit/content/tests/chrome/test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

which would allow it to pick up intermittents from either m-c or beta. A few notes:

1. It could be done manually here to confirm it works
2. There is a 255 character limit on the summary which we wouldn't hit in this case but could hit with other files. Wes, would it be OK to drop the full file path from one of the filenames if that were to happen? Like "Intermittent toolkit/content/tests/chrome/test_largemenu.html test_largemenu.xul" or something?

Wes, would it be OK to drop the full file path from one of the filenames if that were to happen? Like "Intermittent toolkit/content/tests/chrome/test_largemenu.html test_largemenu.xul" or something?

It only needs to match the file name, and it's a substring search, so it can be in a comma separated list, and there doesn't even need to be spaces separating them. The paths can be omitted completely. You should be able to just list the old and new names.

In your example, it could be Intermittent test_largemenu.html,test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

https://hg.mozilla.org/mozilla-central/rev/2fb7a78d5bb5

Seeing this show up in https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-04-11&endday=2019-04-18&tree=all&bug=1410000 / https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=240968363&repo=autoland&lineNumber=14703 so everything seems to be working as expected.