Steps to reproduce this issue:

• open Firefox, any tab, any website
• open DevTools in this tab
• open the Browser Toolbox (note that the fission.autostart and the devtools.browsertoolbox.fission prefs do not need to be set, even though it looked like this problem was fission related at first)
• switch to the inspector panel in the Browser Toolbox and start the picker
• at this stage, the Browser Toolbox window is still in front, which is OK, make sure it doesn't fully cover the Firefox window in the back
• start hovering over elements in the Firefox window, e.g. in the browser toolbar
• notice how hovered elements get a red dashed border, which is good
• now move down towards the DevTools toolbox and start hovering over elements there too

=> The red dashed border does not immediately appear around elements inside the toolbox. This is a regression.

• now focus the browser window, this should fix it.

This might have been a regression from bug 1581925 which moved DevTools to be part of a <browser> element instead of an <iframe>.

Also, I don't exactly know when this occurs yet, but at some stage a JS error occurs too:

JavaScript error: resource://devtools/client/inspector/markup/markup.js, line 725: TypeError: this._highlightedNodeFront is undefined

The stack trace for this is:

console.trace: 
resource://devtools/client/inspector/markup/markup.js 723 _hideBoxModel
resource://devtools/client/inspector/markup/markup.js 684 _onMouseOut

So it looks like on mouse out, there are now cases where this._highlightedNodeFront was not set. This was not the case before.
This error seems easy to silence, and we should probably have done this from the start. It's just odd that it would only show up now. This might have been a regression from bug 1574401.

Florens provided a screencast on Slack which I'm attaching a short version of here.
It shows how the picker can't hover elements inside devtools until the browser window gains focus.

This might have been a regression from bug 1581925 which moved DevTools to be part of a <browser> element instead of an <iframe>.

I confirm this part (having to focus the window first) is a regression from using the <browser> element. If you switch back to using an <iframe> (type "content" or not), then the highlighter will work even without focus.

Ok, let's keep this bug for just this problem then. I'll file another one about the JS error mentioned in comment 0 (which I found to be related to bug 1568157).

Note that this also affects picking elements in content tabs with the Omniscient Browser Toolbox. Content is also hosted in <browser> elements, so it's probably the same issue.