Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=330128575&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WWxlfUJARPiHdkq3CSeVhg/runs/0/artifacts/public/logs/live_backing.log

[task 2021-02-16T17:43:46.004Z] 17:43:46     INFO - TEST-START | devtools/client/framework/test/browser_toolbox_options_disable_js.js
[task 2021-02-16T17:43:46.659Z] 17:43:46     INFO - TEST-INFO | started process screentopng
[task 2021-02-16T17:43:46.819Z] 17:43:46     INFO - TEST-INFO | screentopng: exit 0
[task 2021-02-16T17:43:46.820Z] 17:43:46     INFO - Buffered messages logged at 17:43:46
[task 2021-02-16T17:43:46.821Z] 17:43:46     INFO - Entering test bound 
[task 2021-02-16T17:43:46.821Z] 17:43:46     INFO - Adding a new tab with URL: http://example.com/browser/devtools/client/framework/test/browser_toolbox_options_disable_js.html
[task 2021-02-16T17:43:46.822Z] 17:43:46     INFO - Tab added and finished loading
[task 2021-02-16T17:43:46.822Z] 17:43:46     INFO - Testing that JS is enabled
[task 2021-02-16T17:43:46.823Z] 17:43:46     INFO - TEST-PASS | devtools/client/framework/test/browser_toolbox_options_disable_js.js | Output is "JavaScript Enabled" - "JavaScript Enabled" == "JavaScript Enabled" - 
[task 2021-02-16T17:43:46.823Z] 17:43:46     INFO - Testing that JS is enabled in the iframe
[task 2021-02-16T17:43:46.824Z] 17:43:46     INFO - TEST-PASS | devtools/client/framework/test/browser_toolbox_options_disable_js.js | Output is "JavaScript Enabled" in iframe - "JavaScript Enabled" == "JavaScript Enabled" - 
[task 2021-02-16T17:43:46.824Z] 17:43:46     INFO - Clearing checkbox to re-enable JS
[task 2021-02-16T17:43:46.825Z] 17:43:46     INFO - Buffered messages finished
[task 2021-02-16T17:43:46.825Z] 17:43:46     INFO - TEST-UNEXPECTED-FAIL | devtools/client/framework/test/browser_toolbox_options_disable_js.js | targetList.isJavascriptEnabled is correct before the toggle - Got true, expected false
[task 2021-02-16T17:43:46.825Z] 17:43:46     INFO - Stack trace:
[task 2021-02-16T17:43:46.826Z] 17:43:46     INFO - chrome://mochikit/content/browser-test.js:test_is:1351
[task 2021-02-16T17:43:46.826Z] 17:43:46     INFO - chrome://mochitests/content/browser/devtools/client/framework/test/browser_toolbox_options_disable_js.js:toggleJS:82
[task 2021-02-16T17:43:46.826Z] 17:43:46     INFO - chrome://mochitests/content/browser/devtools/client/framework/test/browser_toolbox_options_disable_js.js:null:20
[task 2021-02-16T17:43:46.827Z] 17:43:46     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1088
[task 2021-02-16T17:43:46.827Z] 17:43:46     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1128
[task 2021-02-16T17:43:46.827Z] 17:43:46     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:948
[task 2021-02-16T17:43:46.828Z] 17:43:46     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:1037
[task 2021-02-16T17:44:31.064Z] 17:44:31     INFO - Not taking screenshot here: see the one that was previously logged
[task 2021-02-16T17:44:31.065Z] 17:44:31     INFO - TEST-UNEXPECTED-FAIL | devtools/client/framework/test/browser_toolbox_options_disable_js.js | Test timed out - 
[task 2021-02-16T17:44:31.154Z] 17:44:31     INFO - Removing tab.
[task 2021-02-16T17:44:31.155Z] 17:44:31     INFO - Waiting for event: 'TabClose' on [object XULElement].
[task 2021-02-16T17:44:31.159Z] 17:44:31     INFO - Got event: 'TabClose' on [object XULElement].
[task 2021-02-16T17:44:31.176Z] 17:44:31     INFO - Tab removed and finished closing
[task 2021-02-16T17:44:31.180Z] 17:44:31     INFO - TEST-PASS | devtools/client/framework/test/browser_toolbox_options_disable_js.js | The main process DevToolsServer has no pending connection when the test ends - 
[task 2021-02-16T17:44:31.195Z] 17:44:31     INFO - GECKO(2725) | MEMORY STAT | vsize 3815MB | residentFast 430MB | heapAllocated 129MB
[task 2021-02-16T17:44:31.196Z] 17:44:31     INFO - TEST-OK | devtools/client/framework/test/browser_toolbox_options_disable_js.js | took 45193ms
[task 2021-02-16T17:44:31.218Z] 17:44:31     INFO - checking window state```

The toolbox is supposed to initialize the "javascriptEnabled" setting for the configuration actor on open(), however we don't wait for the corresponding method to resolve. That's the same pattern we use for the various "apply*Settings" methods of the toolbox: https://searchfox.org/mozilla-central/rev/0379f315c75a2875d716b4f5e1a18bf27188f1e6/devtools/client/framework/toolbox.js#837-839

I believe that what happens here is that we build the UI of the options panel before the TargetConfigurationActor has received the "javascriptEnabled" info from the toolbox. Which means in the UI, the "Disable Javascript" checkbox will be checked. And we have no mechanism to update this checkbox dynamically (and this is very jarring if you Disable Javascript via the Debugger cog menu, the options become out of sync).

And later on when the test calls isJavascriptEnabled again, the ConfigurationActor has now received the javascriptEnabled setting and will complain because the checkbox state doesn't match the configuration.

One of the review comments that I didn't apply (partly because I forgot and partly because I felt it was not necessary) was to add a fallback mechanism in isJavascriptEnabled. Doing that should fix the intermittent.

https://hg.mozilla.org/mozilla-central/rev/a147bbc7e97a