Filed by: shindli [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=176894538&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/dgteqjIGTN6U7oXUC_XOkA/runs/0/artifacts/public/logs/live_backing.log [task 2018-05-04T05:05:53.963Z] 05:05:53 INFO - tests/web-platform/tests/tools/webdriver/webdriver/client.py:438: InvalidArgumentException [task 2018-05-04T05:05:53.963Z] 05:05:53 INFO - TEST-PASS | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_default [task 2018-05-04T05:05:53.964Z] 05:05:53 INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_twice - NoSuchAlertException: no such alert (404): No modal dialog is currently open [task 2018-05-04T05:05:53.964Z] 05:05:53 INFO - new_session = <function create_session at 0x7f9c02b03848> [task 2018-05-04T05:05:53.964Z] 05:05:53 INFO - add_browser_capabilites = <function update_capabilities at 0x7f9c02b03938>

I did some retriggers and backfills to find where this started and worked on this range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=linux%20x64%20wd%20pgo&tochange=f579a28444855e4ccbf45c8c82f9bb78ebbe822b&fromchange=afbec9a6802cb45f5032301b997c15f7d1c25884 The next shift will continue with the retriggers.

Continued the retriggers and the failure rate is 2 out of 20 in general, on the push where this started: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=linux%20x64%20wd%20pgo&tochange=355349b199c457d45e420d775df07f7b6e9df859&fromchange=5bde59ad4082e94f3deea06808152b3ad0077876&selectedJob=176170088 we have 20 failures out of 21.

Thanks Eliza- this is great work and shows the exact problem. I see that :aryx fixed the manifest in the next commit that landed: https://hg.mozilla.org/integration/mozilla-inbound/rev/355349b199c457d45e420d775df07f7b6e9df859

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3) > Thanks Eliza- this is great work and shows the exact problem. > > I see that :aryx fixed the manifest in the next commit that landed: > https://hg.mozilla.org/integration/mozilla-inbound/rev/ > 355349b199c457d45e420d775df07f7b6e9df859 Yes, he did, however the issue still occurs: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=41d06ef57fe6894c4ebd5a2e89b8788b1fd5fd8f&selectedJob=177152800 [task 2018-05-06T03:46:43.085Z] 03:46:43 INFO - if response.status != 200: [task 2018-05-06T03:46:43.085Z] 03:46:43 INFO - > raise error.from_response(response) [task 2018-05-06T03:46:43.085Z] 03:46:43 INFO - E InvalidArgumentException: invalid argument (400): Invalid unhandledPromptBehavior value: ignore [task 2018-05-06T03:46:43.086Z] 03:46:43 INFO - [task 2018-05-06T03:46:43.086Z] 03:46:43 INFO - body = {'capabilities': {'alwaysMatch': {'moz:firefoxOptions': {'binary': '/builds/worker/workspace/build/application/firefox...xn--lve-6lad.web-platform.test,www2.web-platform.test,www1.web-platform.test'}}, 'unhandledPromptBehavior': 'ignore'}}} [task 2018-05-06T03:46:43.086Z] 03:46:43 INFO - method = 'POST' [task 2018-05-06T03:46:43.086Z] 03:46:43 INFO - response = <Response status=400 error=<InvalidArgumentException http_status=400>> [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - self = <Session (disconnected)> [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - url = 'session' [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - tests/web-platform/tests/tools/webdriver/webdriver/client.py:438: InvalidArgumentException [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - TEST-PASS | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_default [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_twice - NoSuchAlertException: no such alert (404): No modal dialog is currently open [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - new_session = <function create_session at 0x7f7ba0038aa0> [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - add_browser_capabilites = <function update_capabilities at 0x7f7ba0038b90> [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - [task 2018-05-06T03:46:43.089Z] 03:46:43 INFO - def test_handle_prompt_twice(new_session, add_browser_capabilites): [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - _, session = new_session({"capabilities": {"alwaysMatch": add_browser_capabilites({"unhandledPromptBehavior": "accept"})}}) [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - value = session.execute_script("window.alert('Hello');window.alert('Bye');") [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - assert value is None [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - session.alert.dismiss() [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - # The first alert has been accepted by the user prompt handler, the second one remains. [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - # FIXME: this is how browsers currently work, but the spec should clarify if this is the [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - # expected behavior, see https://github.com/w3c/webdriver/issues/1153. [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - > assert session.alert.text == "Bye" [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - _ = {'capabilities': {'acceptInsecureCerts': False, 'browserName': 'firefox', 'browserVersion': '61.0a1', 'moz:accessibilityChecks': False, ...}, 'sessionId': 'b581a7cb-8b56-42fb-a9be-a7e4d4f7f92d'} [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - add_browser_capabilites = <function update_capabilities at 0x7f7ba0038b90> [task 2018-05-06T03:46:43.090Z] 03:46:43 INFO - new_session = <function create_session at 0x7f7ba0038aa0> [task 2018-05-06T03:46:43.091Z] 03:46:43 INFO - session = <Session b581a7cb-8b56-42fb-a9be-a7e4d4f7f92d> [task 2018-05-06T03:46:43.091Z] 03:46:43 INFO - value = None [task 2018-05-06T03:46:43.092Z] 03:46:43 INFO - [task 2018-05-06T03:46:43.092Z] 03:46:43 INFO - tests/web-platform/tests/webdriver/tests/execute_script/user_prompts.py:73:

Not sure why this bug has been marked as fixed. As we can see it still occurs. So reopening the bug. I can have a look later today.

A commit as listed in these changesets caused the regression: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=6b980ffeab0c61f0eae10fb6cdac204c34f23c63 Given that this happens for `execute_script` and `execute_async_script` I would point out to bug 1453105 which originally comes from https://github.com/w3c/web-platform-tests/pull/10401.

Ok, so the change as landed via the following PR now closes the second alert as opened by the execute_script call: https://github.com/w3c/web-platform-tests/pull/10401/files#diff-3c9886c5a3df541e02044c66f125020fR65 Given that in Firefox we do not yet support multiple alerts even not within the same browsing context (see bug 1264259) we should completely disable this specific test. Andreas, given that you reviewed this PR and spec change, do you agree?

thanks :whimboo!

What's strange is that this doesn't fail that constantly anymore now that some platforms / builds have been disabled. I wonder if there is some kind of race which causes that to appear.

(In reply to Henrik Skupin (:whimboo) from comment #8) > Given that in Firefox we do not yet support multiple alerts even > not within the same browsing context (see bug 1264259) we should > completely disable this specific test. Andreas, given that you > reviewed this PR and spec change, do you agree? Yes, I agree with that. (In reply to Henrik Skupin (:whimboo) from comment #10) > What's strange is that this doesn't fail that constantly anymore > now that some platforms / builds have been disabled. I wonder if > there is some kind of race which causes that to appear. I think it is likely there might be a collation of problems. Even though Marionette doesn’t support the user prompt handler it does have _some_ support for interacting with user prompts. The test harness tries to clean up any user prompts between tests and this might “conceal” some of the test failures/hangs.

Hm, I wonder how the disabled flag works for subtests as what wpt-sync adds like in this case. When I check the logs I can clearly see that tests are still run. Also it would be in conflict of what I have seen on bug 1447708. James, can you please explain?

We run the test, but ignore the result of the subtest.

I see. Thanks. So let me ask you for review.

Comment on attachment 8974285 [details] Bug 1459118 - [wdspec] Disable test_handle_prompt_twice for execute (async) script. https://reviewboard.mozilla.org/r/242596/#review248578

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8359164a5945 [wdspec] Disable test_handle_prompt_twice for execute (async) script. r=jgraham

This is not fixed but the test has been disabled.

This is still failing for us, and as far as I can see it is somewhat blocked on https://github.com/w3c/webdriver/issues/1153. Andreas or David, do you think we can get some clarification for that spec issue?

Actually this might be dependent on bug 1270585, which covers the real problem and not only the failing test. Appropriate feedback is still welcome.

https://github.com/w3c/webdriver/issues/1153 should be addressed by https://github.com/w3c/webdriver/pull/1248. When a user prompt appears during script evaluation, the script is immediately aborted and WebDriver returns to the user with data value of null. I confirmed a few days ago that this is the current behaviour of Marionette.

So we discussed this problem in our WebDriver Meeting (https://wiki.mozilla.org/WebDriver/Meetings/2018-07-30) today. The implementation of the test is actually still the old behavior before the last change to the spec was done for the two execute script commands. As such the test has to be updated in that it will assert that only a single prompt is open and the second one never has been created.

Well, the outcome of our discussion wasn't that perfect. Issue https://github.com/w3c/webdriver/issues/1153 has not been addressed yet, and any change as mentioned in my last comment would break the IEDriver. As such the issue has been reopened and we still need a clarification for the spec. This bug has to wait until then.

We are trying to build a tool to automatically classify intermittent failures, which would provide a starting point for fixing the bug, reducing the manual work for the developers. We are collecting some feedback on the results, to see if they’re good enough and where we need to improve. For this bug, the tool says that the intermittent failure is most likely a: Concurrency Issue: This includes tests in which failures occur due to thread management issues (different threads or their outcomes depending on an implicit ordering), race conditions and/or deadlocks, and issues related to an asynchronous waits (e.g. a process trying to access an external resource or continuing before the external resource is available). Once you’re done investigating and/or fixing the bug, could you tell me: - Did the tool correctly recognize the type of intermittent failure? - Did the information from the tool help your analysis, the bug fixing process, or anything in the process? (please also let us know how the tool was useful and/or what would improve the tool's usefulness for you)

This is actually blocked by bug 1487358 if opening two alerts is the expected behavior. The discussion is still ongoing on bug https://github.com/w3c/webdriver/issues/1153.

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/646459eb4a35 [wdspec] Re-enable test_abort_by_user_prompt_twice for execute (async) script. r=webdriver-reviewers,maja_zf

Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/fdc7d476adff [wdspec] Wait for second user prompt in test_abort_by_user_prompt_twice tests. r=webdriver-reviewers,jgraham

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/27650 for changes under testing/web-platform/tests

Upstream PR merged by moz-wptsync-bot

Moving bug to Remote Protocol::Marionette component per bug 1815831.