Closed Bug 1329424 Opened 8 years ago Closed 5 years ago

Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoA... (images unexpectedly equal)

Categories

(Testing :: Marionette Client and Harness, defect, P1)

Product:
Testing
Component:
Marionette Client and Harness
Version:
Version 3
Type:
defect

Tracking

(firefox-esr60 disabled, firefox67 disabled, firefox68 disabled, firefox69 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox-esr60 --- disabled
firefox67 --- disabled
firefox68 --- disabled
firefox69 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled])

Attachments

(1 file)

Bug 1329424 - [marionette] Re-enable tests TestScreenCapture[Content/Chrome].test_capture_flags. r=#webdriver
(deleted), text/x-phabricator-request
Details
Summary: Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoAAAANSUhEUgAAAT4AAAIqCAYAAAC9lyYgAAARQElEQVR4nO3dTVIi+4LG4dwPMnIfPcXIUS/hzHtWGMG0p26gxOjsUS9DywF7qA/r67iDfw8ATRBQUUngfZ6IN24J8lGc6++S6rlZ3d/fFzOzpFVd → Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoA... (images unexpectedly equal)
This happens mostly on Windows 10 nowadays. Joel, is there anyone checking for test failures on this new platform for tests?
Flags: needinfo?(jmaher)
we did a lot of work to check for new failures- but this is a "fragile" platform still and some jobs get enabled on accident. Should we leave this running, or disable it for now? If we just have 1 failure, we should fix it or skip it for the time being so the other tests are getting coverage.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) from comment #17) > we did a lot of work to check for new failures- but this is a "fragile" > platform still and some jobs get enabled on accident. Should we leave this > running, or disable it for now? If we just have 1 failure, we should fix it > or skip it for the time being so the other tests are getting coverage. It's up to you. Last week we got 30 failures so that we hit the threshold of a warning for priority. I think it might make sense to skip it on this platform to get it fixed.
while this has a lot of failures, it seems to be very few failures after the 18th.
:gbrown, I think we should disable this test given the very high failure rate
Flags: needinfo?(gbrown)
Whiteboard: [stockwell needswork]
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1dde9a9f7745 Skip TestScreenCaptureChrome.test_capture_flags for frequent failures; r=me,test-only
Flags: needinfo?(gbrown)
Keywords: leave-open
Whiteboard: [stockwell needswork] → [stockwell disabled]
Keywords: test-disabled
Priority: -- → P3
Priority: P3 → P5

As it looks like this test has two issues:

  1. When the chrome dialog gets opened it gets sometimes moved into the background. As such no focus/blur events will be raised. My patch on bug 1504756 refactors the Mn unit tests kinda heavily for opening a new window. Lets see if that helps here.

  2. We call blur() on the textbox, but don't actually wait for the blur event to happen. This can cause a race condition when trying to compare the screenshots between the focused and not focused textbox.

Depends on: 1504756

We should try to re-enable this test now that bug 1335085 has been fixed. Maybe that also helped here.

status-firefox67: --- → disabled
status-firefox68: --- → disabled
status-firefox69: --- → disabled
status-firefox-esr60: --- → disabled
Blocks: 1330560
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Keywords: leave-open
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8eb5ac3760e7 [marionette] Re-enable tests TestScreenCapture[Content/Chrome].test_capture_flags. r=maja_zf

https://hg.mozilla.org/mozilla-central/rev/8eb5ac3760e7

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
status-firefox69: disabled → fixed
Keywords: test-disabled
Priority: P5 → P1
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: