I am working on getting tests to work properly in osx 10.14 vs our current situation on 10.10, I found 9 tests that fail in the gfx/layers/apz/test/mochitest/mochitest.ini manifest, and all of them are skipped on android with a comment "wheel events not supported on mobile"

There are 3 other tests that have that comment that work (test_scroll_*), but the rest fail.

here is a try push showing some failures:
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=osx%2Cmochitest-e10s&revision=27d8e706de22ac93732dab0733347657e2a6433f&selectedJob=243484073

there are more failures, we just give up after 4 failures, so I had a few rounds of skipping tests to find the full list of failures.

here is a clip from a log:
04:59:03 INFO - TEST-START | gfx/layers/apz/test/mochitest/test_group_wheelevents.html
04:59:03 INFO - GECKO(1889) | Parent process: flushed APZ repaints, waiting for callback...
04:59:03 INFO - GECKO(1889) | helper_scroll_on_position_fixed.html | Flushed APZ repaints, waiting for callback...
05:04:22 INFO - TEST-INFO | started process screencapture
05:04:22 INFO - TEST-INFO | screencapture: exit 0
05:04:22 INFO - Buffered messages logged at 04:59:03
05:04:22 INFO - TEST-PASS | gfx/layers/apz/test/mochitest/test_group_wheelevents.html | Starting subtest helper_scroll_on_position_fixed.html
05:04:22 INFO - Buffered messages finished
05:04:22 INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_wheelevents.html | Test timed out.
05:04:22 INFO - SimpleTest.ok@SimpleTest/SimpleTest.js:275:18
05:04:22 INFO - reportError@SimpleTest/TestRunner.js:121:22
05:04:22 INFO - TestRunner._checkForHangs@SimpleTest/TestRunner.js:142:7
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - setTimeout handlerTestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
05:04:22 INFO - TestRunner.runTests/<@SimpleTest/TestRunner.js:381:20
05:04:22 INFO - promise callbackTestRunner.runTests@SimpleTest/TestRunner.js:368:50
05:04:22 INFO - RunSet.runtests@SimpleTest/setup.js:201:14
05:04:22 INFO - RunSet.runall@SimpleTest/setup.js:180:12
05:04:22 INFO - hookupTests@SimpleTest/setup.js:273:12
05:04:22 INFO - parseTestManifest@http://mochi.test:8888/manifestLibrary.js:36:5
05:04:22 INFO - getTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:49:11
05:04:22 INFO - EventHandlerNonNullgetTestManifest@http://mochi.test:8888/manifestLibrary.js:45:3
05:04:22 INFO - hookup@SimpleTest/setup.js:253:5
05:04:22 INFO - EventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fvar%2Ffolders%2F4v%2Fz3hm0241403fv400qb3sqj_h000017%2FT&cleanupCrashes=true:11:1
05:04:23 INFO - GECKO(1889) | MEMORY STAT | vsize 7330MB | residentFast 127MB | heapAllocated 17MB
05:04:27 INFO - TEST-OK | gfx/layers/apz/test/mochitest/test_group_wheelevents.html | took 323802ms

and the referenced screenshot:
https://taskcluster-artifacts.net/OgUNktxuRa6dfYhvBfj5Mw/0/public/test_info/mozilla-test-fail-screenshot_1UI8VG.png

Nothing obvious is standing out, this looks like a timeout- possibly the events are not firing, we need to force the events differently on 10.14, or maybe there are permissions needed for sending certain types of events.

:kats, could you help me find the right person to investigate this?

I guess I can update my machine to 10.14 and try to debug this.

The problem here seems to be that Mojave introduces new sandboxing rules that break CGEventPost which is what we're using to synthesize these mouse events. See https://forums.developer.apple.com/thread/103992 for example, other people have run into this problem. In theory it's supposed to pop up a dialog and prompt the user but I'm guessing that will only happen if we build against the 10.14 SDK; in automation we are building against the 10.11 SDK.

Thanks :kats for figuring this out.

If we build against the 10.14 SDK does that reduce the OS versions we can support (i.e. 10.14+ vs 10.10+)?

What is the risk of these tests being disabled? It sounds like all of the events based tests will be broken to some degree, so disabling all will be a problem.

:dividehex, Maybe there are sandboxing rules we can fix?

I'm not sure about the OS support levels with the different SDKs. I would imagine that newer SDKs should still support older OSes, unless Apple deprecated them. However building against the 10.14 SDK results in other problems (bug 1494022 specifically) so I'm not sure that's a viable path in the near future. Also I should emphasize that I'm only guessing this is related to the SDK; it might be that this problem persists even with the 10.14 SDK. Regardless we'd need bug 1494022 to be addressed before we can verify.

Also note that I explicitly added my locally-built Nightly.app to the Accessibility section of the OS privacy preferences, which in theory is supposed to allow CGEventPost to work again, but it didn't help.

I also filed a bug using Apple's bug reporting system, at https://bugreport.apple.com/web/?problemID=50665633, although I don't expect to get any response anytime soon (and that URL may not be visible to anybody else either, not sure)

Unassigning since I don't think there's anything else I can do here in the short term unless we figure out a way to make CGEventPost work, or find an alternate API to simulate OS-level events.

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)

:dividehex, Maybe there are sandboxing rules we can fix?

Not that I know of. I do know that with Mojave, Apple introduced significant privacy and security features. There are ways to whitelist applications but I believe there are also side requirements to that working such as the app being signed with an Apple developer certificate and the whitelist mobleconfig profile being installed via mdm (which we don't use).

Given that the migration to 10.14 workers is happening soon, we need to find a workaround to using CGEventPost. I'll look into this.

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=4baf63ba7eb8c6489bb0c802be645f3ee42a97c8 seems to work for me locally (modulo unrelated intermittents) and seems to be passing on try, so it should be compatible with both 10.14+ and older macOS.

https://hg.mozilla.org/mozilla-central/rev/80967042eb63

[Tracking Requested - why for this release]:
in order to support upgrading our hardware pool of osx machines from 1010 -> 1014, we need to uplift all changes required for tests to run successfully to firefox 68 in order for it to run on esr68.

Comment on attachment 9071905 [details]
Bug 1549052 - Avoid using CGEventPost for synthesizing wheel events on macOS. r?mstange

Beta/Release Uplift Approval Request

• User impact if declined:
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): we end up with a reduced pool of OSX machines as this code merges to ESR and we need to keep osx 10.10 installations around to support running tests.
• String changes made/needed:

Just to note, the code being touched in this patch is only ever exercised by test code, so it should be safe to uplift. It should have zero user-visible impact in Firefox.

Comment on attachment 9071905 [details]
Bug 1549052 - Avoid using CGEventPost for synthesizing wheel events on macOS. r?mstange

Updates test code for compatibility with macOS 10.14. Approved for 68rc1.

https://hg.mozilla.org/releases/mozilla-beta/rev/e106f30c1f95