there is initial work being done to get tests running on osx 10.14, Overall most things work, but in the debug version of the mochitest-webgl tests, I see some crashes on cleanup and leakchecking:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher%40mozilla.com&searchStr=mac%2Cwebgl&fromchange=96a96f509d98f90ce3f8e35990df75153755b877&tochange=b9ef9517732230f87398e4b26d30ed5c2e34adce

most of the other failures are unexpected-pass, which is great.

My goal is to figure out the crash- do we need to adjust anything on the OS or run our tests/tools differently?

here is a snippet of the crash:
10:25:09 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::gl::MozFramebuffer::CreateWith(mozilla::gl::GLContext*, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, unsigned int, bool, unsigned int, unsigned int)]
10:25:09 INFO - Crash dump filename: /var/folders/j1/6dcylzzs37sfs_4n_8j4_fvc000017/T/tmpqF1uVI.mozrunner/minidumps/63AFC055-75E0-4C57-B312-C6CD9EEE2353.dmp
10:25:09 INFO - Operating system: Mac OS X
10:25:09 INFO - 10.14.0 18A391
10:25:09 INFO - CPU: amd64
10:25:09 INFO - family 6 model 69 stepping 1
10:25:09 INFO - 4 CPUs
10:25:09 INFO -
10:25:09 INFO - GPU: UNKNOWN
10:25:09 INFO -
10:25:09 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
10:25:09 INFO - Crash address: 0x0
10:25:09 INFO - Process uptime: 134 seconds
10:25:09 INFO -
10:25:09 INFO - Thread 0 (crashed)
10:25:09 INFO - 0 XUL!mozilla::gl::MozFramebuffer::CreateWith(mozilla::gl::GLContext*, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, unsigned int, bool, unsigned int, unsigned int) [MozFramebuffer.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 131 + 0x29]
10:25:09 INFO - rax = 0x0000000118b3a87c rdx = 0x0000000000000000
10:25:09 INFO - rcx = 0x000000010d6196a8 rbx = 0x0000000000008cdd
10:25:09 INFO - rsi = 0x00000000000120a8 rdi = 0x00007fff97506028
10:25:09 INFO - rbp = 0x00007ffee290d5a0 rsp = 0x00007ffee290d500
10:25:09 INFO - r8 = 0x00000000000130a8 r9 = 0x0000000000000000
10:25:09 INFO - r10 = 0x00007fff97506048 r11 = 0x00007fff97506040
10:25:09 INFO - r12 = 0x0000000000000000 r13 = 0x00000001281c8000
10:25:09 INFO - r14 = 0x0000000127fc6a00 r15 = 0x00007ffee290d788
10:25:09 INFO - rip = 0x00000001125f7f97
10:25:09 INFO - Found by: given as instruction pointer in context
10:25:09 INFO - 1 XUL!mozilla::gl::MozFramebuffer::Create(mozilla::gl::GLContext*, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, unsigned int, bool) [MozFramebuffer.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 68 + 0x7]
10:25:09 INFO - rbp = 0x00007ffee290d770 rsp = 0x00007ffee290d5b0
10:25:09 INFO - rip = 0x00000001125f719e
10:25:09 INFO - Found by: previous frame's frame pointer
10:25:09 INFO - 2 XUL!mozilla::WebGLContext::EnsureDefaultFB() [WebGLContext.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 649 + 0xb6]
10:25:09 INFO - rbp = 0x00007ffee290d7c0 rsp = 0x00007ffee290d780
10:25:09 INFO - rip = 0x0000000113f37458
10:25:09 INFO - Found by: previous frame's frame pointer
10:25:09 INFO - 3 XUL!mozilla::WebGLContext::ValidateAndInitFB(mozilla::WebGLFramebuffer const*, unsigned int) [WebGLContext.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 1744 + 0x5]
10:25:09 INFO - rbp = 0x00007ffee290d800 rsp = 0x00007ffee290d7d0
10:25:09 INFO - rip = 0x0000000113f3aba0
10:25:09 INFO - Found by: previous frame's frame pointer
10:25:09 INFO - 4 XUL!mozilla::WebGLContext::BindCurFBForDraw() [WebGLContext.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 1771 + 0x11]
10:25:09 INFO - rbp = 0x00007ffee290d820 rsp = 0x00007ffee290d810
10:25:09 INFO - rip = 0x0000000113f28dea
10:25:09 INFO - Found by: previous frame's frame pointer
10:25:09 INFO - 5 XUL!mozilla::ValidateDraw(mozilla::WebGLContext*, unsigned int, unsigned int) [WebGLContextDraw.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 216 + 0x5]
10:25:09 INFO - rbp = 0x00007ffee290d880 rsp = 0x00007ffee290d830
10:25:09 INFO - rip = 0x0000000113f41d10
10:25:09 INFO - Found by: previous frame's frame pointer
10:25:09 INFO - 6 XUL!mozilla::WebGLContext::DrawArraysInstanced(unsigned int, int, int, int) [WebGLContextDraw.cpp:b9ef9517732230f87398e4b26d30ed5c2e34adce : 510 + 0xb]
10:25:09 INFO - rbp = 0x00007ffee290d960 rsp = 0x00007ffee290d890
10:25:09 INFO - rip = 0x0000000113f42581
10:25:09 INFO - Found by: previous frame's frame pointer

:jgilbert- would you have any ideas on why we see this crash?

I see what looks to be the same crash on crashtests:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher%40mozilla.com&fromchange=96a96f509d98f90ce3f8e35990df75153755b877&tochange=b9ef9517732230f87398e4b26d30ed5c2e34adce&searchStr=crashtest

REFTEST PROCESS-CRASH | file:///Users/cltbld/tasks/task_1555172865/build/tests/reftest/tests/dom/canvas/crashtests/844280.html | application crashed [@ mozilla::gl::MozFramebuffer::CreateWith(mozilla::gl::GLContext*, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, unsigned int, bool, unsigned int, unsigned int)]

It's an assert for framebuffer completeness:
https://searchfox.org/mozilla-central/source/gfx/gl/MozFramebuffer.cpp#131

We need to figure out why the framebuffer isn't complete. Do we have any test machines in Mountain View I can debug this on?

we have 5 machines setup in our datacenter- we could get one reserved for you and you can vpn in and remote access the machine. Would you be ok with that?

Sure!

:dividehex, can you loan :jgilbert one of the osx 10.14 machines.

:jgilbert once you get in if you have troubles, ask on irc in #cia; I am in there and can help, likewise others like :ahal, :gbrown, :egao, or :bc.

:jgilbert, please attach your public ssh key to this bug and I'll quarantine a loaner for you to access.

host is quarantined and login info sent via email.

jgilbert needs to be added to ldap group 'vpn_releng_loan'

(In reply to Jake Watkins [:dividehex] from comment #9)

host is quarantined and login info sent via email.

jgilbert needs to be added to ldap group 'vpn_releng_loan'

Done; please let me know when things are sorted and we can remove access.

from bug 1550068, I found 3 tests as the root cause:
[generated/test_2_conformance__rendering__rendering-stencil-large-viewport.html]
[generated/test_2_conformance2__textures__misc__generate-mipmap-with-large-base-level.html]
[generated/test_2_conformance__canvas__drawingbuffer-static-canvas-test.html]

:jgilbert, any updates on this? I believe other than unexpected pass conditions these are the only failures.

I could reproduce the problem on my mac(10.14.2) with the following command.
./mach test dom/canvas/test/webgl-conf/generated/test_2_conformance__rendering__rendering-stencil-large-viewport.html

It hit assert in MozFramebuffer::Create(), since error code was not LOCAL_GL_FRAMEBUFFER_COMPLETE. The error code was LOCAL_GL_FRAMEBUFFER_UNSUPPORTED (0x8CDD)
https://searchfox.org/mozilla-central/source/gfx/gl/MozFramebuffer.cpp#130

When the assert was hit, a requested size was (16383, 16383). And next try of size (8191, 8191) was succeeded. gl->MaxTexOrRbSize() was 16384, then it is weird to failed to allocate with (16383, 16383).

When I tried MozFramebuffer::CreateWith() on mac(10.14.2), (14000, 14000) was succeeded, but (15000, 15000) was failed.

That's pretty weird, but ok. Usually _UNSUPPORTED means that the combination of color-format/depth/stencil is not supported by the driver. I've never heard of max RT size being different from max tex size.

Right, so the GL core 4.1 spec says:

The combination of internal formats of the attached images does not violatean implementation-dependent set of restrictions
[Else] FRAMEBUFFER_UNSUPPORTED

What might be happening here is a allocation failure for one of the attachments. If we don't check for OOM on those allocations, surprise FRAMEBUFFER_UNSUPPORTED would be a symptom.

We should check for OOM during allocations.

Actually that makes a ton of sense that perhaps a 14k*14k*4bpp*4 sample might succeed where a 15k square might fail. I suspect that's what's happening. Let me try on my machine...

Good: 14784
Bad: 14785

mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp 
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ ./a.out 
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
pixels: 218566656 / 0xd071000
bytes: 874266624 / 0x341c4000
status: 0x8cd5
Done.
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp 
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ ./a.out 
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
pixels: 218596225 / 0xd078381
bytes: 874384900 / 0x341e0e04
status: 0x8cdd
Assertion failed: (status == GL_FRAMEBUFFER_COMPLETE), function main, file rgba-d24s8-unsupported.cpp, line 71.
Abort trap: 6

mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
14784 x 14816
pixels: 219039744 / 0xd0e4800
bytes: 876158976 / 0x34392000
status: 0x8cd5
Done.
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
14784 x 14817
pixels: 219054528 / 0xd0e81c0
bytes: 876218112 / 0x343a0700
status: 0x8cdd
Assertion failed: (status == GL_FRAMEBUFFER_COMPLETE), function main, file rgba-d24s8-unsupported.cpp, line 86.
Abort trap: 6

mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
16384 x 13376
pixels: 219152384 / 0xd100000
bytes: 876609536 / 0x34400000
status: 0x8cd5
Done.
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
16384 x 13377
pixels: 219168768 / 0xd104000
bytes: 876675072 / 0x34410000
status: 0x8cdd
Assertion failed: (status == GL_FRAMEBUFFER_COMPLETE), function main, file rgba-d24s8-unsupported.cpp, line 116.
Abort trap: 6

With and without a rgba16f color buffer:

mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
16384 x 10000
pixels: 163840000 / 0x9c40000
bytes: 1966080000 / 0x75300000
status: 0x8cdd
Assertion failed: (status == GL_FRAMEBUFFER_COMPLETE), function main, file rgba-d24s8-unsupported.cpp, line 135.
Abort trap: 6
mozs-MacBook-Pro:test-mac-rgba-d24s8-unsupported jgilbert$ clang++ --std=c++14 -framework OpenGL -DGL_SILENCE_DEPRECATION ../init.cpp ../util.cpp rgba-d24s8-unsupported.cpp && ./a.out
GL_VERSION: 4.1 INTEL-12.9.22
GL_RENDERER: Intel Iris Pro OpenGL Engine
16384 x 10000
pixels: 163840000 / 0x9c40000
bytes: 655360000 / 0x27100000
status: 0x8cd5
Done.

This is pretty wild.

  SIZE = 14784; // ok
  //SIZE = 14785; // bad


  auto WIDTH = SIZE;
  auto HEIGHT = SIZE;

  WIDTH = SIZE+2; HEIGHT = SIZE  ; // ok
  WIDTH = SIZE  ; HEIGHT = SIZE+2; // ok
  WIDTH = SIZE  ; HEIGHT = SIZE+3; // ok
  WIDTH = SIZE  ; HEIGHT = 16383; // bad
  WIDTH = SIZE  ; HEIGHT = 15000; // bad
  WIDTH = SIZE  ; HEIGHT = SIZE+10; // ok
  WIDTH = SIZE  ; HEIGHT = SIZE+100; // bad
  WIDTH = SIZE  ; HEIGHT = SIZE+50; // bad
  WIDTH = SIZE  ; HEIGHT = SIZE+30; // ok
  WIDTH = SIZE  ; HEIGHT = SIZE+40; // bad
  WIDTH = SIZE  ; HEIGHT = SIZE+35; // bad
  WIDTH = SIZE  ; HEIGHT = SIZE+32; // ok
  WIDTH = SIZE  ; HEIGHT = SIZE+33; // bad

  WIDTH = SIZE+32; HEIGHT = SIZE; // ok
  WIDTH = SIZE+33; HEIGHT = SIZE; // bad

  WIDTH = SIZE-1; HEIGHT = SIZE+32; // ok
  WIDTH = SIZE-1; HEIGHT = SIZE+33; // bad
  WIDTH = SIZE-2; HEIGHT = SIZE+33; // bad
  WIDTH = SIZE-10; HEIGHT = SIZE+33; // bad
  WIDTH = SIZE-30; HEIGHT = SIZE+33; // bad
  WIDTH = SIZE-50; HEIGHT = SIZE+33; // ok
  WIDTH = SIZE-40; HEIGHT = SIZE+33; // ok
  WIDTH = SIZE-35; HEIGHT = SIZE+33; // ok
  WIDTH = SIZE-33; HEIGHT = SIZE+33; // ok
  WIDTH = SIZE-32; HEIGHT = SIZE+33; // ok
  WIDTH = SIZE-31; HEIGHT = SIZE+33; // bad
  //
  //WIDTH = 16384; HEIGHT = 13369; // ok
  //WIDTH = 16384; HEIGHT = 13370; // ok
  //WIDTH = 16384; HEIGHT = 14000; // bad
  //WIDTH = 16384; HEIGHT = 13500; // bad
  //WIDTH = 16384; HEIGHT = 13200; // ok
  //WIDTH = 16384; HEIGHT = 13350; // ok
  //WIDTH = 16384; HEIGHT = 13425; // bad
  //WIDTH = 16384; HEIGHT = 13400; // bad
  //WIDTH = 16384; HEIGHT = 13375; // ok
  //WIDTH = 16384; HEIGHT = 13380; // bad
  //WIDTH = 16384; HEIGHT = 13376; // ok
  //WIDTH = 16384; HEIGHT = 13377; // bad

  //WIDTH = 16384; HEIGHT = 13376; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 6688; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 6689; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 6690; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 6688*1.5; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 10032; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 10000; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 8000; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8000; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8917; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8918; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8950; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8975; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8987; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8992; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 8996; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9000; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9500; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 9250; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 9125; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9190; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9220; // RGBA16F bad
  //WIDTH = 16384; HEIGHT = 9205; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9213; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9214; // RGBA16F ok
  //WIDTH = 16384; HEIGHT = 9215; // RGBA16F ok
  WIDTH = 16384; HEIGHT = 9216; // RGBA16F ok  with&without D24S8
  WIDTH = 16384; HEIGHT = 9217; // RGBA16F bad with&without D24S8

  //WIDTH = 16384; HEIGHT = 9216; // RGBA32F bad
  //WIDTH = 16384; HEIGHT = 7000; // RGBA32F bad
  //WIDTH = 16384; HEIGHT = 6000; // RGBA32F bad
  //WIDTH = 16384; HEIGHT = 5000; // RGBA32F bad
  //WIDTH = 16384; HEIGHT = 4500; // RGBA32F ok
  //WIDTH = 16384; HEIGHT = 9216/2; // RGBA32F ok
  //WIDTH = 16384; HEIGHT = 9216/2+1; // RGBA32F bad

Oh, and 16384*16384*rgba8 without d24s8 totally works.

My guess at this point is that there are limits based on the padded-to-PoT bytes-per-framebuffer-pixel, or maybe bytes-per-tile with some tile limit? (also maybe there's always an implicit d24s8 everywhere but rgba8)

Also d24s8 alone has the same limit as rgba8+d24s8.

Condensed:

  SIZE = 14784; // ok
  SIZE = 14785; // bad

  //WIDTH = 16384; HEIGHT = 13376; // ok   // 2^6 * 11 * 19
  //WIDTH = 16384; HEIGHT = 13377; // bad

  //WIDTH = 16384; HEIGHT = 9216; // RGBA16F ok  with&without D24S8  2^10 * 3^2 = 1024*9
  //WIDTH = 16384; HEIGHT = 9217; // RGBA16F bad with&without D24S8

  //WIDTH = 16384; HEIGHT = 9216/2; // RGBA32F ok    with&without D24S8  2^9 * 3^2 = 512*9
  //WIDTH = 16384; HEIGHT = 9216/2+1; // RGBA32F bad with&without D24S8

Reverting to MAX=8192 in generally would probably work for us here.

Also clean up MAX_TEXTURE_SIZE restriction code.

https://hg.mozilla.org/mozilla-central/rev/4b8d5a6d5143

I've run several try pushes with the revision of mozilla-central that should contain this fix, and despite my best attempts I'm still observing multiple PROCESS-CRASH failures on macosx1014, spread across several subsuites in mochitest.

Try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=9fbed80016abf084ade73673fc8c2ddcf59b0955&selectedJob=254006292

test-macosx1014-64/debug-mochitest-browser-chrome-e10s-10 has a process crash that is led with Main app process exited normally.

Is my understanding correct, that this patch should have addressed these types of crashes?

That's a different bug.

That's in fact bug 1558339!

Comment on attachment 9074405 [details]
Bug 1544446 - Restrict macOS MAX_TEXTURE_SIZE to 8192.

Beta/Release Uplift Approval Request

• User impact if declined:
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): I am requesting this in order to support migrating tests from osx 10.10 to 10.14, this solves crashes in a few tests (crashtest, mochitest-webgl).
• String changes made/needed:

Comment on attachment 9074405 [details]
Bug 1544446 - Restrict macOS MAX_TEXTURE_SIZE to 8192.

Uplift request is missing critical information, especially for an uplift to rc.

:jgilbert, can you help me fill out the remaining fields for a beta uplift request in comment 36?

User impact if declined: Bad behavior for a minority of WebGL content on Mojave. (only if it uses very large framebuffer sizes, which is uncommon)
Steps to reproduce: This page should not crash the browser: https://www.khronos.org/registry/webgl/sdk/tests/conformance/rendering/rendering-stencil-large-viewport.html?webglVersion=2&quiet=0&quick=1
String changes made/needed: none

Is that sufficient? I don't have privileges for re-requesting an approval.

Comment on attachment 9074405 [details]
Bug 1544446 - Restrict macOS MAX_TEXTURE_SIZE to 8192.

AIUI Joel needs this on esr68 so I'll move the approval request there.

thanks :jcristau, that is what I need!

Comment on attachment 9074405 [details]
Bug 1544446 - Restrict macOS MAX_TEXTURE_SIZE to 8192.

WebGL fix needed for migrating OSX tests to 10.14. Approved for 68.1esr.

https://hg.mozilla.org/releases/mozilla-esr68/rev/bf902a42a7c4

Hi Jeff, do you need the macos mojave loaner that started with this bug? (t-mojave-r7-466.test.releng.mdc1.mozilla.com) I was checking over machines this morning and found this loaner (but with the bug closed). I'll return it to the staging pool if you don't need it.

Please return it, thanks!