Working bug to enable Reftest on emulator-x86-kk.

Initial patch to enable reftest on emu-x86-kk.

Review commit: https://reviewboard.mozilla.org/r/30079/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/30079/

Review commit: https://reviewboard.mozilla.org/r/30081/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/30081/

Comment on attachment 8705578 [details] MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/1-2/

grandt, can you help to review this patch ? The 1st patch is to enable reftest on B2G-emulator-arm/x86-kk opt build, we used to enable it but running into tons of oranges(Bug1162375). Spent some time to figure out the cause and found that it's related mesa 3D library(8.0.4) installed in docker tester image "taskcluster/tester:0.4.5". The 2nd patch tried to switch mesa driver from llvmpipe to softpipe, which can get most of TC passed[1]. For those oranges(6 remaining), will be followed up via bug 1162375 and target to fix in next run. [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=eba2ecf0ece7

Comment on attachment 8705577 [details] MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production https://reviewboard.mozilla.org/r/30079/#review27275 ::: testing/taskcluster/tasks/branches/base_jobs.yml:263 (Diff revision 1) > + tasks/builds/b2g_emulator_kk_opt.yml: So it looks like there are a few tests that are orange right now but will be showing up as tier 1 on production treeherder. Could we get some feedback from a sheriff on how to proceed? From my understanding this could cause tree closure. Perhaps the sheriffs are ok with starring them under some bug, but I would like to be safe first.

Comment on attachment 8705578 [details] MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test. https://reviewboard.mozilla.org/r/30081/#review27277

(In reply to Greg Arndt [:garndt] from comment #32) > So it looks like there are a few tests that are orange right now but will be > showing up as tier 1 on production treeherder. Could we get some feedback > from a sheriff on how to proceed? From my understanding this could cause > tree closure. Perhaps the sheriffs are ok with starring them under some > bug, but I would like to be safe first. Thanks, garndt. I'll have sheriff's feedback to ensure those oranges would be hidden on staging/production treeherder.

Hi Astley, its no problem to hide the tests, we would need to know which one (like all or like R5, R19 etc). cc'ing Wes since he also does sheriffing

(In reply to Carsten Book [:Tomcat] from comment #35) > Hi Astley, > > its no problem to hide the tests, we would need to know which one (like all > or like R5, R19 etc). > > cc'ing Wes since he also does sheriffing Tomcat, thanks for help. Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40) Please also let me know how to proceed afterward.

(In reply to Astley Chen [:astley] from comment #36) > Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40) > Please also let me know how to proceed afterward. Hey Astley, i created now a filter that hides this tests and also filed a bug to get them unhidden when the tests are fixed (bug 1239951)

https://reviewboard.mozilla.org/r/30079/#review27275 > So it looks like there are a few tests that are orange right now but will be showing up as tier 1 on production treeherder. Could we get some feedback from a sheriff on how to proceed? From my understanding this could cause tree closure. Perhaps the sheriffs are ok with starring them under some bug, but I would like to be safe first. By comment#37, sheriff had created a filter to hide known oranges. I'd trigger a new try run and see if everything is in good shape prior to patch landed. Thanks.

(In reply to Carsten Book [:Tomcat] from comment #37) > (In reply to Astley Chen [:astley] from comment #36) > > Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40) > > Please also let me know how to proceed afterward. > > Hey Astley, > > i created now a filter that hides this tests and also filed a bug to get > them unhidden when the tests are fixed (bug 1239951) Hi tomcat, from the several try results, I found that oranges appeared to be intermittent within below chunks: 5,8,9,13,14,21,22,23,26,29,31,32,33,34,38,39,40. To ensure production tests wont be impacted by intermittent oranges, I will suggest to hide ALL reftest chunks on production and leave those oranges open on staging for try. Can you help to do that ? Thanks. Hi garndt, since now we have sheriffs assist to hide the chunks, can you help to review again ? Thanks.

Comment on attachment 8705577 [details] MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30079/diff/1-2/

Comment on attachment 8705578 [details] MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/2-3/

Comment on attachment 8705577 [details] MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production https://reviewboard.mozilla.org/r/30079/#review28801 Just a comment about how the symbols show up on TH and about the oranges. Looks good. ::: testing/taskcluster/tasks/tests/b2g_emulator_reftest.yml:35 (Diff revision 2) > + treeherderEnv: with adding this to production as a tier 1 job, do you still want the group symbol to be tc-R? I think this would show up as: tc-R(R1 R2 R3 ...) Maybe it could be R(1 2 3 4) Also looking at treeherder it seems that these are orange? does the other commit fix those? If not, having these show up on production as a tier 1 job probably will be an issue. Just something to keep in mind :)

Ah, disregard my comment about the oranges, I noticed the comments above about having them hidden.

(In reply to Astley Chen [:astley] from comment #41) > (In reply to Carsten Book [:Tomcat] from comment #37) > > (In reply to Astley Chen [:astley] from comment #36) > > > Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40) > > > Please also let me know how to proceed afterward. > > > > Hey Astley, > > > > i created now a filter that hides this tests and also filed a bug to get > > them unhidden when the tests are fixed (bug 1239951) > > Hi tomcat, from the several try results, I found that oranges appeared to be > intermittent within below chunks: > 5,8,9,13,14,21,22,23,26,29,31,32,33,34,38,39,40. > To ensure production tests wont be impacted by intermittent oranges, I will > suggest to hide ALL reftest chunks on production and leave those oranges > open on staging for try. > Can you help to do that ? Thanks. > should be done now!

https://reviewboard.mozilla.org/r/30079/#review28801 > with adding this to production as a tier 1 job, do you still want the group symbol to be tc-R? > > I think this would show up as: > tc-R(R1 R2 R3 ...) > > Maybe it could be R(1 2 3 4) > > Also looking at treeherder it seems that these are orange? does the other commit fix those? If not, having these show up on production as a tier 1 job probably will be an issue. Just something to keep in mind :) The group symbol "tc-R" is used to distinguish general reftest and Mulet Reftest(group symbol "R") and it seems a convention used in somewhere. So far I have no idea on a better naming and no plan to fix it in this bug since the intention of this bug is to enable reftest on B2G KK emulator. Thanks for pointing out of this, I'll be back to check in bug1162375.

Comment on attachment 8693469 [details] [diff] [review] Enable-reftest-on-emu-x86-kk-Treeherder-production.patch move to review board

Comment on attachment 8699301 [details] [diff] [review] Enable reftest on emu-arm-kk for test moved to review board.

failed to apply: apply changeset? [ynmpcq?]: y applying 7ac662108697 patching file testing/taskcluster/tasks/branches/base_jobs.yml Hunk #1 FAILED at 238 1 out of 1 hunks FAILED -- saving rejects to file testing/taskcluster/tasks/branches/base_jobs.yml.rej patch failed to apply abort: fix up the merge and run hg transplant --continue

Comment on attachment 8705577 [details] MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30079/diff/2-3/

Comment on attachment 8705578 [details] MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/3-4/

(In reply to Carsten Book [:Tomcat] from comment #53) > failed to apply: > > apply changeset? [ynmpcq?]: y > applying 7ac662108697 > patching file testing/taskcluster/tasks/branches/base_jobs.yml > Hunk #1 FAILED at 238 > 1 out of 1 hunks FAILED -- saving rejects to file > testing/taskcluster/tasks/branches/base_jobs.yml.rej > patch failed to apply > abort: fix up the merge and run hg transplant --continue rebased and require garndt to review again. thanks.

Reviewd.

tomcat, can you help again ? thanks a lot.

https://hg.mozilla.org/integration/mozilla-inbound/rev/7926adaf2ba7 https://hg.mozilla.org/integration/mozilla-inbound/rev/2f1fb9cc37d9

https://hg.mozilla.org/mozilla-central/rev/7926adaf2ba7 https://hg.mozilla.org/mozilla-central/rev/2f1fb9cc37d9