Closed
Bug 1035226
Opened 10 years ago
Closed 9 years ago
Running Gaia integration tests against Mulet
Categories
(Firefox OS Graveyard :: Gaia, defect)
Tracking
(firefox42 fixed)
RESOLVED
FIXED
FxOS-S4 (07Aug)
Tracking | Status | |
---|---|---|
firefox42 | --- | fixed |
People
(Reporter: gerard-majax, Assigned: gerard-majax)
References
Details
(Whiteboard: [systemsfe])
Attachments
(6 files, 5 obsolete files)
(deleted),
text/x-log
|
Details | |
(deleted),
patch
|
ochameau
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
Details | Diff | Splinter Review | |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
gerard-majax
:
review+
|
Details | Diff | Splinter Review |
We wakt to run Gaia Integration tests against a mulet build
Assignee | ||
Comment 1•10 years ago
|
||
This bug is for tracking other bugs.
Assignee | ||
Comment 2•10 years ago
|
||
So, current status:
> 292 passing (1h)
> 48 pending
> 26 failing
That's not that bad !
Updated•10 years ago
|
Whiteboard: [systemsfe]
Assignee | ||
Comment 3•10 years ago
|
||
Result of a run against current master, with the fix for bug 1053185 relanded:
> 310 passing (2h)
> 65 pending
> 64 failing
This was run by:
> $ xvfb-run -a make test-integration RUNTIME=/home/alex/codaz/Mozilla/b2g/gecko/obj-mulet/dist/firefox/firefox shared/test/integration/tbpl-manifest.json
What should we do now ?
Attachment #8452154 -
Attachment is obsolete: true
Flags: needinfo?(poirot.alex)
Comment 4•10 years ago
|
||
Can you try running it with:
TEST_MANIFEST=./shared/test/integration/tbpl-manifest.json
?
I don't understand why tests pass when running against b2g desktop, do they??
By default we are using local-manifest.json, which contains no blacklist at all,
so I'm expecting to see the same tests failing when running on b2g desktop,
as tbpl-manifest.json explicitely disable many tests on desktop due to the lack of shims.
Flags: needinfo?(poirot.alex)
Comment 5•10 years ago
|
||
This modifies our mozharness script to pass RUNTIME to 'make test-integration' for Mulet.
Attachment #8499066 -
Flags: review?(poirot.alex)
Updated•10 years ago
|
Assignee: lissyx+mozillians → jgriffin
Comment 6•10 years ago
|
||
Builders added:
+ Ubuntu VM 12.04 x64 Mulet cedar opt test gaia-js-integration-1
+ Ubuntu VM 12.04 x64 Mulet cedar opt test gaia-js-integration-2
+ Ubuntu VM 12.04 x64 Mulet cedar opt test gaia-js-integration-3
+ Ubuntu VM 12.04 x64 Mulet cedar opt test gaia-js-integration-4
Attachment #8499076 -
Flags: review?(jlund)
Comment 7•10 years ago
|
||
Oops, didn't mean to re-assign this, just doing the infra bits.
Assignee: jgriffin → lissyx+mozillians
Comment 8•10 years ago
|
||
Comment on attachment 8499066 [details] [diff] [review]
Pass RUNTIME to Gij for Mulet,
Review of attachment 8499066 [details] [diff] [review]:
-----------------------------------------------------------------
Looks good, just some aesthetics comments.
::: scripts/gaia_integration.py
@@ +50,5 @@
> + ]
> +
> + # for Mulet
> + if 'firefox' in self.binary_path:
> + cmd += ['RUNTIME=%s' % self.binary_path]
Two comments:
- Can't we use `env` instead of hacking cmd string like that?
env['RUNTIME'] = self.binary_path
Or RUNTIME is one of these variable that can only be passed as make argument?
- And may be doing it no matter what is binary path? (It sounds less magical to pass an explicit path to the runtime to test rather than depending on putting the binary in a arbitrary path.)
Attachment #8499066 -
Flags: review?(poirot.alex) → review+
Comment 9•10 years ago
|
||
(In reply to Alexandre Poirot [:ochameau] from comment #8)
> Comment on attachment 8499066 [details] [diff] [review]
> Pass RUNTIME to Gij for Mulet,
>
> Review of attachment 8499066 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> Looks good, just some aesthetics comments.
>
> ::: scripts/gaia_integration.py
> @@ +50,5 @@
> > + ]
> > +
> > + # for Mulet
> > + if 'firefox' in self.binary_path:
> > + cmd += ['RUNTIME=%s' % self.binary_path]
>
> Two comments:
> - Can't we use `env` instead of hacking cmd string like that?
> env['RUNTIME'] = self.binary_path
> Or RUNTIME is one of these variable that can only be passed as make argument?
I haven't tried it, but some of the arguments are of that type, so it seems to make sense to be consistent.
> - And may be doing it no matter what is binary path? (It sounds less magical
> to pass an explicit path to the runtime to test rather than depending on
> putting the binary in a arbitrary path.)
Yep, I'll make this non-conditional.
Updated•10 years ago
|
Attachment #8499076 -
Flags: review?(jlund) → review+
Comment 10•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Created attachment 8499066 [details] [diff] [review]
> Pass RUNTIME to Gij for Mulet,
>
> This modifies our mozharness script to pass RUNTIME to 'make
> test-integration' for Mulet.
Mozharness patch, with changes suggested above: https://hg.mozilla.org/build/mozharness/rev/5c900c15ff70
Comment 11•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #10)
> (In reply to Jonathan Griffin (:jgriffin) from comment #5)
> > Created attachment 8499066 [details] [diff] [review]
> > Pass RUNTIME to Gij for Mulet,
> >
> > This modifies our mozharness script to pass RUNTIME to 'make
> > test-integration' for Mulet.
>
> Mozharness patch, with changes suggested above:
> https://hg.mozilla.org/build/mozharness/rev/5c900c15ff70
I restored the original patch and landed, since passing RUNTIME for the b2gdesktop case broke the tests on cedar:
https://hg.mozilla.org/build/mozharness/rev/97f6c98a5437
Comment 12•10 years ago
|
||
Comment 13•10 years ago
|
||
Committed code from this bug has been rolled out to production.
Comment 14•10 years ago
|
||
Committed code from this bug has been rolled out to production.
Comment 15•10 years ago
|
||
(whoops, sorry for writing that twice!)
Assignee | ||
Comment 16•10 years ago
|
||
Can we close this ? I'm unable to run those as of now, as documented in bug 1100345, while it was working previously.
Depends on: 1100345
Comment 17•10 years ago
|
||
Here is the tests running on taskcluster:
https://treeherder.allizom.org/#/jobs?repo=try&revision=d58202a02481
Comment 18•10 years ago
|
||
Comment 19•10 years ago
|
||
Note that we may just end up running tests on the new taskcluster infra as it just works in matter of minutes...
I'll submit taskcluster patch to bug 1099238.
If there is value in running them on the old infra, please shout.
Comment 20•10 years ago
|
||
I'm not sure whether there's value in fixing these here or not, but we probably should until they're running successfully in TC and being sheriffed.
Attachment #8572742 -
Flags: review?(jlund)
Updated•10 years ago
|
Assignee: lissyx+mozillians → jgriffin
Status: NEW → ASSIGNED
Comment 22•10 years ago
|
||
Comment on attachment 8572742 [details] [diff] [review]
Pass a config file when running Gij on mulet on cedar,
Review of attachment 8572742 [details] [diff] [review]:
-----------------------------------------------------------------
reading bug history suggests that I missed this in my earlier review. who needs configs anyway!?
Attachment #8572742 -
Flags: review?(jlund) → review+
Comment 23•10 years ago
|
||
Comment on attachment 8572742 [details] [diff] [review]
Pass a config file when running Gij on mulet on cedar,
https://hg.mozilla.org/build/buildbot-configs/rev/e77c49f5e338
Comment 24•10 years ago
|
||
Assignee | ||
Comment 25•10 years ago
|
||
What is the status of this bug ? I've hacked a little bit and it seems green enough, doesn't it?
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b95f3c102aa5&exclusion_profile=false
Flags: needinfo?(jgriffin)
Flags: needinfo?(coop)
Comment 26•10 years ago
|
||
It does look fine. You should ask James Lal to turn it on; he's handling B2G-specific TC scheduling, I believe.
Flags: needinfo?(jgriffin)
Comment 28•10 years ago
|
||
Hrm- what is the question here ? You do not need permission from me (or anyone actually :)) to land this if it's green aside from the reviews you have already gotten.
Flags: needinfo?(jlal)
Comment 29•10 years ago
|
||
(In reply to James Lal [:lightsofapollo] from comment #28)
> Hrm- what is the question here ? You do not need permission from me (or
> anyone actually :)) to land this if it's green aside from the reviews you
> have already gotten.
^^ This.
Flags: needinfo?(coop)
Assignee | ||
Comment 30•10 years ago
|
||
(In reply to James Lal [:lightsofapollo] from comment #28)
> Hrm- what is the question here ? You do not need permission from me (or
> anyone actually :)) to land this if it's green aside from the reviews you
> have already gotten.
Well I want to know:
- if people are working on this
- if the patch in this try request is the proper way to deal with it
- if the triggered run looks okay to you
Assignee | ||
Comment 31•9 years ago
|
||
Assignee | ||
Comment 32•9 years ago
|
||
Here is a recent mass retrigger: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1524decd5e94
So it looks like Gij 9 would need some love. Ryan what is your advice on this for the failure rate of others? There's a lot of blue that gets green in the end, so technically it's not orange, but it probably means there are intermittents there. Too much would mean wasting resources?
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 33•9 years ago
|
||
Rebased on current master.
Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=81480b83ce3e
Assignee | ||
Comment 34•9 years ago
|
||
Typo in the groupName
Attachment #8638902 -
Attachment is obsolete: true
Assignee | ||
Comment 35•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #33)
> Created attachment 8638902 [details] [diff] [review]
> Run Gij on Mulet r=...
>
> Rebased on current master.
> Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=81480b83ce3e
That has much much more green \o/
Assignee | ||
Comment 36•9 years ago
|
||
Comment on attachment 8638909 [details] [diff] [review]
Run Gij on Mulet r=...
Greg, can you review this? Also, once it lands, what are the steps to make it visible and useful?
Attachment #8638909 -
Flags: review?(garndt)
Comment 37•9 years ago
|
||
I've spent the last two weeks trying to get Gij running green enough to turn off the auto-retries, as they've been a disaster from a test stability standpoint. So no, I'm not willing to regress that on a new platform. Feel free to disable your way to victory, though. Seems that's the most-likely way you're going to get to green in a timely manner with this suite.
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 38•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #37)
> I've spent the last two weeks trying to get Gij running green enough to turn
> off the auto-retries, as they've been a disaster from a test stability
> standpoint. So no, I'm not willing to regress that on a new platform. Feel
> free to disable your way to victory, though. Seems that's the most-likely
> way you're going to get to green in a timely manner with this suite.
Does it means we need only green, or a one/two blue are okay?
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 39•9 years ago
|
||
Ryan, on this new try https://treeherder.mozilla.org/#/jobs?repo=try&revision=81480b83ce3e I suspect it contains the results of your efforts.
I see more blue on Mulet than B2G Desktop for Gij jobs 3, 4, 7, 9, 11, 12, 17 and 20. Would it be possible that we:
- land the current changes to run Gij on Mulet for the trees you sherrifs
- only show for now the test suites that are 100% green
- that I continue to work on either fixing or disabling the remaining failures that makes blue ?
Comment 40•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #38)
> Does it means we need only green, or a one/two blue are okay?
Consider anything blue to be orange w/ auto-retries disabled. At that point, it comes down to meeting the Job Visibility requirements (<5% failure rate, etc).
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
(In reply to Alexandre LISSY :gerard-majax from comment #39)
> Would it be possible that we:
> - land the current changes to run Gij on Mulet for the trees you sherrifs
> - only show for now the test suites that are 100% green
> - that I continue to work on either fixing or disabling the remaining
> failures that makes blue ?
Sounds fine to me.
By chance, do you know what needs to be done to turn off the auto-retries? If it can be done per-suite, let's get that done on Mulet first as well.
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 41•9 years ago
|
||
I think it's the 'rerun' field of the yml file. I'll continue working on this with this field removed. Let's go with the plan then! Thanks!
Assignee | ||
Comment 42•9 years ago
|
||
This time, let's disable reruns so we only get either green or orange. That
should help deciding regarding the job visibility criterions
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6715020461d5
Attachment #8638909 -
Attachment is obsolete: true
Attachment #8638909 -
Flags: review?(garndt)
Attachment #8638927 -
Flags: review?(garndt)
Comment 43•9 years ago
|
||
OOC, why does the groupName need to be changed for this? Mulet is already included elsewhere in the platform, so it seems redundant to include it there as well. It actually makes things like filtering on Treeherder more complicated, so it'd be preferable to avoid changing it unless you absolutely have to for some reason.
Assignee | ||
Comment 44•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #43)
> OOC, why does the groupName need to be changed for this? Mulet is already
> included elsewhere in the platform, so it seems redundant to include it
> there as well. It actually makes things like filtering on Treeherder more
> complicated, so it'd be preferable to avoid changing it unless you
> absolutely have to for some reason.
I'd be happy that you show me the value you would like, I don't have any specific reason for this other than exposing Mulet's name. But if you tell me you already have the information coming down from elsewhere and this is making your life bad, no problem :).
So, I should keep "groupName: Gaia JS Integration test" ?
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 45•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #42)
> Created attachment 8638927 [details] [diff] [review]
> Run Gij on Mulet
>
> This time, let's disable reruns so we only get either green or orange. That
> should help deciding regarding the job visibility criterions
>
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=6715020461d5
So: Gij 3, Gij 11, Gij 17, Gij 19 and Gij 20 are in bad shape.
Assignee | ||
Comment 46•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #42)
> Created attachment 8638927 [details] [diff] [review]
> Run Gij on Mulet
>
> This time, let's disable reruns so we only get either green or orange. That
> should help deciding regarding the job visibility criterions
>
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=6715020461d5
On Gij 3, it's FTU PseudoLocalization that is intermittent.
On Gij 11, it's System LockScreen related tests that are intermittents.
On Gij 17, it's localization tests in verticalhome that are intermittents.
For Gij 19 and 20, there is no test reported in the failures so I'd consider them related to the Gaia test infra unless someone can explain me :)
I think that each of you is responsible for apps impacted. Could you please have a look at those intermittents?
Flags: needinfo?(sfoster)
Flags: needinfo?(kgrandon)
Flags: needinfo?(gweng)
Flags: needinfo?(gaye)
Flags: needinfo?(aus)
Comment 47•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #46)
> On Gij 17, it's localization tests in verticalhome that are intermittents.
I've pushed an attempted fix for this in bug 1117630.
Flags: needinfo?(kgrandon)
Comment 48•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #46)
> On Gij 3, it's FTU PseudoLocalization that is intermittent.
For this one there appears to not be a stack, but my guess is that we need to change the wait after selecting a new language.
Currently we wait for mozL10n.readyState === 'complete', but it's possible to be in that state before selecting a new language. An alternative approach might be to use executeAsyncScript(), calling marionetteScriptFinished() after a 'localized' event.
Comment 49•9 years ago
|
||
(In reply to Kevin Grandon :kgrandon from comment #48)
> (In reply to Alexandre LISSY :gerard-majax from comment #46)
> > On Gij 3, it's FTU PseudoLocalization that is intermittent.
>
> For this one there appears to not be a stack, but my guess is that we need
> to change the wait after selecting a new language.
>
> Currently we wait for mozL10n.readyState === 'complete', but it's possible
> to be in that state before selecting a new language. An alternative approach
> might be to use executeAsyncScript(), calling marionetteScriptFinished()
> after a 'localized' event.
I'm currently trying this fix in bug 1181419.
Comment 50•9 years ago
|
||
Let's handle the FTU one in bug 1181419.
Assignee | ||
Comment 51•9 years ago
|
||
(In reply to Kevin Grandon :kgrandon from comment #47)
> (In reply to Alexandre LISSY :gerard-majax from comment #46)
> > On Gij 17, it's localization tests in verticalhome that are intermittents.
>
> I've pushed an attempted fix for this in bug 1117630.
Thanks, let's check this here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d93af9c3c2ee
Assignee | ||
Comment 52•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #51)
> (In reply to Kevin Grandon :kgrandon from comment #47)
> > (In reply to Alexandre LISSY :gerard-majax from comment #46)
> > > On Gij 17, it's localization tests in verticalhome that are intermittents.
> >
> > I've pushed an attempted fix for this in bug 1117630.
>
> Thanks, let's check this here:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d93af9c3c2ee
Kevin, seems like it is not fixed :(
Flags: needinfo?(kgrandon)
Updated•9 years ago
|
Attachment #8638927 -
Flags: review?(garndt) → review+
Assignee | ||
Comment 53•9 years ago
|
||
So, setting checkin-needed for attachment 8638927 [details] [diff] [review]. This should make those tests ran on all try and then once we have it we will be able to turn on suites to be displayed for sherrifs :)
Keywords: checkin-needed
Assignee | ||
Comment 54•9 years ago
|
||
Added dependent bugs to track greening each test suite.
Comment 55•9 years ago
|
||
Since you NI me in the individual bug, clean this one.
Flags: needinfo?(gweng)
Comment 56•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #44)
> So, I should keep "groupName: Gaia JS Integration test" ?
I think that the groupName/job name, etc, should be as generic as possible. Metadata like platform/flavor shouldn't be included as that information is already available elsewhere. Adding redundant information to the name just makes filtering rules, data mining, etc, more difficult to do.
Does that make sense? :)
Flags: needinfo?(ryanvm)
Assignee | ||
Comment 57•9 years ago
|
||
Fixed groupName
Attachment #8638927 -
Attachment is obsolete: true
Attachment #8639270 -
Flags: review+
Comment 58•9 years ago
|
||
Keywords: checkin-needed
Assignee | ||
Comment 59•9 years ago
|
||
Added the missing bits so that it is ran on all trees
Attachment #8639270 -
Attachment is obsolete: true
Attachment #8639446 -
Flags: review+
Assignee | ||
Updated•9 years ago
|
Keywords: checkin-needed
Comment 60•9 years ago
|
||
https://hg.mozilla.org/integration/b2g-inbound/rev/ee429ddf420a (backout)
https://hg.mozilla.org/integration/b2g-inbound/rev/71ad4ff41fe7
Keywords: checkin-needed
Comment 61•9 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #52)
> Kevin, seems like it is not fixed :(
Will try to handle this in bug 1187716. Thanks.
Flags: needinfo?(kgrandon)
Comment 62•9 years ago
|
||
Comment 63•9 years ago
|
||
I'm going to reset my ni? flag here since the actual specific issues are filed into separate bugs. I'll report my findings there.
Flags: needinfo?(aus)
Updated•9 years ago
|
Target Milestone: --- → FxOS-S4 (07Aug)
Assignee | ||
Updated•8 years ago
|
Flags: needinfo?(gaye)
You need to log in
before you can comment on or make changes to this bug.
Description
•