Andrew Halberstadt [:ahal] Assignee
	Description • 12 years ago

First step we should try and get as many reftests passing on B2G emulators as possible. Reftests require a specific screen size, so many of them will fail when running on actual phone devices.

We can leverage http://mxr.mozilla.org/mozilla-central/source/build/mobile/b2gautomation.py and copy the methods used for reftests on native fennec: http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/remotereftest.py

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 1 • 12 years ago

Note:on Android we only run half the reftests.  The main reason is bug 638815.  For crashtest/jsreftest we can run all but a few.

	Andrew Halberstadt [:ahal] Assignee
	Comment 2 • 12 years ago

Instructions for running:

1) Install requirements (https://raw.github.com/jonallengriffin/b2gautomation/master/b2gautomation/runmochi-requirements.txt)
>  source objdir/_virtualenv/bin/activate
>  pip install -r runmochi-requirements.txt

2) Set environment variables
>  export MOZ_HOST_DIR=path/to/xpcshell/bin/directory
>  export B2G_PATH=path/to/B2G/repo
>  export ADB_PATH=path/to/adb
>  export B2G_EMULATOR=arm # only if you are using the emulator
>  export REFTEST_PATH=path/to/reftest/manifest/from/gecko/root # default runs all tests

3) Run the tests
> make -C objdir reftest-b2g

You can also run the tests from the python file directly.
python runreftestsb2g.py --help for equivalent options. If you use this method you need to symlink the gecko root directory into the script directory as a side effect of the reftest server requiring the document root to be contained within the script directory.

Note that the screen size isn't getting set so expect to see some failures, as well if I run all the tests as a suite I get a seg fault in libc. I suspect it's a dumping issue that jmaher found earlier. I will try to find a work around.

	Andrew Halberstadt [:ahal] Assignee
	Comment 3 • 12 years ago

I wasn't setting the reftest manifest path in the make target properly. This fixes that.

	Andrew Halberstadt [:ahal] Assignee
	Comment 4 • 12 years ago

I also forgot to discard my changes to bootstrap.js, just ignore that.

	Jonathan Griffin (:jgriffin)
	Comment 5 • 12 years ago

Comment on attachment 638720 [details] [diff] [review]
WIP Reftests running on emulator

Review of attachment 638720 [details] [diff] [review]:
-----------------------------------------------------------------

Looking good!  A few comments:

::: build/mobile/b2gautomation.py
@@ +118,5 @@
>                  print currentlog
>                  if 'INFO SimpleTest FINISHED' in currentlog:
>                      return 0
> +                elif hasattr(self, 'logFinish') and self.logFinish in currentlog:
> +                    return 0

We should probably just add a default .logFinish attr to the class to match the mochitest case, and then override it for reftest.

::: layout/tools/reftest/reftest-content.js
@@ +119,5 @@
>  
>  function OnInitialLoad()
>  {
> +#if REFTEST_B2G
> +#else

This can just be:

#ifndef REFTEST_B2G

@@ +130,5 @@
>  
>      var initInfo = SendContentReady();
>      gBrowserIsRemote = initInfo.remote;
>  
>      addEventListener("load", OnDocumentLoad, true);

If we don't use this (since we're calling OnDocumentLoad directly), we should #ifndef this out, and move the direct call to OnDocumentLoad into this function, so it's easier to find where it's getting called.

	Andrew Halberstadt [:ahal] Assignee
	Comment 6 • 12 years ago

(In reply to Jonathan Griffin (:jgriffin) from comment #5) 
> >      addEventListener("load", OnDocumentLoad, true);
> 
> If we don't use this (since we're calling OnDocumentLoad directly), we
> should #ifndef this out, and move the direct call to OnDocumentLoad into
> this function, so it's easier to find where it's getting called.

Reftests have this insane way of starting the next test which involves incrementing a pointer to the current test and then reloading the document such that OnDocumentLoad gets called again. So this is actually needed. The rest I agree with, I still have some cleaning up to do.

	Andrew Halberstadt [:ahal] Assignee
	Comment 7 • 12 years ago

Updated patch with various fixes and modifications. Not incremental as the previous patch never got formally reviewed.

	Andrew Halberstadt [:ahal] Assignee
	Comment 8 • 12 years ago

I'll push to try after review.

	Jonathan Griffin (:jgriffin)
	Comment 9 • 12 years ago

Comment on attachment 650258 [details] [diff] [review]
Patch 1.0 - infrastructure to run reftests on b2g

Review of attachment 650258 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good.  At some point in the future, we should definitely think about consolidating all the code between B2G and remote versions, and between mochitest and reftest, but I suppose it's best to wait until related mozbase and virtualenv issues are worked out.

There should be a license header in b2g_start_script.js.

::: testing/testsuite-targets.mk
@@ +162,5 @@
> +RUN_REFTEST_B2G = rm -f ./$@.log && $(PYTHON) _tests/reftest/runreftestb2g.py \
> +  --remote-webserver=10.0.2.2 --b2gpath=${B2G_PATH} --adbpath=${ADB_PATH} \
> +  --emulator=${B2G_EMULATOR} --xre-path=${MOZ_HOST_BIN} $(SYMBOLS_PATH) \
> +  --ignore-window-size $(EXTRA_TEST_ARGS) $(1) | tee ./$@.log
> +

Can we default B2G_EMULATOR to "arm" if undefined?

	David Baron :dbaron:
	Comment 10 • 12 years ago

(In reply to Andrew Halberstadt [:ahal] from comment #6)
> Reftests have this insane way of starting the next test which involves
> incrementing a pointer to the current test and then reloading the document
> such that OnDocumentLoad gets called again.

You mean loading about:blank (rather than reloading the document)?  about:blank is loaded between tests so that we can correctly attribute assertions to tests, including assertions that happen while the page is being torn down.

	Andrew Halberstadt [:ahal] Assignee
	Comment 11 • 12 years ago

I decided to just remove the --emulator option from testsuite-targets.mk since it doesn't have to be an emulator build. It can get specified in $EXTRA_TEST_ARGS if needed.

Pushed a try run: https://tbpl.mozilla.org/?tree=Try&rev=d8b815a0aa69

	Andrew Halberstadt [:ahal] Assignee
	Comment 12 • 12 years ago

Jgriffin, could you commit this on my behalf? Thanks!

	Jonathan Griffin (:jgriffin)
	Comment 13 • 12 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/62346f5a2fc6

	Ryan VanderMeulen [:RyanVM]
	Comment 14 • 12 years ago

https://hg.mozilla.org/mozilla-central/rev/62346f5a2fc6