To track our continued progress towards complete Puppeteer support
we need to create a new Tier-2/Tier-3 TaskCluster job running the
Puppeteer test suite on try.

The Puppeteer test suite requires Node.js, but since this is now a
build-time dependency we should be fine in that area. My current
thinking is to vendor the test suite in central, then have a mach
command for pulling in new changes from upstream.

I need to vendor the Puppeteer repository along with its dependencies
in central for this bug. My initial plan is to put the dependencies
under third_party/nodejs/ and annontate them with moz.yaml files,
and to put the Puppeteer repo under remote/test/puppeteer/.

I then intend to introduce a new mach command (something like
./mach remote vendor-puppeteer) to pull in the latest changes from
GitHub. This will allow the remote protocol to track progress towards
achieving full Puppeteer API support.

Does that all sound reasonable to you?

mach vendor already exists and seems like it should be the frontend here even if it requires a custom subcommand like mach vendor dav1d.

That seems reasonable. What third party dependencies do you have?
We meet tuesdays at 12:30pm est in my vidyo room to discuss node mc topics if you'd like to discuss in more depth.

CC Dan Mosedale

We may start with a forked version of puppeteer. For example, to workaround bug 1543115. But I imagine we will hit some other similar blockers. I can easily see us having to put some special bits to run the tests for firefox.
What is your plan to accomodate that?

Also, is there an urge to support mach vendor to update puppeteer?
Wouldn't it be easier to clone it manually and rather focus on getting the test running?
I imagine we could do that and still flag the files correctly as vendor via moz.yaml?

I don’t think ./mach vendor supports Node.js quite yet, but my
thinking was to manage the third-party dependencies using that so
that they could live under third_party/.

As we need to lock the Puppeteer version to the remote agent and
have the ability to apply custom patches to it, I want to manually
export a git clone into remote/test/puppeteer/. I mentioned
./mach remote vendor-puppeteer because those steps can
be automated: not as a replacement for ./mach vendor.

For info I made a fork of puppeteer 1.6.2 that works with the current Firefox Nightly and doesn't rely on Juggler. It normally sets the correct preferences for the profile. I'm not sure we want to use exactly that version here, but at least it should work: https://github.com/juliandescottes/pptr-ff

Yes, I did the same but with only the minimal amount of changes
required to make npm test run:
https://github.com/andreastt/puppeteer/tree/firefox

With my branch you still use output variables such as
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD and CHROME.

Since we arrived at the conclusion that we probably need to fork
Puppeteer regardless in order to add support for Firefox, going
with your fork long-term seems like the right thing to do.

Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.

The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.

The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.

The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.

Puppeteer is licensed under the Apache-2.0 license.

No code from Puppeteer gets included in Firefox.

Introduces "./mach puppeteer-test" command for running the Puppeteer
tests against the remote agent. This has to be a top-level command
because the automatic test detection system in mach does not allow
us to delegate to a subcommand such as "./mach remote puppeteer-test".

The tests run against a fork of Puppeteer with hotfixes needed for
it to work with the CDP implementation in Firefox. This fork is
located at https://github.com/andreastt/puppeteer/tree/firefox, and
vendored under remote/test/puppeteer/ in a previous commit in this series.

We vendor the Puppeteer library, but not its dependencies.
When "npm install" is called in remote/test/puppeteer/, it puts
its dependencies under remote/test/puppeteer/node_modules/ and
generates a remote/test/puppeteer/package-lock.json file. We do
not want these to be checked in.

DONTBUILD

Adds a --symlink <PATH> flag that instead of checkout the Puppeteer
source code from a git repo, allows one to symlink a local directory
into remote/test/puppeteer. This is useful for local development.

The Puppeteer test flavour are functional tests for the CDP-based
Puppeteer library from Google, that we want to run against our
implementation of CDP for Firefox.

They are distinct from the Firefox Puppeteer tests based on Marionette.

Makes it possible for mach to resolve test paths for Puppeteer,
so that individual tests can be run from the command line using
"./mach test", as such:

% ./mach test remote/test/puppeteer/test/screenshot.spec.js

As the Puppeteer test suite is imported from upstream and we cannot
change this directory at will (i.e. to add test manifest files),
we take the same approach as for WPT and populate the manifest by
recursively walking the remote/test/puppeteer/test/**/*.spec.js file tree.

This makes it possible to for mach to determine which test types
are likely to be relevant.

This adds a "remote(pup)" TaskCluster job for running the Google
Puppeteer tests against the new Firefox remote agent.

Earlier in this patchset we added a "./mach puppeteer-test" command
that wraps npm, and this being a source-test with access to the
full checkout, invokes this command directly instead of going
through mozharness.

try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=588eeea15278d70f7ae4ecaaff355803c58c8deb&selectedJob=254822601

It is expected to be failing on Tier-3 because we don’t support all the tests yet.

try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=c92334de41fcf97a830edf649617c7365ef73641

remote(pup) is expected to fail.

Backed out 11 changesets (bug 1540655) for test_resolve.py perma failures

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7&selectedJob=261606119

failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=261606119&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7&searchStr=mb

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=261606138&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7

backout: https://hg.mozilla.org/integration/autoland/rev/79127c2c712388688e6cbe5748b0d14d6efb766d

This was a test failure in test_resolve.py caused by is_puppeteer_path()
trying to match on None values. I’ve rebased and fixed the test.

https://hg.mozilla.org/mozilla-central/rev/f2b9955b0822
https://hg.mozilla.org/mozilla-central/rev/8f22cff02e35
https://hg.mozilla.org/mozilla-central/rev/5b769a365120
https://hg.mozilla.org/mozilla-central/rev/15c8a10a3994
https://hg.mozilla.org/mozilla-central/rev/beab1be0f328
https://hg.mozilla.org/mozilla-central/rev/4da1734351e6
https://hg.mozilla.org/mozilla-central/rev/bf91c2f4f386
https://hg.mozilla.org/mozilla-central/rev/23931313b016
https://hg.mozilla.org/mozilla-central/rev/7712e6870bd3
https://hg.mozilla.org/mozilla-central/rev/836e30244634
https://hg.mozilla.org/mozilla-central/rev/8a9e9189cd98