Central as Beta simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=7cc2d6c073df873d0a8990f70eefbda1cf86ecc8&selectedJob=228835648

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=228835648&repo=try&lineNumber=2510

Log snippet:

15:36:46 INFO - TEST-START | browser/components/newtab/test/browser/browser_as_render.js
15:36:46 INFO - TEST-INFO | started process screenshot
15:36:46 INFO - TEST-INFO | screenshot: exit 0
15:36:46 INFO - Buffered messages logged at 15:36:46
15:36:46 INFO - Entering test bound test_render_search
15:36:46 INFO - TEST-PASS | browser/components/newtab/test/browser/browser_as_render.js | Got the search box - {} == true -
15:36:46 INFO - TEST-PASS | browser/components/newtab/test/browser/browser_as_render.js | Search box is localized - "Search the Web" != "search_web_placeholder" -
15:36:46 INFO - Leaving test bound test_render_search
15:36:46 INFO - Entering test bound test_render_search_handoff
15:36:46 INFO - Buffered messages finished
15:36:46 INFO - TEST-UNEXPECTED-FAIL | browser/components/newtab/test/browser/browser_as_render.js | Got the search handoff button - null == true -
15:36:46 INFO - Stack trace:
15:36:46 INFO - resource://testing-common/content-task.js line 59 > eval:test_render_search_handoff:5
15:36:46 INFO - resource://testing-common/content-task.js:null:60
15:36:46 INFO - Leaving test bound test_render_search_handoff
15:36:46 INFO - Entering test bound test_render_topsites
15:36:47 INFO - TEST-PASS | browser/components/newtab/test/browser/browser_as_render.js | Got the top sites section - {} == true -

:k88hudson , could this be caused by https://bugzilla.mozilla.org/show_bug.cgi?id=1528410#c2 ?

Is this a recent regression or there hasn't been a central-as-beta until now?

The test should be setting the pref explicitly:

https://searchfox.org/mozilla-central/rev/e6520f0a4dd5d7474c32a1164744953ea59be0d0/browser/components/newtab/test/browser/browser_as_render.js#15-20

But the pref is default true only for Nightly:

https://searchfox.org/mozilla-central/rev/e6520f0a4dd5d7474c32a1164744953ea59be0d0/browser/app/profile/firefox.js#1301-1305

Yes, this is puzzling because I dont think there are any relevant differences between what is in Beta 66 and Nightly 67.

Looks like this was reported as intermittent before - Bug 1519829. So I guess something else changed that made it permanent. Probably a timing kind of issue? Are we waiting long enough for page to get rendered?

Oh right, and the code and pref are already on mozilla-beta, so I guess something must have changed recently for nightly 67 then.

The screenshot from test failure shows the -> arrow button, so it's showing the non-handoff version…

https://taskcluster-artifacts.net/F_U8PivvSmyZ8f22de412A/0/public/test_info/mozilla-test-fail-screenshot_r6qxjy.png

And curiously only fails on beta, so something with the pref being default false then switching to true.

This was caused by the patch for bug 1527702. Maybe should be uplifted together?

PR coming in a few.

This won't need to be uplifted to 66 beta for bug 1527702. This bug only affects 67.

Merged to master https://github.com/mozilla/activity-stream/commit/de302156cfeae3585f3d06f3f51819ec9ae9c112

https://hg.mozilla.org/mozilla-central/rev/f98b2b564191

Fixed verified in today's beta-sims: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=78e8be5e2e6573bb1ce027625ff96205c6efe765&searchStr=browser&group_state=expanded