As a follow up to the prototype built for HAR generation in WebDriver BiDi (Bug 1770496) we want to check how the new implementation behaves in terms of performance.

We regularly received the feedback that recording HAR with Firefox DevTools created incorrect timings, much slower than how it would happen without DevTools.

The goal here is to collect such scenarios, where DevTools HAR recording has a visible impact on the page load performance. Then run those scenarios against the BiDi prototype to see if we have the same issue. And if we do, attempt to identify which part of the network observer logic is actually responsible for the performance hit.

Based on this information we should be able to create a HAR generation solution which has less performance impact than the DevTools one.

Hi Dave,

It was mentioned that the performance team might have some scenarios where using DevTools to record a HAR file would produce noticeably slower timings than a real navigation (ie without DevTools). Do you know of any scenario like that, could you share it here if yes?

Thanks!

Hi Julian,
sorry for jumping in, I've seen the performance difference, the more assets/requests a page do it's easier to see. If you use Browsertime to measure, we use https://github.com/firefox-devtools/har-export-trigger to get the devtools HAR. In Mozillas tests you use --skipHar that disables the har export trigger.

I just tried with https://en.wikipedia.org/wiki/COVID-19 and I could see the difference in the numbers, so running something like browsertime https://en.wikipedia.org/wiki/COVID-19 -b firefox --video --visualMetrics -c cable -n 5 --skipHar vs browsertime https://en.wikipedia.org/wiki/COVID-19 -b firefox --video --visualMetrics -c cable -n 5 I see a difference in 20-30% on firstPaint/first contentful paint/firstVisualChange, meaning it's 20-30% faster without devtools.

Maybe I can help making a branch of Browsertime where we hack triggering bidi to get the HAR so its easier to compare?

Thanks for jumping in Peter. Also adding a needinfo for :acreskey as I recall he may have some experience here too.

Thanks a lot for the input Peter, this is super helpful!

Maybe I can help making a branch of Browsertime where we hack triggering bidi to get the HAR so its easier to compare?

I think that's a good next step!
To make that work I think we also need a binary where BiDi's HAR recording would start automatically based on a pref for instance.
Would it be OK if it doesn't actually save the HAR to a file? I haven't added that part yet, but what's important is the fact that it is recording anyway, right ?

I think I should be able to tweak browsertime to set a preference (eg remote.bidi.har-experiment=true) in the --skipHar code path, and that should already give me some idea about the impact?

I'll get started on a way to record HAR from BiDi based on a pref, in the meantime let me know if something sounds terribly wrong with my plan :)

Would it be OK if it doesn't actually save the HAR to a file? I haven't added that part yet, but what's important is the fact that it is recording anyway, right ?

Yes sounds good. Even though that the HAR is not saved, is there a way to still verify that the preferences is set correctly so I know that I triggered the right path?

This sounds great and will be very useful.
One can pass arbitrary preferences into browsertime, e.g. --firefox.preference gfx.webrender.enabled:true so the prototyping should work well.

Thanks all for the input, I have a few early results, but I will still need more time to analyze the results and confirm them. For now what I can see is that BiDi HAR recording has an impact, but much smaller than the devtools HAR.

Here are my results for 5 runs on https://en.wikipedia.org/wiki/COVID-19

The BiDi implementation is a bit simplified compared to the devtools one (we don't monitor network information in content processes for instance) so those results are probably not very accurate. But simply running the scenario, I can see that devtools are started on the inspector which has a lot of impact on content processes, especially during the load. I will try to replace it with a blank tool, to see if that helps.

And from there I'll start bisecting a bit the BiDi and DevTools implementations to see what impacts performance.

Additionally, if remote.bidi.har-experiment.path is set to a folder, the browser
will attempt to save the har recording to a file before quitting.

The patch at https://phabricator.services.mozilla.com/D155837 should allow anyone to test in an automation setting.
Just pass remote.bidi.har-experiment=true to start recording HAR on the first tab which will be loaded.
And you can pass remote.bidi.har-experiment.path="/some/valid/path/" if you want the HAR file to be recorded under /some/valid/path/bidi-har-${timestamp}.har on exit.

I will add a link to a try build with binaries in case anyone wants to test this. Otherwise I will do more tests later this week.

Julian, should this bug be in M4 or shall we put it as P2 into the backlog?

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #10)

Julian, should this bug be in M4 or shall we put it as P2 into the backlog?

I would set it as P2. I was really looking forward to having a test case, so I wanted to do some investigation. But it's true that more than that will take some time now, and should be correctly planned.

I will unassign and move this to the backlog for now.

I will add a link to a try build with binaries in case anyone wants to test this.

https://treeherder.mozilla.org/jobs?repo=try&revision=11a990a81cb90c1aa75721a459a3d3144a01e1c1

Via bug 1806833 we will create a demo script to record HAR files. I assume that we will be able to use that one to get started with performance checks and comparisons.

Here's a performance profile while recording network events on a recent Nightly: https://share.firefox.dev/3zZ0iOU

One thing that stands out, the _log method from WebSocketConnection shows up relatively high, considering that trace logs are disabled. We should probably avoid building the strings when trace logs are disabled.

Edit: Second profile with many more requests, but with a fix for the _log from WebSocketConnection: https://share.firefox.dev/3MAiI09 . We see the security info parsing taking a long time here. Which is also interesting because this is one of the things which BiDi is not reading at the moment.