Peter La Reporter
	Description • 12 years ago

What makes it feel slow/broken?

Almost every app feels slow to launch on the initial launch (initial launch is defined as first time launching after a device reset).  Many apps take 5 seconds to launch, and some take much longer (E-Mail, Cost Control, Calendar).

Did it prevent you from doing what you wanted? Why?

It slows me down and gives me a bad first impression as a user.  It sets my expectations that the OS is going to be slow.

How does this make you feel?

[ ]  :)  I feel happy about it
[ ]  :|  Meh
[X]  :(  I'm upset
[ ] >:O  I'm angry

Device: Unagi, Nov. 22 Nightly.

Details:

Comparing with our reference competitor platform (Android ICS on Otoro), the typical launch times are as follows.  The first number = initial launch time, the second number = subsequent launch time.

Our goal should be to do as well as ICS for initial launch times.

  Benchmarks

  B2G on Unagi

  Settings     - 4.1    / .7s
  Browser      - 2.5s   / .7s
  Camera       - 4.7s   / 1.7s
  Dialer       - 5s     / .7s
  Contacts     - 5.7    / .7s
  E-Mail       - 9-11.5 / .7s (GMail/8 Messages in inbox/Wi-Fi)
  Cost Control - 12.7s  / .7s
  Calendar     - 11.3s  / 1s
  Clock        - 4.8    / 1.5s
  SMS          - 3s     / 1s
  Gallery      - 4s     / 1.5s
  Music        - 5s     / 1s
  Video        - 5s     / 1s
  Calculator   - 3.4s   / 1s
  FM Radio     - 3.4s   / 1s
  Marketplace  - 4.5-6s / 1.4s

  Android ICS on Otoro

  Settings     - .5s    / .5s
  Browser      - 3s     / .7s
  Camera       - 3.5s   / 2.6s
  Dialer       - 2s     / .7s
  Contacts     - 1.3    / .7s
  E-Mail       - 3      / .7s
  Cost Control - N/A
  Calendar     - 1.9s   / .7s
  Clock        - 2.5    / 1.1s
  SMS          - ??     / .7s
  Gallery      - 1.7s   / .7s
  Music        - 1.5s   / .7s
  Video        - 1s     / .7s
  Calculator   - 1.5s   / .7s
  FM Radio     - 1.6s   / 1s
  Marketplace  - N/A

Bonus: can you attach a video of the problem?
Yes

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me
	Comment 1 • 12 years ago

We have to be very careful here to compare cold-launch times to cold-launch times.  ICS, and soon b2g, will launch some things in the background to improve startup time.

How are you measuring launch time?  That is, what series of steps are you following to measure, how do you define "launched", and how are you measuring times.

	Peter La Reporter
	Comment 2 • 12 years ago

I am comparing cold launch with cold launch.  The first column of numbers is cold launch time, the second column is subsequent launch.

"cold launch" is after you reset the device, unlock the device, and launch the app.

"launched" is defined as from the moment the icon is tapped to the moment the app is interactible.

I am timing with a stopwatch, so the times will have an error of ~200ms or less.

Question: should a bug per app be created for launch times or is one bug sufficient?

	Peter La Reporter
	Comment 10 • 12 years ago

Nominating for Basecamp Blocking+.

Cold app launch times are currently much longer than Android on Otoro, with the exception of the browser, and is contributing to a bad user experience.

However, we need a metric for determining when this bug is considered fixed.  Please advise...

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me
	Comment 11 • 12 years ago

(In reply to Peter La from comment #2)
> I am comparing cold launch with cold launch.  The first column of numbers is
> cold launch time, the second column is subsequent launch.
> 
> "cold launch" is after you reset the device, unlock the device, and launch
> the app.

That's not going to result in an apples to apples comparison in general, because apps can be prelaunched.  (We'll be investigating this for b2g in bug 815560.)  This also doesn't account for app assets being in disk cache.

Better is to kill any processes related to the app being tested in between runs, and do multiple trials and throw out outlier numbers.

> "launched" is defined as from the moment the icon is tapped to the moment
> the app is interactible.

What does "interactible" mean?

> Question: should a bug per app be created for launch times or is one bug
> sufficient?

One per app is better for tracking purposes.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me
	Comment 12 • 12 years ago

(In reply to Chris Jones [:cjones] [:warhammer] from comment #11)
> (In reply to Peter La from comment #2)
> > I am comparing cold launch with cold launch.  The first column of numbers is
> > cold launch time, the second column is subsequent launch.
> > 
> > "cold launch" is after you reset the device, unlock the device, and launch
> > the app.
> 
> That's not going to result in an apples to apples comparison in general,
> because apps can be prelaunched.  (We'll be investigating this for b2g in
> bug 815560.)

Although I should add that an apples to oranges comparison is also useful, since that's competitive parity.

	Peter La Reporter
	Comment 13 • 12 years ago

Hi Chris,

Thanks for the quick responses.

Interactible means you can start using the app - it is ready to respond to user events.  Often an app will launch and render, but does not respond to user events right away.

As for doing proper cold launches, I don't currently know how to kill process related to each app, or what those processes are.  I would need help with this, or have the individual bug fixers do the more rigorous testing per app to get definitive numbers.

I can file a separate bug for each app.

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me
	Comment 14 • 12 years ago

That definition sounds pretty hard to apply consistently to me.  I'd recommend we look at three things
 - when the UI first appears
 - when the UI first "looks like" it can be interacted with, for example when the first buttons/links appear
 - if you can come up with a consistently-applicable definition, when those buttons can actually be interacted with

As for "properly" measuring cold-start, I changed my mind :).  Let's go with apples-to-oranges.  That's what users will see.

Let's make sure to reboot both OSes in between each series of measurements though, and wait a minute or so before starting to measure.  Otherwise your data will be hard to reproduce.

	David Scravaglieri [:scravag]
	Comment 15 • 12 years ago

I considerate this as a Meta Bug, but the solution will be different from an app to another and the measurement must follow Chris suggestion.
Please add dependents bug to this.

	Peter La Reporter
	Comment 16 • 12 years ago

I'm creating separate bugs right now.  Decided not to wait until David has his new test numbers in.  David, can you add your numbers to all the separate bugs once you've got them?  (see all blocking bugs)

Thanks.

	Peter La Reporter
	Comment 17 • 12 years ago

Note on method used for timing:

1) Restart the device
2) Unlock the device
3) Wait 1 minute for the device to settle
4) Using a stopwatch program (I used iPhone), start the timer exactly when you tap on the application's icon
5) Pick an interactible element on the first screen that is shown at app launch (best if it's a button), and when you see this element, start tapping on it repeatedly.
6) As soon as the button responds, you can stop the timer.

It may take several tries to get it right.  You can practice without doing steps 1-3 first, then do the real timing with a device restart.

Furthermore, you may want to record 3 different times based on what Chris Jones outlined above: 1) when you first see the UI, 2) when the UI first looks interactible, 3) when the UI actually becomes interactible.

	Peter La Reporter
	Comment 18 • 12 years ago

These are some informal updated startup times I took last week on Jan. 14th build for Otoro.  Unfortunately I don't have numbers for Otoro from October, so I am listing them next to the Unagi numbers from October.

Timing method used was the same for both:

1 Restart device
2 Unlock device
3 Wait 30 seconds
4 Tap icon and start stopwach
5 Wait until App can be interacted with and stop stopwatch (this means the UI has fully rendered and no visible loading is taking place).

The process was done 2 times for each app.

--

Oct. Unagi vs Jan. 14 Otoro Numbers

--

  App Startup Times

  App            Unagi/Oct Otoro/Jan

  Settings     - 4.1       / 3.0 -> 27%
  Browser      - 2.5s      / 1.5 -> 40%
  Camera       - 4.7s      / 5
  Dialer       - 5s        / 2.3 -> 54%
  Contacts     - 5.7       / 3.6 -> 36%
  E-Mail       - 9-11.5    / 5.2 (no accounts setup)
  Cost Control - 12.7s     / 1.8 -> 78%
  Calendar     - 11.3s     / 3.3 -> 70%
  Clock        - 4.8       / 2.7 -> 43%
  SMS          - 3s        / 2.2 -> 27%
  Gallery      - 4s        / 2.7 -> 32%
  Music        - 5s        / 2.8 -> 44%
  Video        - 5s        / 5
  FM Radio     - 3.4s      / 2   -> 41%
  Marketplace  - 4.5-6s    / 1.9 (no connectivity)

  Return to Homescreen

  - 1-3 sec typical, sometimes 3+ / Currently: ~.5s or less

--

	Andrew Overholt [:overholt]
	Comment 19 • 12 years ago

This is a meta bug and we only block on bugs that are actionable.

	Jason Smith [:jsmith]
	Comment 21 • 12 years ago

Please read comment 19. Don't renom this one. Thanks.

	Casey Yee
	Comment 22 • 12 years ago

cc Arun for additional metrics.

	Darius K.
	Comment 23 • 12 years ago

Per comment 19 -- if this bug is not actionable, is there a reason this bug is still open with a NEW status? I think something like this should be marked RESOLVED WONTFIX or INVALID or similar so the bug DB can stay relevant. Or is adding the keyword "meta" enough and people should just know to filter those bugs out if they're looking to contribute?

	Dietrich Ayala (:dietrich)
	Comment 24 • 12 years ago

Hi Darius. This is very common in Mozilla development to have a meta bug like this which identifies a set of individually actionable items of work, and can itself be closed once all of those items are closed.

If you see a meta-bug like this, check out each of the dependent bugs to see which are unowned. Feel free to email me directly if you want more info.

	Hubert Figuiere [:hub]
	Comment 25 • 11 years ago

FIXED with all the performance work done in the last month.