Per http://mzl.la/1k6ZmHK, Video is currently at/above 1100ms for its load time. Our acceptance metric for performance is either no regression from last release, or within the responsiveness guidelines, whichever is greater. This is a regression from 1.3's behavior (http://mzl.la/1jaJ2ZP) at ~750ms. Our responsiveness guideline for cold launch is 1000ms (https://wiki.mozilla.org/FirefoxOS/Performance/UserStories, Perception of Progress). Cold launch of Video should be brought within 1000ms. Requesting 1.4+ due to importance of the app.

This may be related to the tablet patch because we use some media queries in css. CC fred and chens here.

Tracking 997101 as a dependency, not a duplicate. The bug is being over 1000ms as an acceptance goal, and there are a number of avenues that can probably be taken to fix this. I'd prefer not to dupe the acceptance issue with an assumed root cause.

Hema, Please have someone on the media team look into this. Mike

Punam, As discussed in standup today, please investigate for other bottlenecks while the dependency is also being worked on in parallel. Feel free to reach out to Johu and performance engineers if you have questions. Thanks Hema

Tried b2gperf --delay=10 Video on 1.4 and 1.5/m-c builds 1.4 load-time 2014-05-01 16:54:41,301 B2GPerfRunner INFO | Results for Video, cold_load_time: median:1131, mean:1342, std: 375, max:2330, min:990, all:1189,1132,1111,1123,1121,1902,1129,1132,1890,1107,1136,2330,1104,1113,1875,1122,1119,1896,1135,1105,1960,1130,1122,1885,1161,1120,1130,990,1152,1864 1.5 load-time 2014-05-01 15:42:08,238 B2GPerfRunner INFO | Results for Video, cold_load_time: median:1079, mean:1080, std: 22, max:1162, min:1037, all:1162,1114,1079,1080,1089,1089,1079,1075,1059,1084,1042,1086,1082,1081,1066,1061,1087,1076,1066,1102,1037,1087,1079,1069,1125,1077,1065,1074,1075,1069 Hi John, Per http://mzl.la/1k6ZmHK , gaia branch - master , device - hamachi, load time for video app dropped in 1000s on 28 april and stayed around that since in 1.5. I checked gaia side and i don't see any video patch in that time frame that could have that impact. Do you know any gecko improvement that landed this or last week since (~ 27 april) on master that could have reduced load time. Also, please share if you have inputs on how to bring down the load - time for video app in 1.4. Thanks!

> Hi John, > Per http://mzl.la/1k6ZmHK , gaia branch - master , device - hamachi, load > time for video app dropped in 1000s on 28 april and stayed around that since > in 1.5. I checked gaia side and i don't see any video patch in that time > frame that could have that impact. Do you know any gecko improvement that > landed this or last week since (~ 27 april) on master that could have > reduced load time. > typo as you must have guessed, it's 1000ms

We had found a performance issue on CSS media query, bug 997101. We used few of them to support tablet version. If we can reduce them with css pre-processor or use JavaScript to load them dynamicly, https://groups.google.com/forum/#!topic/mozilla.dev.gaia/dVM02zQkmTk, that may help.

According to alive, the dropping down of load time at last week may be caused by a back out code of system app. It may goes back when the code land back.

per Comment 8, would you consider to use media-query-matching method to filter out @media for phone-sized device at build time? https://github.com/gasolin/provecss#media-query-matching

There is another alternative. We may try to use GAIA_DEVICE_TYPE=phone and GAIA_DEVICE_TYPE=tablet within Makefile to eliminate one media query: <link rel="stylesheet" type="text/css" href="style/video_tablet.css" media="(min-height: 768px) and (min-width: 768px)" />

Ran b2gperf --delay=10 Video and below are the load times 1) hamachi m-c Buildid:20131211040203 (without tablet fixes) 2014-05-02 14:02:37,708 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:615, mean:830, std: 302, max:1270, min:592, all:612,1253,592,596,1258,608,847,1244,598,602,1270,605,627,1261,595,595,1240,604,609,1252,614,627,1253,597,618,1255,613,617,1240,599 2 ) hamachi m-c build: 20131215040201 (with bug 903920 - tablet fixes) 2014-05-02 14:38:02,848 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:722, mean:766, std: 141, max:1290, min:695, all:850,709,719,851,722,731,1271,715,712,1290,716,712,740,715,719,719,723,725,728,742,725,695,734,717,727,721,719,722,729,696 3) hamachi 1.4 build 20140505000201 with media queries 2014-05-05 11:23:42,919 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1155, mean:1160, std: 30, max:1260, min:1124, all:1248,1124,1136,1199,1141,1128,1178,1155,1177,1151,1126,1166,1155,1138,1156,1156,1168,1136,1260,1172,1163,1138,1168,1172,1153,1154,1157,1164,1152,1131 4) hamachi 1.4 build 20140505000201 without media queries - by removing below include from video app index.html ( <link rel="stylesheet" type="text/css" href="style/video_tablet.css" media="(min-height: 768px) and (min-width: 768px)" /> ) and make reset-gaia 2014-05-05 11:06:17,275 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1127, mean:1125, std: 30, max:1179, min:1037, all:1176,1119,1138,1142,1144,1037,1179,1081,1136,1128,1122,1100,1125,1147,1126,1179,1107,1118,1135,1128,1088,1116,1077,1165,1136,1104,1142,1096,1126,1148 As per load time in 1 and 2, after flatfish tablet changes load time was still below 1000 ms for Video app and indicates that tablet changes are not solely responsible for the ~1100 ms load time. I see Video statistics and performance testing related fixes(Bug 948095 , Bug 922609) that landed in video. Will check and update bug if those have an impact on load_time.

> > 4) hamachi 1.4 build 20140505000201 without media queries - by removing > below include from video app index.html > ( <link rel="stylesheet" type="text/css" href="style/video_tablet.css" > media="(min-height: 768px) and (min-width: 768px)" /> ) and make reset-gaia John, can you pl. validate that removing the above include is enough to measure load time regression due to media query in video? > 2014-05-05 11:06:17,275 B2GPerfRunner INFO    | Results for Video, > cold_load_time: median:1127, mean:1125, std: 30, max:1179, min:1037, > all:1176,1119,1138,1142,1144,1037,1179,1081,1136,1128,1122,1100,1125,1147, > 1126,1179,1107,1118,1135,1128,1088,1116,1077,1165,1136,1104,1142,1096,1126, > 1148 > > As per load time in 1 and 2, after flatfish tablet changes load time was > still below 1000 ms for Video app and indicates that tablet changes are not > solely responsible for the ~1100 ms load time. I see Video statistics and > performance testing related fixes(Bug 948095 , Bug 922609) that landed in > video. Will check and update bug if those have an impact on load_time. Bug 948095 fix has been uplifted to 1.3. Bug 922609 helps track performance using PerformanceTestingHelper on cold launch. I tried reverting the patch on 1.4 to measure impact but the load time stays in ~1100 ms without 922609 fix. I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms), however failed to map it to any specific update in video app. John, do you know any updates in this time frame that could have impacted video app. Thanks

(In reply to Punam Dahiya from comment #13) > John, can you pl. validate that removing the above include is enough to > measure load time regression due to media query in video? I had asked the same question at bug 997101 and the reporter of it. We may test it. > > As per load time in 1 and 2, after flatfish tablet changes load time was > > still below 1000 ms for Video app and indicates that tablet changes are not > > solely responsible for the ~1100 ms load time. I see Video statistics and > > performance testing related fixes(Bug 948095 , Bug 922609) that landed in > > video. Will check and update bug if those have an impact on load_time. Thanks for this information. According to the finding of bug 977101, it should affect the cold launch time because we use some media queries in tablet to have totally different layout between landscape and portrait. > > Bug 948095 fix has been uplifted to 1.3. Bug 922609 helps track performance > using PerformanceTestingHelper on cold launch. I tried reverting the patch > on 1.4 to measure impact but the load time stays in ~1100 ms without 922609 > fix. > I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms), > however failed to map it to any specific update in video app. John, do you > know any updates in this time frame that could have impacted video app. > Thanks Sorry, I don't know which updates landed at that period affecting video app. I think we should ask sheriff or performance team to help us to find them out.

> I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms), > however failed to map it to any specific update in video app. John, do you > know any updates in this time frame that could have impacted video app. > Thanks Punam, Thanks so much for testing and narrowing down the regression date range on cold launch time. Mike, who is the best person to help further to nail down the root-cause. Per Punam's investigation, no changes went into video app between jan 30th and feb 8th (impact from using media query is being investigated on a dependent bug). Perhaps there are other changes during this period that are indirectly impacting video app. We will need your team's help here. Thanks Hema

Mike, We need your team's help to make further progress on this bug. Thanks!

Jon, Please help Hema's team root cause this issue. Thanks, Mike

Unassigning myself as i have run out of ideas . Waiting for performance team (comment #17) to provide input.

Jon/Mike -- we need your team's help to make further progress thanks hema

I will be looking at this tomorrow - sorry, was in Mountain View for a week, and had to concentrate on power related issues while I was there.

So I spent most of today trying to do a developer build with the right revisions to produce the "before" numbers (~800 ms) and so far I haven't had any luck. I will continue to work on this tomorrow.

Punam, can you point me to the specific build you have for Hamachi that shows a ~811 ms video app load time using b2gperf? I've tried recreating developer builds from various points in January, and nothing I've tried has scored lower than 1046 ms.

(In reply to Jon Hylands [:jhylands] from comment #22) > Punam, can you point me to the specific build you have for Hamachi that > shows a ~811 ms video app load time using b2gperf? I've tried recreating > developer builds from various points in January, and nothing I've tried has > scored lower than 1046 ms. I agree it's hard to pin a buildid , but the b2gperf video app cold launch time showed significant increase from jan 30th to feb 8th hamachi m-c jan 30th BuildId: 20140130040201 2014-05-02 19:27:25,589 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:811, mean:1014, std: 397, max:1832, min:775, all:923,828,1832,805,788,1823,810,814,872,804,775,812,893,813,1829,788,804,1781,800,808,1786,792,815,808,787,797,810,838,807,1783 B2GPerf hamachi m-c feb 8th BuildId:20140208040204: 2014-05-05 12:45:34,095 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1156, mean:1153, std: 36, max:1271, min:1092, all:1187,1230,1105,1139,1124,1165,1171,1157,1125,1171,1173,1143,1271,1128,1159,1145,1092,1121,1171,1137,1134,1177,1156,1119,1121,1109,1165,1185,1156,1167

So, after a few days of trying to get consistent results, I finally started to get somewhere today. I was able to flash my Hamachi with a 1.3 build that gave 600-800ms results, and then reproduce the results using m-c daily builds from Jan 30 and Feb 8 (Note that I tried this last week and was getting 1000+ ms numbers, not sure why it changed now). However, while trying to narrow down the results to a finer grained date range, I ended up having to reflash my phone (for the 5th or 6th time in the past few days) and the Teleweb tool crashed partway through flashing, and the phone appears to be bricked at this point. I'm going to try and fix it, but with this being a holiday in the US, I don't hold much hope. I will be at a conference starting tomorrow morning for the rest of the week, so someone else will have to do this. The location of the daily builds I used is here: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/ Specifically, Jan 30: (755 ms) https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/2014/01/2014-01-30-16-02-00/ Feb 8: (1130 ms) https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/2014/02/2014-02-08-16-02-01/ Mike, can you hand this off to someone else to investigate?

Jon will continue to work on this now that he's returned from JSConf.

I managed to unbrick my phone this morning using fastboot, and I've been continuing on this today. Together with what I found last Monday, these are my results so far (using nightly builds): https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-b2g28_v1_3-hamachi-eng Jan 31 (16:40) - median 692 Feb 8 (00:40) - median 697 Feb 16 (00:40) - median 694 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng Jan 30 (16:02) - median 755 Jan 31 (09:54) - median 955 Jan 31 (16:02) - median 969 Feb 1 (04:02) - median 1188 Feb 1 (16:02) - median 1217 Feb 2 (16:02) - median 973 Feb 3 (04:02) - median 979 Feb 4 (04:02) - median 1083 Feb 4 (16:02) - median 1282 Feb 5 (16:02) - median 1327 Feb 8 (16:02) - median 1130 I'm going to get the raw datazilla database dump from jeads for this time period, so hopefully that will help narrow things down a little more. From the above data, I would have to say there appear to be a couple issues - one that happened on Jan 30/31, and one that happened on Feb 4.

I've done some more tests, and it looks like the 755 listed above on Jan 30 was out-of-the-ordinary: Jan 15 (04:14) - median 947 Jan 20 (04:02) - median 962 Jan 27 (16:05) - median 1107 Jan 29 (16:02) - median 967 Given that almost all the values are between 950 and 1200, and the current m-c values (June 2) are about 1200, I think the direction we've been looking is not valid (i.e., the regression did not happen between Jan 30 and Feb 8 as previously reported). It is very clear that there has been a major regression between 1.3 and now, but we will need to dig further to find it.

Graph showing Video Cold Load time in January 2014, master, hamachi

After analyzing the data from all of January (Jeads exported the January datazilla database for me), it appears that there are two main regression points, although the second one will have to wait until I can see February's data to see what the trend is. The first happens on or about Jan 16, where there is a 100 ms regression (750 -> 850), although by January 28 it was back down to ~780 for a few runs. On Jan 31, there was a large jump (790 -> 930), but its not clear until I get the Feb data if it is transitory spike or an actual regression.

Graph showing 2 regressions (each > 100ms) in late Jan/early Feb for Video cold load times (hamachi, master)

So, with this new data, its pretty clear I have to eat my words from command 27, and the interesting regressions did in fact happen between Jan 30 and Feb 8. I'm now going to attempt to create developer builds with the specific Gaia/Gecko revisions for each regression, and bisect them.

the pip requirements file that works for running the b2gperf video test on the first known bad revisions. use this like so: pip install -r requirements_1002842.txt

Here's the bisect for one of the regression ranges: LKG Gecko: eb3bc59 LKG Gaia: d74eeec Results for Video, cold_load_time: median:1057, mean:1098, std: 148, max:1837, min:1006, all:1270,1092,1837,1085,1205,1178,1072,1023,1035,1061,1048,1111,1023,1054,1058,1049,1006,1032,1075,1038,1041,1028,1092,1053,1100,1089,1065,1056,1041,1030 FKB Gecko: f448244 FKB Gaia: e6b5fc4 Results for Video, cold_load_time: median:1304, mean:1337, std: 101, max:1750, min:1245, all:1340,1349,1304,1305,1353,1261,1304,1363,1563,1460,1291,1750,1413,1340,1272,1283,1364,1286,1292,1272,1302,1264,1299,1265,1387,1286,1330,1329,1245,1247 FKB Gecko: f448244 LKG Gaia: d74eeec Results for Video, cold_load_time: median:1483, mean:1470, std: 112, max:1659, min:1260, all:1549,1608,1395,1571,1590,1483,1261,1473,1552,1635,1260,1509,1659,1487,1294,1472,1484,1512,1304,1458,1478,1537,1272,1548,1464,1483,1260,1464,1499,1548 LKG Gecko: eb3bc59 FKB Gaia: e6b5fc4 Results for Video, cold_load_time: median:1086, mean:1092, std: 78, max:1340, min:1005, all:1331,1029,1030,1124,1103,1087,1105,1022,1090,1052,1026,1090,1060,1090,1052,1091,1088,1005,1029,1053,1019,1086,1058,1059,1087,1340,1224,1118,1150,1074 gecko regression gecko: 59ded1d Results for Video, cold_load_time: median:1076, mean:1329, std: 393, max:2438, min:1028, all:1189,1043,1029,1804,1104,1125,2438,1056,1058,1810,1065,1366,1924,1065,1051,1830,1038,1073,1818,1028,1074,1848,1079,1171,1836,1043,1041,1797,1042,1039 good gecko: 4cfb8bb Results for Video, cold_load_time: median:1126, mean:1359, std: 446, max:2435, min:1026, all:1202,1053,1039,1829,1130,1159,1123,1061,1063,2417,1056,1259,1850,1286,1070,2416,1109,1056,1815,1075,1055,1824,1228,1130,2435,1088,1062,1802,1059,1026 good gecko: 78f7c38 Results for Video, cold_load_time: median:1102, mean:1372, std: 461, max:2461, min:1030, all:1232,1044,1041,1842,1151,1126,2461,1076,1078,1819,1107,1094,1795,1056,1030,1819,1099,1103,1891,1135,1082,1835,1086,1044,2420,1051,1064,2417,1079,1102 good gecko: 21ca356 Results for Video, cold_load_time: median:1322, mean:1329, std: 72, max:1622, min:1251, all:1288,1317,1356,1257,1310,1301,1371,1340,1360,1316,1265,1622,1348,1483,1343,1355,1267,1281,1330,1325,1276,1347,1251,1265,1348,1389,1320,1266,1334,1258 bad gecko: 5a5fa6d Results for Video, cold_load_time: median:1326, mean:1324, std: 52, max:1521, min:1247, all:1369,1267,1336,1291,1399,1356,1294,1285,1294,1269,1338,1521,1360,1321,1300,1340,1332,1270,1353,1275,1285,1369,1279,1347,1247,1303,1342,1343,1363,1293 bad gecko: 9b46866 Results for Video, cold_load_time: median:1084, mean:1248, std: 381, max:2416, min:1029, all:1238,1069,1047,1802,1126,1157,1819,1051,1076,2403,1029,1141,1091,1097,1070,1066,1038,1827,1054,1076,1088,1079,1072,1151,1088,1090,1081,2416,1040,1071 good gecko: 3e8ad39 Results for Video, cold_load_time: median:1105, mean:1290, std: 370, max:2446, min:1006, all:1195,1084,1055,1815,1075,1126,1134,1056,1039,1818,1023,1400,1865,1111,1106,2446,1053,1006,1834,1077,1128,1826,1043,1057,1073,1061,1105,1859,1159,1088 good 5a5fa6d369ab2e3c430cb9e1a5e909e9922c4a3f is the first bad commit commit 5a5fa6d369ab2e3c430cb9e1a5e909e9922c4a3f Author: Fabrice Desré <fabrice@mozilla.com> Date: Tue Jan 14 16:00:25 2014 -0800 Bug 950266 - Re-enable the Nuwa process on B2G by default r=me :040000 040000 ee714dd593b91ae3bbabf139a301cc601103a0bf 8d2cb676eead40c2c8966fcc0f7edaa973e3921f M b2g :040000 040000 9eedfb6d546b4159f1258b1f60dbc93ad1171f18 505b5fb5f9f5d61d83c679abcce42fc1af467c41 M dom :040000 040000 98db54314717da56e9be72226f4f86ad402f0bc0 1d41e67a6a5e335e4384b3cd41ca1bb9264be388 M testing :040000 040000 e1107a1151393292aba2cb757a708097f65b45f9 c7c26367ed629078fc1484abaacf8cf8dda0b2e2 M toolkit

Here's the other regression: LKG Gecko: a8d1e0f LKG Gaia: 6c9fac8 Results for Video, cold_load_time: median:853, mean:1128, std: 423, max:1798, min:824, all:980,1794,840,847,1777,881,848,1771,824,847,1757,826,855,1767,840,840,1798,852,849,838,858,1760,827,831,1755,866,838,1787,849,863 FKB Gecko: 4652f1b FKB Gaia: f9013a0 FKB Gecko: 4652f1b LKG Gaia: 6c9fac8 Results for Video, cold_load_time: median:873, mean:875, std: 40, max:1029, min:809, all:1029,879,959,888,927,849,885,875,845,864,891,846,878,872,855,849,883,843,865,885,860,881,831,867,860,809,893,879,827,905 LKG Gecko: a8d1e0f FKB Gaia: f9013a0 Results for Video, cold_load_time: median:991, mean:1140, std: 372, max:2329, min:944, all:1085,964,996,1824,1009,1004,1054,1014,947,959,987,944,1812,977,1126,1059,974,981,2329,964,978,1022,954,986,954,977,1041,1041,2267,975 gaia regression gaia: 75122a8 Results for Video, cold_load_time: median:995, mean:1097, std: 319, max:2294, min:923, all:1201,957,961,1923,1013,1034,990,956,996,995,953,1006,1859,998,963,1040,923,979,972,992,1010,1078,1008,1011,2294,953,944,957,925,1024 bad gaia: 869e51f Results for Video, cold_load_time: median:879, mean:1084, std: 391, max:1822, min:823, all:989,1780,882,990,1773,892,838,1814,896,862,1779,910,827,1778,840,850,1782,834,871,840,823,876,830,839,833,841,1822,893,913,844 good gaia: 6b4def0 Results for Video, cold_load_time: median:1022, mean:1239, std: 495, max:2501, min:946, all:1046,962,1024,2501,1049,1031,1039,1006,947,1791,961,995,2462,1041,951,2287,972,998,946,1053,1831,1010,1029,2304,961,1020,980,952,995,1026 bad gaia: caf4dde Results for Video, cold_load_time: median:1008, mean:1103, std: 304, max:2330, min:935, all:1077,935,959,1144,1009,1005,946,972,1015,1804,1005,971,1010,1129,1034,972,1011,958,1787,1024,1008,2330,1008,1039,1023,964,1008,1011,968,991 bad caf4dde01228fe880bf93e151f0c2057f58bea41 is the first bad commit commit caf4dde01228fe880bf93e151f0c2057f58bea41 Author: Kevin Grandon <kevingrandon@yahoo.com> Date: Thu Jan 30 11:31:12 2014 -0800 Bug 962655 - Use build-time preference to enable rocketbar r=vingtetun, yurenju :100644 100644 e0cb7b4a0cd0573fa5772e5a75dad77925f022e7 a3c8aeea716dceff8fd3a2324caeababebcf7081 M Makefile :040000 040000 6c321ae23f26a20e692c4e61075a026e3fb19ede 8b6a00fc847bd24c7002bfaa06e35db94087e427 M apps :040000 040000 48a71e9e1aaf50b497ca2149d82ab0d50677c42e 5196eb22b5e00be17e03f1880b42e9a3f5ae3bfa M build :040000 040000 7a510590d6fa1256375db1ef3d6bd6cb56b727d3 2c60420d65162d9c2ddcd8320272e2dabc546ddf M shared :040000 040000 567ad65ba8ba36379103b790778be2f9608ed32b bb9214743c012902ecc03557542693a86c0c2e09 M tests

I've created two new bugs to handle the followups for these two regressions, and marked them both with 1.4? Bug 1024923 (regression identified in comment 34) Bug 1024928 (regression identified in comment 33)

Calling this "fixed" for tracking purposes though it seems to me that this bug as-filed isn't really fixed until the dependent bugs are also fixed.