Mike Schroepfer Reporter
	Description • 17 years ago

graphs.mozilla.org/dgraph.html

Does an awesome job by letting us look at individual page run results.  In order to better diagnose perf issues it would be useful to have the following:

1) The ability to graph the performance of a particular page over time - so just like graphs.mozilla.org/graph.html for a single page.   

2) The ability to compare two sets of page runs.  A common comparison is trying to compare two builds and understand what's changed - so it would be great to be able to graph two builds against each other and order the results by biggest delta between them rather than page number.  So the first point on the left is the page run with the biggest delta between the two builds, next point over is the next biggest delta, etc.

	Mike Schroepfer Reporter
	Comment 1 • 17 years ago

Trev/Jeremy - let me know if you need any more info on this bug.  #1 would be killer to get done...

	Jeremy Orem [:oremj] Assignee
	Comment 2 • 17 years ago

I don't really understand what I am looking at on dgraph.html.  

For example I select a branch, machine, test name, and test.  The tests seem to have times associated with them, but the graph has no labels on the x-axis.  What does the x-axis represent?

	Mike Schroepfer Reporter
	Comment 3 • 17 years ago

(In reply to comment #2)
> I don't really understand what I am looking at on dgraph.html.  
> 
> For example I select a branch, machine, test name, and test.  The tests seem to
> have times associated with them, but the graph has no labels on the x-axis. 
> What does the x-axis represent?
> 

The x-axis varies based on the test.  The easiest to understand is Tp:

http://graphs.mozilla.org/dgraph.html#spst=range&spstart=0&spend=392&bpst=cursor&bpstart=0&bpend=392&name=tp_loadtime&m1tid=2&m1bl=0&m1avg=0

In general dgraph is graphing all of the sub-tests within a test suite.  So in the above graph it is showing you the page load time per page (e.g. load time for cnn.com, msn.com, etc).  which make it easier to understand the composite Tp score (which is an average of the mean of load times for each of the pages).   

So this graph is useful if you are diving into test and trying to understand why the big number has changed.  Make sense? 

	Jeremy Orem [:oremj] Assignee
	Comment 4 • 17 years ago

I have a working version of #1 at http://oremj.khan-vm.mozilla.org/edgraph.html (must be on VPN).  Getting that data is a bit slow, because the tables are not really set up to generate this graph.

This is the query I am currently using:

    cur.execute("""
                    SELECT dataset_info.date,avg(dataset_values.value)
                        FROM dataset_info
                        JOIN dataset_extra_data 
                            ON dataset_extra_data.dataset_id = dataset_info.id
                        JOIN dataset_values
                            ON dataset_extra_data.time = dataset_values.time
                            AND dataset_info.id = dataset_values.dataset_id
                        WHERE 
                            (dataset_info.machine,dataset_info.test,dataset_info.test_type,dataset_info.extra_data,dataset_info.branch) = (SELECT machine,test,test_type,extra_data,branch from dataset_info where id = ? limit 1) 
                        AND dataset_extra_data.data = ?
                        GROUP BY dataset_info.date ORDER BY dataset_info.date
                """, (setid,extradata))

	Jeremy Orem [:oremj] Assignee
	Comment 6 • 17 years ago

I'm not sure I fully understand what is wanted on #2.  Would it be possible to explain what is supposed to happen in relation to one of the graphs that is already being generated?  

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 7 • 17 years ago

Hmm #2 is I think similar to bug 390757 that I filed last night; basically I think it's the same as the current dgraph display with solid colors, but sorted by biggest delta, as opposed to by page order.  So, for example, given two graphs, Blue and Orange,  I'd want to see something like:

B     O
BB   OB
OOBOOBB
OOOBBBB

that is, the pages where Blue is slowest would be at the start, with blue being the bar in the "back", and orange in the front.. where Orange is slowest, the should be the bar in the back and blue in the front.  So the graph would be sorted by pages where blue got slower, followed by pages where blue got faster.

This doesn't extend cleanly to > 2 graphs, but you could probably do it by just sorting the values for each particular page in order of slowest-to-fastest, then sorting by the slowest of each group.

	Stuart Parmenter
	Comment 8 • 17 years ago

I would actually prefer them to stay in a constant order.  Having them move around would be confusing.

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 9 • 17 years ago

I think the idea was to make it an option.

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 10 • 17 years ago

Comment on attachment 275184 [details] [diff] [review]
Adds features number 1

This looks ok, though I agree that the database isn't really set up for this type of info... leading to a bunch of extra work.

Maybe at some point we should split out the per-test data for each run into a table instead of just storing it as a blob?

	Jeremy Orem [:oremj] Assignee
	Comment 11 • 17 years ago

Checking in dumpdata.cgi;
/cvsroot/mozilla/webtools/new-graph/dumpdata.cgi,v  <--  dumpdata.cgi
new revision: 1.4; previous revision: 1.3
done
RCS file: /cvsroot/mozilla/webtools/new-graph/edgraph.html,v
done
Checking in edgraph.html;
/cvsroot/mozilla/webtools/new-graph/edgraph.html,v  <--  edgraph.html
initial revision: 1.1
done
Checking in getdata.cgi;
/cvsroot/mozilla/webtools/new-graph/getdata.cgi,v  <--  getdata.cgi
new revision: 1.9; previous revision: 1.8
done
Checking in js/TinderboxData.js;
/cvsroot/mozilla/webtools/new-graph/js/TinderboxData.js,v  <--  TinderboxData.js
new revision: 1.6; previous revision: 1.5
done
RCS file: /cvsroot/mozilla/webtools/new-graph/js/edGraphFormModule.js,v
done
Checking in js/edGraphFormModule.js;
/cvsroot/mozilla/webtools/new-graph/js/edGraphFormModule.js,v  <--  edGraphFormModule.js
initial revision: 1.1
done
Checking in js/graph.js;
/cvsroot/mozilla/webtools/new-graph/js/graph.js,v  <--  graph.js
new revision: 1.13; previous revision: 1.12
done
Checking in sql/schema.sql;
/cvsroot/mozilla/webtools/new-graph/sql/schema.sql,v  <--  schema.sql
new revision: 1.2; previous revision: 1.1
done

	Mike Schroepfer Reporter
	Comment 12 • 17 years ago

Can we get this pushed to stage/production?

	Mike Schroepfer Reporter
	Comment 13 • 17 years ago

(In reply to comment #7)
> Hmm #2 is I think similar to bug 390757 that I filed last night; basically I
> think it's the same as the current dgraph display with solid colors, but sorted
> by biggest delta, as opposed to by page order.  So, for example, given two
> graphs, Blue and Orange,  I'd want to see something like:
> 
> B     O
> BB   OB
> OOBOOBB
> OOOBBBB
> 
> that is, the pages where Blue is slowest would be at the start, with blue being
> the bar in the "back", and orange in the front.. where Orange is slowest, the
> should be the bar in the back and blue in the front.  So the graph would be
> sorted by pages where blue got slower, followed by pages where blue got faster.
> 
> This doesn't extend cleanly to > 2 graphs, but you could probably do it by just
> sorting the values for each particular page in order of slowest-to-fastest,
> then sorting by the slowest of each group.
> 


Yep that's basically right.  Trying to make it easy to find the pages with biggest deltas.

	Jeremy Orem [:oremj] Assignee
	Comment 14 • 17 years ago

(In reply to comment #12)
> Can we get this pushed to stage/production?
> 

Just pushed it to stage.  You can see the new graphs at http://graphs-stage.mozilla.org/edgraph.html

	Mike Schroepfer Reporter
	Comment 15 • 17 years ago

I'd love to pull some numbers out of production for this - any chance we can get it pushed to prod today?

	Jeremy Orem [:oremj] Assignee
	Comment 16 • 17 years ago

This patch depends on MySQL, so that patch will have to be pushed at the same time.  Downtime should be about 1 hour.

	Jeremy Orem [:oremj] Assignee
	Comment 17 • 17 years ago

Vlad, do you have any suggestions on completing part 2 of this bug?  Right now it looks like a large portion of the graphing javascript will have to be rewritten, because each graph cannot see the other datasets that are being displayed.  Is there a easy way to access block all graphs, compare datasets, then show all graphs?

As far as ordering goes my idea is to compare each data set to every other dataset average the deltas and order by the greatest averages.

For example:
A = {myspace:3,google:4,yahoo:12}
B = {myspace:5, google: 1, yahoo:2}
C = {myspace:1, google: 8, yahoo: 4} 

First loop:

google  = abs(A[google] - B[google])
google += abs(A[google] - C[google])

myspace  = abs(A[myspace] - B [myspace])
myspace += abs(A[myspace] - C [myspace])

yahoo  = abs(A[yahoo] - B[yahoo])  
yahoo += abs(A[yahoo] - C[yahoo])  

second loop:

google += abs(B[google] - C[google])
myspace += abs(B[myspace] - C [myspace])
yahoo += abs(B[yahoo] - C[yahoo])  

order( google/3, myspace/3, yahoo/3 )

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 18 • 17 years ago

Well, the graphing code is generic -- it'll graph whatever it's given.  So you want to do the munging/sorting on the datasets first, and only then add them to the graph.  For dgraph, you basically just need to figure out a new order of sample points, and then apply that order to all datasets.

	Jeremy Orem [:oremj] Assignee
	Comment 19 • 17 years ago

This patch adds feature number two.  The graphs are generated quickly since all of the sorting is done in SQL.

	Jeremy Orem [:oremj] Assignee
	Comment 20 • 17 years ago

You can see that patch in action at http://oremj.khan-vm.mozilla.org/dgraph.html

	Jeremy Orem [:oremj] Assignee
	Comment 21 • 17 years ago

This is a good example of what the patch does: http://oremj.khan-vm.mozilla.org/dgraph.html#spst=range&spstart=0&spend=392&bpst=cursor&bpstart=0&bpend=392&name=tp_loadtime&m1tid=325&m1bl=0&m1avg=0&m2tid=11824&m2bl=0&m2avg=0

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 22 • 17 years ago

(In reply to comment #19)
> Created an attachment (id=280525) [details]
> Orders discrete graphs by delta (Feature 2)
> 
> This patch adds feature number two.  The graphs are generated quickly since all
> of the sorting is done in SQL.

Wait, why do we care about having the sorting done on the server side?  The sorting isn't expensive, and I'd rather not have the server doing any of this work -- the display shouldn't depend on the order stuff comes back from the server.

	Jeremy Orem [:oremj] Assignee
	Comment 23 • 17 years ago

Patch now sorts data client side instead of server side.

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 24 • 17 years ago

Comment on attachment 282488 [details] [diff] [review]
Orders discrete graphs by delta (Feature 2)

So I think this looks ok -- more comments in the requestDataSetFor function that was added would be good, to explain what's being done at each step along the way since it's pretty opaque at the moment.  Alice, can you glance at this as well?

	Jeremy Orem [:oremj] Assignee
	Comment 25 • 17 years ago

If Alice grants her review I'll add some comments to explain what is going on before I check in this patch.

	alice nodelman [:alice] [:anode]
	Comment 26 • 17 years ago

From the linked graph it looks like the stats displayed at the bottom of the page have been lost (the average/max/min for each data set being graphed).  I don't know if people were actually using that data, so it can probably go away - but then the header should be cleaned up so that it doesn't look like there is data missing.

	Jeremy Orem [:oremj] Assignee
	Comment 27 • 17 years ago

Good catch.  That problem is fixed now.

	alice nodelman [:alice] [:anode]
	Comment 28 • 17 years ago

Playing with it a little more I have another question.  While this makes the data for tp_loadtime look great, it makes the graphs for tp_Private Bytes and tp_Working Set take upwards of forever to load, and also loses the graphs pretty arc as it shows memory footprint growing over the course of the run of the test.  So, it appears like this is only a reasonable way of displaying a certain type of discrete graphs.

Perhaps this should be made into a check box?  I fear that we will be balancing the gain that we are making for tp_loadtime graphs with the loss of other graphs.

	alice nodelman [:alice] [:anode]
	Comment 29 • 17 years ago

Actually, attempting to graph a tp_Private Bytes vs. tp_Private Bytes raised a couple of exceptions and never graphed.

	Jeremy Orem [:oremj] Assignee
	Comment 30 • 17 years ago

You should no longer see exceptions or loading that takes forever.  With this resolved is the ordering still a problem?

	Jeremy Orem [:oremj] Assignee
	Comment 31 • 17 years ago

It seems like this would still be a good ordering for all discrete graphs since someone would typically, afaik, want to look for large deltas between different tests.

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 32 • 17 years ago

well, private bytes/working set is over time, right?  It doesn't make sense to me to do any sorting over time.  Tp is over a non-time index at which point the order doesn't matter, which is why we can get away with sorting (and why sorting is useful).

	alice nodelman [:alice] [:anode]
	Comment 33 • 17 years ago

Here's with the changes:
http://oremj.khan-vm.mozilla.org/dgraph.html#spst=range&spstart=0&spend=1851&bpst=Cursor&bpstart=0&bpend=1851&name=tp_Private%20Bytes&m1tid=4019&m1bl=0&m1avg=0&m2tid=4041&m2bl=0&m2avg=0

Here's without:
http://graphs.mozilla.org/dgraph.html#spst=range&spstart=0&spend=2586&bpst=cursor&bpstart=0&bpend=2586&name=tp_Private%20Bytes&m1tid=3&m1bl=0&m1avg=0&m2tid=23&m2bl=0&m2avg=0

In the second graph you can (sort of) see when cycle collection occurs along with the arc as the memory footprint grows over the course of the test.  I think that by sorting this graph you are losing that data.

As vlad says, those graphs are actually time based and don't make sense when sorted.

	Jeremy Orem [:oremj] Assignee
	Comment 34 • 17 years ago

How can I tell which ones are over time and which are not?

	alice nodelman [:alice] [:anode]
	Comment 35 • 17 years ago

We don't currently store that information - since before we didn't have any other choice but to display the data sequentially it wasn't an issue.

	Jeremy Orem [:oremj] Assignee
	Comment 36 • 17 years ago

What would be the best way to decide how to sort the graph?

	alice nodelman [:alice] [:anode]
	Comment 37 • 17 years ago

I think that without schema work, the best way would be to have a check box - much like we have for auto-scaling or adding an average line.  That would leave it up to the user which form of the graph (sorted/unsorted) that they find most useful to look at.

	Jeremy Orem [:oremj] Assignee
	Comment 38 • 17 years ago

I added a checkbox to the form for "Sort by greatest delta".

	alice nodelman [:alice] [:anode]
	Comment 39 • 17 years ago

I like this a lot better, my final concern is that you have to re-graph to switch between sorted/unsorted.  The auto-scaling can be flipped back and forth without re-graphing - which is a pretty nice feature.

Unless vlad objects, I don't think that that would be enough to stop the patch.  Probably worthy of filing a separate bug to figure out later.

	Jeremy Orem [:oremj] Assignee
	Comment 40 • 17 years ago

I added auto re-graphing when that checkbox gets clicked.

	alice nodelman [:alice] [:anode]
	Comment 41 • 17 years ago

Comment on attachment 282488 [details] [diff] [review]
Orders discrete graphs by delta (Feature 2)

I'm approving the current changes that I see on the test server that has the ability to switch between sorted/unsorted.

	Jeremy Orem [:oremj] Assignee
	Comment 42 • 17 years ago

Checking in dgraph.html;
/cvsroot/mozilla/webtools/new-graph/dgraph.html,v  <--  dgraph.html
new revision: 1.8; previous revision: 1.7
done
Checking in getdata.cgi;
/cvsroot/mozilla/webtools/new-graph/getdata.cgi,v  <--  getdata.cgi
new revision: 1.10; previous revision: 1.9
done
Checking in js/TinderboxData.js;
/cvsroot/mozilla/webtools/new-graph/js/TinderboxData.js,v  <--  TinderboxData.js
new revision: 1.7; previous revision: 1.6
done
Checking in js/graph.js;
/cvsroot/mozilla/webtools/new-graph/js/graph.js,v  <--  graph.js
new revision: 1.27; previous revision: 1.26
done

graphs-stage is also updated.

	Reed Loden [:reed]
	Comment 43 • 17 years ago

Can you please attach the new patch to the bug since what is attached isn't what was landed? That's what should have been reviewed...