I have a theory (from bug 908888) that our weekend failure rate is noticeably less than that of the weekdays. We have http://brasstacks.mozilla.com/orangefactor/index.html which graphs total failures/pushes for each day. This is useful, but it only deals with starred failures, so we have less data to work with and maybe not as accurate. also the retries/etc. are not necessarily in there. here is an example of a graph with weekends highlighted: http://www.flotcharts.org/flot/examples/visitors/index.html

Sir, I want to work on this as my first bug. Can you please let me know how to get started? Thanks!

to get started on this bug, first clone the repository: https://github.com/dminor/ouija make sure you can run it locally and see a webpage on your localcomputer. Next, I would create a new page- weekday.html or something like that- it could be a copy of an existing page (http://54.215.155.53/slave_failures.html). Summarize by platform (machine names denote the type) and figure out the pass rate and failure rate (total number of jobs vs total number of failing jobs). Once you have this data per platform, you can build a set of arrays and pass it into flot to display over time.

I saw in irc you have it setup locally, lets try to map the next steps out: 1) in src/server.py, look at run_resultstimeseries_query, this gets all results for a given platform but orders it by duration 2) create a function similar to run_resultstimeseries_query where you summarize the failures,totals for each day, ideally per platform 3) hook it up so you can get a list of these values from the server via a json endpoint (again use run_resultstimeseries_query as an example) 4) now create a new html page. I would copy something like static_results_timeseries.html as that already has the query and flot graphing. 5) once you have the data graphed, you can tweak the flot graph to highlight the weekends.

Sir, I also want to work on this as my first bug. I have a problem setting it up. It shows "not found" error on localhost server. This is from handler404. Please share your irc handle for further guidance from your side.

I am :jmaher on irc, next week would be a better time to chat on IRC. Do you have apache setup and the apache config in place? Have you restart apache after doing so?

I copied the file from the repo and had made the neccessary changges in paths. It was probably the case that I wasnt passing it the correct URL.However, the pages /data/slaves and /data/results are working fine, but the page /data/platform gives this error: A server error occurred. Please contact the administrator. The log on the terminal being: Traceback (most recent call last): File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run self.result = application(self.environ, self.start_response) File "./server.py", line 375, in application response_body = request_handler(query_dict) File "./server.py", line 42, in wrapper result = func(*args, **kwargs) File "./server.py", line 264, in run_platform_query platform = query_dict['platform'] KeyError: 'platform' 127.0.0.1 - - [15/Feb/2014 02:18:53] "GET /data/platform HTTP/1.1" 500 59

have you tried going to the static webpages? i.e. localhost/android_failures.html ? Those call the backend queries.

No, the static ones are not opening. the config is still not in place. I merely copied the default file from the repo to my apache sites-available dir, making the neccessary changes in urls. Then i restarted apache. But still came the handler404 error "not found"

you will need to make sure the config has the proper paths for your system.

Hello, i have been setting this up. I have it running, already got the data grouped by result, day and platform. For the graph, im finding that the results_timeseries.html is not using flotcharts but just Charts.min.js. The charts should be moved to flotcharts (so the highlighted weekend can be applied)?. Or stay with Charts.js and find another way to display the weekend data?.

We really need to pick somebody to work on this so we don't have 3 people working on the same bug. if we can do weekends with Charts.min.js, lets figure it out. Otherwise we could switch to flot. I had mentioned flot as an example as I knew what could be done.

I have modified the MySQL query in get_resultstimeseries_query function to group the data by day with total number of failures and total number of jobs per given day.I have added a function similar to get_resultstimeseries_query which basically sends the data to plot a graph between "Total Number of Jobs vs Days" and "Total Number of Failure Jobs vs Days". Though, I am not very sure whether this is what is required.

I have assigned this to Srinath R, thanks for working on this. Have you had the opportunity to plot this on a graph? We will need to keep the other pages/graphs as they are, so most likely we will need to have copied get_resultstimeseries_query to a new query or modified the pages that use that data.

Yes. I copied the results_timeseries HTML to create a new HTML file which uses Chart.min.js to plot a graph between "Total Number of Jobs vs Days" (highlighted as blue) and "Total Number of Failure Jobs vs Days" (highlighted as orange). I have only have four days in the database though (from 2014-02-03 to 2014-02-14). I haven't disturbed other pages (just created another function similar to get_resultstimeseries_query ). The new page I created plots the graph for "android2.2" and "android4.0". Before I highlight the weekends can you please let me know whether I am approaching it in the correct direction.

I couldn't implement highlighting the weekends in Chart.js ,so I used flot instead for this task alone. I have added another function get_results_flot_day_query similar to get_resultstimeseries_query which sends the data in the form flot requires. The resultant graph is highlighting the weekends as shown here http://www.flotcharts.org/flot/examples/visitors/index.html.

thanks for making it work with flot so the weekends can be highlighted easier. I had originally thought of graphing total_failures/total_pushes as a line, but graphing both lines would be more interesting! I believe you are heading in the right direction- lets just make this work for all platforms (we can do that after you are comfortable with the android platforms.

Comment on attachment 8377863 [details] [diff] [review] Have added the feature for plotting total failures/pushes per day graph using flot with weekends highlighted. (Platforms are Android2.2 and Android4) Review of attachment 8377863 [details] [diff] [review]: ----------------------------------------------------------------- overall I like this, thanks for the patch and making this work so well! here is what I would like to add: 1) all platforms 2) the total jobs vs failures is a large difference, so much that the orange line is not visible. I would like to adjust the axis, so the left side is total jobs and the right side is total failures, this can be done in flot with multiple axes: http://www.flotcharts.org/flot/examples/axes-multiple/index.html maybe a single graph where we turn on/off each given platform? (flot example: http://www.flotcharts.org/flot/examples/series-toggle/index.html) My thoughts are to fix the comments in this bug, and we can file followup bugs to tweak this and make it more useful. r- as I would like to solve 1+2 above. ::: src/server.py @@ +2,4 @@ > > import json > import re > +import calendar I am not sure if calendar is available on all distributions, is there a specific version required? ::: static/js/flot_chart.js @@ +1,1 @@ > +function renderChart(raw_data,targetParent){ nit, space before the {, and 4 space indent, not 8 @@ +3,5 @@ > + var markings = [], > + d = new Date(axes.xaxis.min); > + > + // go to the first Saturday > + d.setUTCDate(d.getUTCDate() - ((d.getUTCDay() + 1) % 7)) nit, please add a semicolon to the end of this statement. @@ +15,5 @@ > + // extends to infinity upwards and downwards > + > + do { > + markings.push({ xaxis: { from: i, to: i + 2 * 24 * 60 * 60 * 1000 } }); > + i += 7 * 24 * 60 * 60 * 1000; can we make: let day = 24*60*60*1000; do { markings.push({ xaxis: { from: i, to: i + 2*day } }); i += 7 * day; }

Comment on attachment 8379934 [details] [diff] [review] This patch includes - Plotting total failures on the right side of the graph so that it is not overshadowed by total jobs. Enabling the graph for all platforms on a single graph via radio buttons. Review of attachment 8379934 [details] [diff] [review]: ----------------------------------------------------------------- Would it be possible to get this patch without the entire flot tree? I would recommend pulling in the specific flot js files, i.e. jquer.flot.XYZ.js, and if there is not a license header, add the license.txt from flot.

Comment on attachment 8380755 [details] [diff] [review] Same as last patch but the entire flot JS packages are not included. Only 3 packages which are used in the code are included. Review of attachment 8380755 [details] [diff] [review]: ----------------------------------------------------------------- really close, lots of little nits. Please ensure that blank lines are blank, and that you adhere as best as possible to the style of the existing files. For new files, add spaces between things as indicated in a few comments. A lot of little details, lets clean those up and try one more time. Thanks for making the patch much simpler. I have followed most of the logic so far. ::: src/server.py @@ +195,5 @@ > + > + cursor.close() > + > + data_platforms[platform] = {'data': data, 'dates': get_date_range(dates)} > + nit: 173,176,197,199 all have spaces/tabs while the lines are blank, make them blank lines. ::: static/js/get_data_flot.js @@ +1,1 @@ > +//ths function is used to get the data for all platforms from the python webserver. change "//ths" to "// This" @@ +1,2 @@ > +//ths function is used to get the data for all platforms from the python webserver. > +function getData(platforms){ nit: space between ){ @@ +7,5 @@ > + url: '/data/results/flot/day', > + type: 'get', > + dataType: 'json', > + async: false, > + success: function(data){ nit: space between ){ @@ +18,5 @@ > +//this function is used to convert the data obtained from getData into a format required by flot. > +function getDataSets(data){ > + var datasets = {}; > + > + $.each(data,function(platformName,platformValue){ nit, add a space between ){, and after the , nit, please remove the spaces on the blank line (lines 21, 25) @@ +25,5 @@ > + > + var failures = { > + color: 'orange', > + data: platformValue.data.failures, > + label: 'failures'+ '-' + platformName, nit: space between '+ @@ +27,5 @@ > + color: 'orange', > + data: platformValue.data.failures, > + label: 'failures'+ '-' + platformName, > + lines: {show:true, color:'orange'}, > + yaxis:2 nit: space between :2 @@ +36,5 @@ > + label: 'total jobs' + '-' + platformName, > + lines: {show:true, color:'blue'}, > + }; > + > + datasets[platformName].data = [totals,failures]; nit: space after comma: totals,failures ::: static/js/plotGraph.js @@ +60,5 @@ > + grid: { > + markings: weekendAreas > + } > + }; > + 36,48,64 all have spaces on a blank line ::: static/results_day_flot.html @@ +14,5 @@ > + //getData is a function defined in get_data_flot.js. It sends a get request to the python webserver to get data for all platforms > + var data_platform = getData(); > + > + var datasets = {}; > + nit: remove spaces from blank lines on lines 16, 18

Comment on attachment 8380822 [details] [diff] [review] Fixed the whitespace errors. Added minified versions of flot JS packages. Review of attachment 8380822 [details] [diff] [review]: ----------------------------------------------------------------- please upload a new patch with the few nits addressed and we can check this in and deploy! ::: static/js/plotGraph.js @@ +37,5 @@ > + > + var choiceContainer = $("#choices"); > + > + $.each(datasets,function(key,val) { > + if(key === 'android4.0'){ nit: add a space between if( and between ){ @@ +67,5 @@ > + function plotAccordingToChoices() { > + var data = []; > + choiceContainer.find("input:checked").each(function() { > + var key = $(this).attr("value"); > + if(key && datasets[key]){ nit: add a space between if( and ){ @@ +77,5 @@ > + $('#startDate').text(data_platform[key].dates.startDate.split(" ")[0]); > + $('#endDate').text(data_platform[key].dates.endDate.split(" ")[0]); > + }); > + > + if(data.length > 0){ nit: add a space between if( and ){ ::: static/results_day_flot.html @@ +5,5 @@ > + <link rel="stylesheet" type="text/css" href="/css/style.css"> > + <script src="http://code.jquery.com/jquery-2.1.0.min.js"></script> > + <script src="js/flot/jquery.flot.min.js"></script> > + <script src="js/flot/jquery.flot.time.min.js"></script> > + <script src="js/flot/jquery.flot.selection.min.js"></script> is this a diff against your previous patch? we already have jquery.flot.*.js ? @@ +14,5 @@ > + // getData is a function defined in get_data_flot.js. It sends a get request to the python webserver to get data for all platforms > + var data_platform = getData(); > + var datasets = {}; > + > + // Probably there is a better way to do this without using setTimeout. Though I am doing every get request synchronously,somehow the graph is not getting plotted if I don't use setTimeout. Should figure out the problem and do it in a better way without using setTimeout. thanks for the comment.

Comment on attachment 8381480 [details] [diff] [review] Addressed the required nits in the last patch. Review of attachment 8381480 [details] [diff] [review]: ----------------------------------------------------------------- thanks!

are you sure this latest patch is a full patch? for example it references results_day_flot.html which doesn't exist in the ouija repository. Can you get me an entire diff from the existing source tree in one single patch?

results_day_flot.html is a new file I have added. That is the page which renders the graph. It was not there previously (in the origina ouija repository). I have already included results_day_flot.html in the patch.

the last patch doesn't apply, it shows modifications to results_day_flot.html. Are there multiple patches here required to get your changes?

I am sorry , I didn't understand what you mean. My latest patch modified results_day_flot.html. I did "git diff <commit from my first patch which rendered the graph for Android 2.2 and Android 4 alone> HEAD". results_day_flot.html has been modified from the first patch I uploaded to the most recent one.

Comment on attachment 8382346 [details] [diff] [review] A fresh patch which includes all the changes from the beginning. Review of attachment 8382346 [details] [diff] [review]: ----------------------------------------------------------------- thanks ,this is exactly what we need.

on github: https://github.com/dminor/ouija/commit/60c0bffad8a9177ad7a1b0b4628965311a215179

this is live: http://54.215.155.53/results_day_flot.html thanks for your hard work on this patch!