Joel Maher ( :jmaher ) (UTC -8) Reporter
	Description • 10 years ago

This bug is related to Bug 1093939 where we will build a list of tests which have results changing on weekends as compared to weekdays.

as documented in a blog post:
http://elvis314.wordpress.com/2014/10/30/a-case-of-the-weekends/

we have less data points on weekends, but also less noise in our tests.

There are many ways to figure out what is going on, but first we need to understand what our problem is before measuring.

In this bug we will need to query the number of test jobs run per day and detect a pattern.

So for example, take tpaint on linux32:
http://graphs.mozilla.org/graph.html#tests=%5B%5B82,131,33%5D%5D&sel=none&displayrange=30&datatype=running

One way to do this is to use the graph api (https://wiki.mozilla.org/Perfomatic:API) to query the specific test and history.  Then parse the data and build a list over time.  I am not sure if we should have total jobs run, or total changesets for the day.  I think total jobs run is what is important.

The end result will be an array of data points for a given time range (30 days) for each test/platform combination.

	r.phate@gmail.com
	Comment 1 • 10 years ago

I would like to work on this bug.

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 2 • 9 years ago

hi r.phathe, I actually believe this is solved.  This is my fault for not getting back to you many days ago.  If you are still interested in hacking, we have plenty of cool bugs to work on.  Just please let me know what languages and technologies or projects interest you.  For example we have a list of a*team related projects here:
https://wiki.mozilla.org/Auto-tools/Projects/Everything#Project_Table

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 4 • 9 years ago

Comment on attachment 8557966 [details] [diff] [review]
Added count  mode to report.py

Review of attachment 8557966 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks :adusca, this is great stuff.  I have some suggestions to make here which would help make the code clearer!

::: talos/scripts/report.py
@@ +26,5 @@
>  def get_tuple(test, platform):
>      return [(compare.test_map[test]['id'], get_branch(platform), compare.platform_map[platform], test, platform)]
>  
>  
> +def generate_report(tuple_list, filepath, mode='v'):

I would ensure we use more decriptive names:
variance
count

@@ +125,3 @@
>  
> +    else:
> +        generate_report(tuple_list, filepath=f + '.csv')

couldn't we just have two modes: variance, count, then default args.mode = 'variance', we could pass that in to generate_report and not need the if/else condition here.

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 6 • 9 years ago

Comment on attachment 8557988 [details] [diff] [review]
Added count  mode to report.py

Review of attachment 8557988 [details] [diff] [review]:
-----------------------------------------------------------------

looks great!

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 7 • 9 years ago

https://hg.mozilla.org/build/talos/rev/3f154d192726