We're having requests coming in to mash up the status data around our localizations. As that is really intended to be a mash-up of multiple datasources, we'll need some kind of report on what's happening in litmus. My suggestion is to add a query to json.cgi to return, for - a given product - a given branch - either -- a given timeframe (start-end, and last-day/week/month) -- or a testday_id the number of completed tests per locale, which is probably a tuple: tests run, unique test cases that fail, undecided and pass. Maybe the number of testers, too. I expect the result to be cached on the mash-up side, probably after some post-processing, too. So I would worry less about the complexity and load of the actual queries, those would be run from a cron job. Tim, I'd expect to hook up 'tests run in the last month' initially, does that sound like a good thing to know for you? Given the tuple of counts, we should get an idea of coverage, success, and from the total of test results, an idea of the strength of the litmus based test community. These are the queries I come up with, and that'd be interesting to me. Looking at this from the perspective of those that want to see the mash-up, is this reasonable to implement in a short timeframe, and if so, what'd be 'short'?

(In reply to comment #0) > Looking at this from the perspective of those that want to see the mash-up, is > this reasonable to implement in a short timeframe, and if so, what'd be > 'short'? Depends on how you want this info displayed. If you really only need to access the data via JSON, I can do that pretty quickly -> probably a few hours worth of effort. Prettying it up for display within Litmus (which makes sense to me, but maybe not on the first pass) will take a little longer.

So far, I only need json, or some machine readable format.

The JSON side is done, so you can grab your reports now if you want. The db load is negligible, so I haven't bothered with any caching. Here are some examples so you can figure out the syntax: https://litmus.mozilla.org/json.cgi?l10n=1&product_id=1&branch_id=15&locale=fr https://litmus.mozilla.org/json.cgi?l10n=1&product_id=1&branch_id=15&testday_id=74&locale=de https://litmus.mozilla.org/json.cgi?l10n=1&product_id=1&branch_id=15&start_timestamp=20071214070000&finish_timestamp=20071214170000 https://litmus.mozilla.org/json.cgi?l10n=1&product_id=1&branch_id=15&start_timestamp=2%20months%20ago&finish_timestamp=1%20month%20ago Syntax essentials: * l10n=1 : (required to get l10n aggregate stats) * product_id=# : (required) can be looked up through the Litmus admin interface if not known * branch_id=# : (required) can be looked up through the Litmus admin interface if not known * locale=? : (optional) will narrow your results to a specific locale, otherwise all matching locales will be returned. * one of: ** testday_id=# : (optional) will cause a testday object to be included in your results. The start and finish timestamp for the testday will be used to bound your query. ** start_timestamp=? and finish_timestamp=? : (optional) parses date arguments using the perl Date::Manip syntax (http://search.cpan.org/dist/Date-Manip/Manip.pod#EXAMPLES). Easiest to submit these dates as YYYYMMDDHHmmss, but whatever you want within the Date::Manip syntax framework should work. ** nothing : by default, the query will return the results matching the other criteria for the previous month. Shall I slap a UI on this?

Axel: had a chance to try this out? Do you need a GUI for this near-term?

Sorry for the lag, I doubt I'll need a GUI anytime soon. As we have the l10n fx3 tests, https://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=7&branch_id=19&testgroup_id=89&subgroup_id=883, is it possible to restrict the search to that set? We might be interested in platform coverage, too. And yes, I know that I'm totally changing the spec here, sorry.

coop, can you help us here? It'd be a lot easier to use the stats here than to do this manually for checking in to our pre-RC1 test coverage. Thanks

Hrm. Something seems to be broken here. Shouldn't the following two queries give matching results? https://litmus.mozilla.org/search_results.cgi?order_by_created=&order_by_testcase_id=&order_by_product=&order_by_platform=&order_by_branch=&order_by_locale=&timespan=-14&summary=&product=7&platform=&branch=19&locale=&limit=100 https://litmus.mozilla.org/json.cgi?l10n=1&product_id=7&branch_id=19&start_timestamp=-14 Note, the first query returns 36 results, so the limit is big enough. For Japanese and Polish, the test numbers are not consistent with itself, for example, "pl":{"num_results":7,"num_unclear":0,"num_testers":1,"num_pass":4,"num_fail":0} reports, as I read it, 7 results, 4 of which passed, none fail, non unclear. The html query shows 12 test results, one of which failed.

Zach, coop seems to be traveling, could you have a quick look? Maybe I'm just flawed, or it's easy to fix, in the latter case, you could save us a bunch of manual book keeping work. Thanks.

I'll have a look this week, but a 2-month lag between bug entries means other stuff has more priority right now.

Two things: 1) I've made a change to json.cgi to explicitly set the finish timestamp as NOW() if you don't provide it. This will help bound the query. 2) If you're going to use the -14 syntax in your query, please give it a unit to work with, e.g. https://litmus.mozilla.org/json.cgi?l10n=1&product_id=7&branch_id=19&start_timestamp=-14%20days

https://litmus.mozilla.org/search_results.cgi?order_by_created=&order_by_testcase_id=&order_by_product=&order_by_platform=&order_by_branch=&order_by_locale=&timespan=-14&summary=&product=7&platform=&branch=19&locale=fr&limit=100 shows 33 results, with at least 16 unique testcases and two testers (v on win, Cedric on linux), and https://litmus.mozilla.org/json.cgi?l10n=1&product_id=7&branch_id=19&start_timestamp=-14%20days&locale=fr reports {"fr":{"num_results":6,"num_unclear":0,"num_testers":1,"num_pass":3,"num_fail":0}} Same for https://litmus.mozilla.org/json.cgi?l10n=1&product_id=7&branch_id=19&start_timestamp=04/01/2008&end_timestamp=04/07/2008&locale=fr, that should include April 5th, right? Which is where the French testing happened.

This is because there are testcases that belong to Firefox/3.0 Branch being run as part of the Firefox l10n Localizers/3.0 l10n Branch testing. I don't know why we needed a new product/branch for l10n testing when simply creating an l10n test run would have sufficed. Anyway, it's non-trivial to unravel the SQL to separate the two. I had hoped for a quick fix, but I'll have to wade into this on Monday. Adding an option to limit by the l10n test run *should* return matching data here.

I guess Tony is the right guy to answer comment 12?

Tony, Chris, could you come up with a joint idea on how to organize the testcases and then how to get the right queries up?

yeah, tony and I started kicking around some ideas on this yesterday. he set me up with a litmus admin account and I've been poking at the results. I think the product/branch might be a way that we can surface the tests we want run to localization teams a bit easier in the tester/extension and litmus home page UI. if there is a way for a "testrun" to be created and surfaced to all localization team tester then testrun's would work. something like this pseudo code might give us lists of how many tests run, and then we could do another pass looking for tests that failed to follow up on. There are 18 test cases in the firefox3-l10n-test-run so each locale/platform pass ought to return 18 results. foreach testrun in ( fx3-l10n-test-run(16) ) for each locale in http://mxr.mozilla.org/mozilla/source/browser/locales/shipped-locales for each platform in 1(windows),2(linux),13(mac intel) https://litmus.mozilla.org/advanced_search.cgi?test_run=TEST_RUN&platform=PLATFORM&locale=LOCALE&timespan=-14&end_date=No&limit=1000' for example https://litmus.mozilla.org/advanced_search.cgi?test_run=16&platform=1&locale=de&timespan=-14&end_date=No&limit=1000' shows me 18 test cases run/passed on de/windows in the last 14 days 2008-04-11 15:23:55 120823 4366: Sidebar Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120822 4368: Preferences/Options Window Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120821 4367: Menu Items/Context Menus Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120820 5186: Default Search Engine List Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120832 4361: About dialog Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120819 4364: Bookmarks Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120831 4362: Help & Support Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120818 5184: Default Bookmarks in Toolbar Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120830 4365: Surf's up! Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120829 3964: Basic Printing Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120828 4314: Show Cookies Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120827 5183: Password Manager completion Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120826 4372: Find toolbar Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120825 5182: Downloads manager: context menu Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:55 120824 4363: Addons Manager Tabs Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:54 120817 5185: Default Bookmarks in Bookmark Library Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:54 120816 5187: Default Start Page and First Run Startup Firefox l10n Localizers Windows 3.0 l10n Branch de 2008-04-11 15:23:54 120815 4357: Installation, New Profile, Migration Firefox l10n Localizers Windows 3.0 l10n Branch de There are some things that need a bit more investigation. For example when I drill down into the test results for one of the test cases like https://litmus.mozilla.org/single_result.cgi?id=120823 I see Product: Firefox Platform: Windows Operating System: Windows XP Build ID: 2008041005 Locale: German (de) User Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; de; rv:1.9b5) Gecko/2008032619 Firefox/3.0b5 which makes me think the user might have entered windows as the platform they ran the tests on, but the user agent reported the test was run on mac... both could be spoofed, but maybe we ought to go with user agent instead of what the user entered into the litmus test form...

I would think that it's perfectly fine to test in one build and fill out litmus in the other, even on another machine. Note that the number of tests run doesn't really make you cover all tests, you can run one test 50 times and that'd be it. But let's not assume malicious intent here, I think that total number of tests run per platform per locale is a good measure, and the total number of tests is a ballmark for the evaluation of the results. Beyond that, I'm all for relative measures, cooptetion ftw.

AFAICT, coop's question in comment 12 still needs an answer.

I guess its tony that needs to answer the question about why do we need a new product/branch for l10n testing? I'm not sure I can help there. I did have some additional thoughts about test case organization. Take a test case like https://litmus.mozilla.org/show_test.cgi?id=3948 == Standard Installation that is part of testgroup: 3.0 Basic Functional Tests (BFTs) what kind of changes might we need to "import/clone" that en-US version into a test for each locale and allow it to be translated and customized to be more suited to a bunch of brazillian, spanish, french, or german testers? In that way we we would have a default test in english but if you were running a german build BFT and there was a translated version of the installation test available, then that is what you would see? Guessing this might be hard, but it might be a goal we want to shoot for to engage a larger number of worldwide testers on localized builds. We would also need to recruit a larger number of translators to help get test cases localized. We can kick around this idea here in the bug and get some feedback from localization team leads on the feasibility of this proposal at the summit, as part of this discussion. http://wiki.mozilla.org/Summit2008/Sessions/Proposals/Localizing_Firefox_3#QA_Testing

(In reply to comment #18) > I guess its tony that needs to answer the question about > > why do we need a new product/branch for l10n testing? > > I'm not sure I can help there. > > I did have some additional thoughts about test case organization. > > Take a test case like > > https://litmus.mozilla.org/show_test.cgi?id=3948 == Standard Installation > > that is part of > > testgroup: 3.0 Basic Functional Tests (BFTs) > > what kind of changes might we need to "import/clone" that en-US version into a > test for each locale and allow it to be translated and customized to be more > suited to a bunch of brazillian, spanish, french, or german testers? > > In that way we we would have a default test in english but if you were running > a german build BFT and there was a translated version of the installation test > available, then that is what you would see? > > Guessing this might be hard, but it might be a goal we want to shoot for to > engage a larger number of worldwide testers on localized builds. We would also > need to recruit a larger number of translators to help get test cases > localized. > > We can kick around this idea here in the bug and get some feedback from > localization team leads on the feasibility of this proposal at the summit, as > part of this discussion. > > http://wiki.mozilla.org/Summit2008/Sessions/Proposals/Localizing_Firefox_3#QA_Testing > Hi there, i was not aware of the discussion here, so i apologize for not chiming in earlier. The original intent of creating a separate test run for localizers in a separate branch was for simplicity; outside the convoluted firefox 2.x and 3.x product branches. I wanted localizers to quickly access a localizer active run via the litmus page, and keep their test run simple to what would deem a build pass prior to their tree checkins. I also didnt want to stir any confusion between bfts and ffts and smoketests to those unfamiliar with the differences, so having a seperate branch seemed safe to me. Also, i was trying to differentiate what Mozilla QA testers and the community typically touch, versus a place for localizers to test with only. I also wanted to keep the branch open for any testcases that *might* not be part of Firefox 3.0, but relevant to localization. while i have yet to find an example of that, i didnt sense there was an issue to seperate it. Let me know if this really causes issues with aggregating reports. My reasoning for separating it was more for easier management and visibility, but if its causing major issues, we can work it back into branch.

This bug is not about localizing testcases, please don't add confusion to this bug. Tony, I think that coop's question is not so much about why we have a specific branch or not, but more about why only some of the testcases in the testrun are actually in that one branch, and others are in the other branch. At least that's how I read comment 12. I personally think that we should fork existing testcases for the l10n test runs to adapt them to l10n. Usually, the work steps for l10n tests are rather similar to functional tests, but the expected result is different. Whether that's making a case for a separate branch for those tests is something I don't know.

(In reply to comment #20) > I personally think that we should fork existing testcases for the l10n test > runs to adapt them to l10n. Usually, the work steps for l10n tests are rather > similar to functional tests, but the expected result is different. Whether > that's making a case for a separate branch for those tests is something I don't > know. I suggest you do whatever is going to make the most sense to testers in terms of naming and cloning testcases. My problem with is with how they are organized at higher levels, specifically that we have overloaded the product and branch categories with l10n-specific versions, e.g. Product:Firefox l10n localizers. The testcases that belong to these categories are still Firefox testcases, they just happen to be for l10n. We would be better served IMO by using the normal product/branch for these (e.g. Firefox/3.0 Branch) and then creating special l10n-specific subgroups/testgroups/test runs. I can do batch ops in the db if you can come up with a plan for reorganizing testcases, or think it's even necessary. Inertia is powerful, and we may not want to confuse testers who already know where to find these tests. Your call.

(In reply to comment #21) > > I can do batch ops in the db if you can come up with a plan for reorganizing > testcases, or think it's even necessary. Inertia is powerful, and we may not > want to confuse testers who already know where to find these tests. Your call. > Hey coop, okay, please go ahead and clone the Product:Firefox l10n localizer suite over to Firefox/3.0 Branch/L10n Localizers. It makes more sense like you said, for future naming and cloning. Thanks.

(In reply to comment #22) > Hey coop, okay, please go ahead and clone the Product:Firefox l10n localizer > suite over to Firefox/3.0 Branch/L10n Localizers. It makes more sense like > you said, for future naming and cloning. Thanks. The top-level l10n product and branches have now been remapped to the simpler Firefox equivalents. All existing references to those original products/branches have also been remapped, so nothing should have been lost. Existing test runs/testdays/testgroups/subgroups should be accessible in the same spots as before.

Axel: can you please revisit your queries above and see whether the numbers look right now?

Looks great Coop. thanks.

Sorry, reopening. https://litmus.mozilla.org/json.cgi?l10n=1&product_id=1&branch_id=15&testday_id=74&locale=de reports {"num_results":"159","num_unclear":"3","num_pass":"143","num_testers":"3","num_fail":"6"} which still seems to not match number-wise. The numpass doesn't match https://litmus.mozilla.org/testday_report.cgi?start_timestamp=20071214070000&finish_timestamp=20071214170000&product=1&branch=15&testgroup=&build_id=&locale=de either, which I figure it probably should.

Coop, can you investigate? raising Priority level for eyes

I actually ended up digging into the code a bit tonight, and I think I found something constructive: The total number of tests run is gotten at http://mxr.mozilla.org/mozilla/source/webtools/litmus/Litmus/DB/Testresult.pm#569 and the individual fail/pass/unclear numbers are gotten at http://mxr.mozilla.org/mozilla/source/webtools/litmus/Litmus/DB/Testresult.pm#602 There is actually a difference in the SQL queries, that is, in the latter case, we do a distinct(tr.testcase_id) before doing the count(), which explains the smaller numbers. Reading my initial comment, hrm, ooops, the mismatch in the numbers is actually what I requested. There is one thing still though that bothers me, the numbers are strings. So in js, if I do res.num_fail + res.num_pass + res.num_unclear I get a HUGE number, whereas Number(res.num_fail) + Number(res.num_pass) + Number(res.num_unclear) is much smaller ;-) I don't really understand why we end up with numbers. I did some preliminary testing, it seems that JSON.pm can do that, so I guess the sql statements return strings. Can we convert those values to numbers still?

Axel: how's this? https://litmus.mozilla.org/json2.cgi?l10n=1&product_id=1&branch_id=26&testday_id=120&locale=de I've included both distinct result numbers and totals, and they should all be numbers now. The db was returning everything as strings, so I'm adding 0 to everything (ugh) to cast them as numbers.

That's nifty. Marking FIXED, thanks a lot.