$ ./mach test-info netwerk/test/unit/test_simple.js --verbose ===== netwerk/test/unit/test_simple.js ===== Found netwerk/test/unit/test_simple.js in source control. netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini flavor: xpcshell Querying ActiveData... 2018-10-22 10:38:02.532934 {"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-15"}}}, {"lt": {"run.timestamp": {"date": "2018-10-22"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"} 2018-10-22 10:38:03.017720 <Response [400]> Error running mach: ['test-info', 'netwerk/test/unit/test_simple.js', '--verbose'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: HTTPError: 400 Client Error: BAD REQUEST for url: https://activedata.allizom.org/query File "/home/gbrown/src/testing/mach_commands.py", line 515, in test_info self.set_test_name() File "/home/gbrown/src/testing/mach_commands.py", line 645, in set_test_name data = self.submit(query) File "/home/gbrown/src/testing/mach_commands.py", line 682, in submit response.raise_for_status() File "/home/gbrown/src/third_party/python/requests/requests/models.py", line 840, in raise_for_status raise HTTPError(http_error_msg, response=self)

needinfo me so I do not forget

gbrown, please verify this is no longer a problem.

Getting further, but still failing: $ ./mach test-info netwerk/test/unit/test_simple.js --verbose ===== netwerk/test/unit/test_simple.js ===== Found netwerk/test/unit/test_simple.js in source control. netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini flavor: xpcshell Querying ActiveData... 2018-10-23 08:24:54.644148 {"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-16"}}}, {"lt": {"run.timestamp": {"date": "2018-10-23"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"} 2018-10-23 08:25:02.682551 <Response [200]> Found records matching 'netwerk/test/unit/test_simple.js' in ActiveData. Test results for netwerk/test/unit/test_simple.js on mozilla-central,mozilla-inbound,autoland between 2018-10-16 and 2018-10-23 2018-10-23 08:25:02.734695 {"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "netwerk/test/unit/test_simple.js"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-16"}}}, {"lt": {"run.timestamp": {"date": "2018-10-23"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]} 2018-10-23 08:25:04.219446 <Response [400]> Error running mach: ['test-info', 'netwerk/test/unit/test_simple.js', '--verbose'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: HTTPError: 400 Client Error: BAD REQUEST for url: http://activedata.allizom.org/query File "/home/gbrown/src/testing/mach_commands.py", line 517, in test_info self.report_test_results() File "/home/gbrown/src/testing/mach_commands.py", line 713, in report_test_results data = self.submit(query) File "/home/gbrown/src/testing/mach_commands.py", line 682, in submit response.raise_for_status() File "/home/gbrown/src/third_party/python/requests/requests/models.py", line 840, in raise_for_status raise HTTPError(http_error_msg, response=self)

The query no longer fails, but the > "groupby": ["build.platform", "build.type", "run.type"] is still double counting

checking in here Kyle, is this getting closer to being fixed?

this is now working. :gbrown, please confirm and close

$ ./mach test-info testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html --verbose ===== testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html ===== Found testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html in source control. testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html not found in any test manifest! Querying ActiveData... 2018-11-05 12:38:58.747000 {"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["testing\\web-platform\\tests\\encoding\\legacy-mb-korean\\euc-kr\\euckr-decode-ksc_5601.html", "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"]}}, {"regexp": {"result.test": ".*testing\\\\web\\-platform\\\\tests\\\\encoding\\\\legacy\\-mb\\-korean\\\\euc\\-kr\\\\euckr\\-decode\\-ksc\\_5601\\.html.*"}}, {"regexp": {"result.test": ".*testing\\/web\\-platform\\/tests\\/encoding\\/legacy\\-mb\\-korean\\/euc\\-kr\\/euckr\\-decode\\-ksc\\_5601\\.html.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"} 2018-11-05 12:39:08.960000 <Response [200]> Unable to find matching records in ActiveData; using testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html! Test results for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05 2018-11-05 12:39:09.082000 {"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]} 2018-11-05 12:39:15.959000 <Response [200]> No test result data found. 2018-11-05 12:39:15.966000 {"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.ok": "T"}}, {"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "average", "name": "average", "value": "result.duration"}, {"aggregate": "min", "name": "min", "value": "result.duration"}, {"aggregate": "max", "name": "max", "value": "result.duration"}, {"aggregate": "count"}]} 2018-11-05 12:39:23.198000 <Response [200]> Test durations for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05 No test durations found. 2018-11-05 12:39:23.204000 {"limit": 1000, "from": "unittest", "where": {"and": [{"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "select": ["build.platform", "build.type", "run.type", "run.name"], "format": "list"} 2018-11-05 12:39:23.890000 <Response [200]> Test tasks for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05 No test tasks found. Bugzilla quick search for 'testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html,testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html': so this still doesn't work :(

maybe this is part of the ingestion backlog. I am working on increasing this ingestion speed.

Another example, that breaks in a different way: $ ./mach test-info test_simple.js --verbose ===== test_simple.js ===== Found netwerk/test/unit/test_simple.js in source control. Build configuration changed. Regenerating backend. No handlers could be found for logger "mozbuild.frontend.reader" netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini flavor: xpcshell Querying ActiveData... 2018-11-05 11:02:40.400358 {"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"} 2018-11-05 11:02:46.457527 <Response [200]> Found records matching 'netwerk/test/unit/test_simple.js' in ActiveData. Test results for netwerk/test/unit/test_simple.js on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05 2018-11-05 11:02:46.503368 {"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "netwerk/test/unit/test_simple.js"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]} 2018-11-05 11:03:02.023607 <Response [200]> Error running mach: ['test-info', 'test_simple.js', '--verbose'] The error occurred in the implementation of the invoked mach command. This should never occur and is likely a bug in the implementation of that command. Consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: KeyError: u'failures' File "/home/gbrown/src/testing/mach_commands.py", line 517, in test_info self.report_test_results() File "/home/gbrown/src/testing/mach_commands.py", line 724, in report_test_results failures = record['failures'] This is curious in that the query succeeds and returns results for the count column, but not for 'failures': { "from":"unittest", "format":"list", "limit":100, "where":{"and":[ {"eq":{"result.test":"netwerk/test/unit/test_simple.js"}}, {"in":{"build.branch":["mozilla-central","mozilla-inbound","autoland"]}}, {"gt":{"run.timestamp":{"date":"2018-10-29"}}}, {"lt":{"run.timestamp":{"date":"2018-11-05"}}} ]}, "groupby":["build.platform","build.type","run.type"], "select":[ {"aggregate":"count"}, { "aggregate":"sum", "default":0, "name":"failures", "value":{"case":[{"then":1,"when":{"eq":{"result.ok":"F"}}}]} } ] } "run" "build" "count" "{\"platform\":\"win64\",\"type\":\"debug\"}" 447 "{\"platform\":\"maxosx64\",\"type\":\"debug\"}" 441 "{\"platform\":\"linux32\",\"type\":\"debug\"}" 423 ...

New version deployed :gbrown, please test again. :jmaher is not getting any data, maybe you can see what's still wrong

Discussed on irc this morning. It looks like the only remaining issue is https://github.com/mozilla/ActiveData/issues/101.

I'll just work around the activedata issue to get this working again.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/fb4be273bff4 Allow for missing 'failures' data in 'mach test-info'; r=me,a=test-only