1) There are some performance tests that have self validation that does not have a correct "reference" value 2) If validation currently fails not all tests will be tracked as a failure when running via runtests.py

This patch was previously approved by dschaffe on bug 513229

untyped jsbench testcases will now properly self validate and if they fail validation they will not output the "metric" causing them to properly fail when run via performance/runtests.py

- important change here is the moldyn now actually works, previously was generating bogus calculations (NaN), now works properly

- important change here is the moldyn now actually works, previously was generating bogus calculations (NaN), now works properly

I am getting RayTracer failure on my desktop mac for typed and untyped. Have you seen it? Validation failed Pixel checksum = 302690 Reference value = 302532 Validation failed Pixel checksum = 302690 Reference value = 302532 test failed validation! jsbench/RayTracer.as no test result - test output: ['Validation failed\n', 'Pixel checksum = 302690\n', 'Reference value = 302532\n', 'Validation failed\n', 'Pixel checksum = 302690\n', 'Reference value = 302532\n', 'test failed validation!\n']

The following performance tests are failing verification: jsbench/Euler.as jsbench/FFT.as jsbench/RayTracer.as jsbench/typed/Euler.as jsbench/typed/FFT.as jsbench/typed/RayTracer.as sunspider/as3vector/s3d-cube.as

Simple patch that checks for validation failure in performance tests and then does not record metric when validation fails. Tests need to be fixed before submitting this patch as it will fail in the performance phase.

Comment on attachment 431365 [details] [diff] [review] Explicitly look for validation failures in tests, do not record results Should this code just be removed instead of just commented out? for f in out1: for line in f: print(line.strip()) Other patches will need to be scrubbed and rebased before this can land

Are we certain that all failures result in the string "Validation failed" ? For example, jsbench/LUFact.as prints a computed and a reference value which differ in my test run, but no explicit error message is displayed.

Rebased patch. RayTracer now has 2 'valid' results since Math.tan() produces different results on mac-intel which changes the checksum of the image (bug #594836) - important change here is the moldyn now actually works, previously was generating bogus calculations (NaN), now works properly

With the current set of patches self validating tests are working correctly and if self validation fails the runtests will not record a metric value (both because no metric is generated but also because "validation failed" is found in the output).

status?

This is a rebased and combined patch of previous patches. I have run this patch through the following configurations: mac 32/64, win32/64, linux 32/64 and android-arm

changeset: 6255:c69cfc4d7c51 user: Brent Baker <brbaker@adobe.com> summary: Bug 526234: enable self validation to fail a performance testcase and not record a metric (r=dschaffe) http://hg.mozilla.org/tamarin-redux/rev/c69cfc4d7c51

jsbench validation for LUFact, Moldyn and Raytrace are failing on linux32, this is not an injection and has been found to happen as far back as coral. This is being tracked via bug# 654498. Closing this bug since all available work/patches have been committed