The standalone CMS testing framework needs some updating to deal with new requirements imposed on it. At minimum, we need the following: 1-Support for tests of different types, rather than just CRC verification tests (We need these to test things like bug 460629). 2-Support for shorter test runs so that we can plug the framework into the tinderboxen without wasting 7 minutes of everyone's time on each checkin. 3-Support for less ad-hoc statistics and for tracking errors all the way through the execution of the tests. 4-The tests need to be plugged into the build. We then want to add tests for: 1-bug 460629 2-bug 458164 I've got a baseline patch that covers most of the framework changes except for plugging things in to be run automatically. I'll then put together patches for the new test cases we want to cover.

Added the baseline overhaul patch

Also note that I forgot to mention that one of the specific test configurations was causing 4 off-by-2 issues between the fixed path and the floating point path. Per se there's nothing incorrect about this, and it's always been like this, but it required the person running the tests to know what should happen and look for it. Since we want the tests to be more automatic, and we can count on having no off-by-two issues otherwise (which gives us a great deal of test coverage), we just remove that configuration.

Ideally we'd also like to add tests for bug 460520.

Comment on attachment 350192 [details] [diff] [review] Baseline overhaul patch to cover most of the infrastructural changes > # Color Management Test Directive File > # > # Format: > # InputProfileFilename OutputProfileFilename <CRC32 For Each Intent> > # Looks like you've changed the format - might want to change the documentation too :) r+ with that change.

So, we should maybe update this for QCMS?

yikes, I'd more or less forgotten about this. Jeff - this is more or less black-box testing, so it shouldn't be too hard to update for qcms. Let me know what you think.

Comment on attachment 350192 [details] [diff] [review] Baseline overhaul patch to cover most of the infrastructural changes I guess this is way out of date now. Luckily we don't need sr on it anymore!

reassigning to jeff.