In bug 717062 a regression was introduced, and found in bug 906682. Afaict we do not have a good way to write tests to prevent regressions likes this. We need some way to write regression tests.

Any ideas guys? just remove the needinfo if not.

The last time this cropped up (simple-storage bustage IIRC) I suggested that we needed a way to extend the test harness so we could pre-populate the test profile with some files. It was difficult to do with testpkgs but now we have testaddons I bet it might be simpler, include some special directory in the add-on that gets copied into the test profile or something to set up some data as if saved from a previous run of the add-on?

I believe issues are mainly caused by the fact that self module produces different pointers, either by changes in the harness-options.json or our interpretation, causing add-on to access wrong prefs branches, wrong files on the disk etc.. Looking at the tests for self: https://github.com/mozilla/addon-sdk/blob/master/test/test-self.js It's no wonder we don't catch these regressions because we just test for `id` and a `data.url` :( I tend to think that adding test to self with hardcoded values would catch most of the regressions as such.

(In reply to Irakli Gozilalishvili [:irakli] [:gozala] [@gozala] from comment #3) > I believe issues are mainly caused by the fact that self module produces > different pointers, either by changes in the harness-options.json or our > interpretation, causing add-on to access wrong prefs branches, wrong files > on the disk etc.. > > Looking at the tests for self: > https://github.com/mozilla/addon-sdk/blob/master/test/test-self.js > > It's no wonder we don't catch these regressions because we just test for > `id` and > a `data.url` :( > > I tend to think that adding test to self with hardcoded values would catch > most > of the regressions as such. I'm not sure I see how any test in the test-self.js file could have prevented the regression that I mentioned in comment #0