(not currently active) Ted Mielczarek Assignee
	Description • 16 years ago

We'd like to be able to run our tests on an arbitrary build. One way to do this is to build a normal build, with --enable-tests, and then upload the build and some other bits (test harnesses, maybe test files) so someone else could download those and run the tests on the build. If we choose to do this, we should ensure that enabling tests on our build machines doesn't include any extra code that we normally wouldn't ship. We might also need to audit the packaging manifests for Mac, where we don't have an explicit list of files to package.

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 1 • 16 years ago

Even if we confirm this is ok today, one concern is that it might become untrue in future... is there any way to automatically check this in ongoing basis?

	(not currently active) Ted Mielczarek Assignee
	Comment 2 • 16 years ago

We could probably do something in an automated fashion to ensure it doesn't creep into source files, but I honestly think this is less of an issue if we actually start shipping builds that were built --enable-tests. At that point, people will clearly be able to see what's shipped in the nightlies, and we'll make it clear to people that they're built --enable-tests, so I don't really expect problems.

	Jeff Walden [:Waldo]
	Comment 3 • 16 years ago

Or perhaps just remove ENABLE_TESTS entirely so that you don't do anything to build that code, it's just always built?  I prefer that to keeping --enable-tests optional.

	(not currently active) Ted Mielczarek Assignee
	Comment 4 • 16 years ago

Not an unreasonable idea. We'd have to fix a few things first. (reftest breaks universal builds, for example.)

	Phil Ringnalda (:philor)
	Comment 5 • 16 years ago

Yeah, Mac packaging would need rather a lot of additions to NO_PKG_FILES. And the thought that we wouldn't add test files without remembering to remove them from Mac builds is pretty funny, when you consider how often we add non-test things we need, and fail to remember to add them to packages files.

	(not currently active) Ted Mielczarek Assignee
	Comment 6 • 16 years ago

Wow, that's even worse than I thought. I agree, we can't even remember to add files that the build needs to the packaging manifests, and we clearly don't keep up on the NO_PKG_FILES list.

I think we'll need to make Mac use a packaging manifest like Linux and Windows to make this a viable option. We'll still have the problem of people forgetting to package files, but at least it will be consistent across platforms.

	Mark Banner (:standard8)
	Comment 7 • 15 years ago

(In reply to comment #6)
> I think we'll need to make Mac use a packaging manifest like Linux and Windows
> to make this a viable option. We'll still have the problem of people forgetting
> to package files, but at least it will be consistent across platforms.

A thought occurs to me. The comm-central tinderboxes have a package-compare step that compares the files listed in the package file to those on disk.

If we combined it with a test-files file (i.e. these files are test only), we could then tidy it up and enhance it and then when deviations are detected the tinderboxes turn orange (or red, or whatever colour your fancy except green or yellow).

	(not currently active) Ted Mielczarek Assignee
	Comment 8 • 15 years ago

I clicked through every result in the MXR link in the URL, and verified that nowhere do we do anything more serious than "DIRS += tests" conditional on ENABLE_TESTS.