Periodically the build system seems to suddenly start ignoring any changes I make to some of the source files I am changing. This seems to happen often after a compile failure of some kind. Subsequently any changes to the file that failed will get ignored. This leads to me wasting lots and lots of time while I try to figure out why on earth code I'm changing has no impact in the build.

I have discovered that I can get out of this mess by removing the relevant .o file and running mach configure.

An example I just hit, the source file I was editing:

-rw-r--r-- 1 dave staff 13990 Apr 1 15:54 DocumentL10n.cpp

And the object file it was building into:

-rw-r----- 1 dave staff 3492564 Mar 29 10:19 obj-browser-dbg-full/intl/l10n/Unified_cpp_intl_l10n0.o

The source code is over two days newer than the compiled object.

In this state the build would complete with no errors and run just fine. After removing the object file and reconfiguring the build showed that the source wouldn't build at all dues to compile errors.

What's the content of the corresponding file in obj-browser-dbg-full/intl/l10n/.deps when that happens?

(In reply to Mike Hommey [:glandium] from comment #1)

What's the content of the corresponding file in obj-browser-dbg-full/intl/l10n/.deps when that happens?

Unfortunately of course while this bug is super-frustrating I don't know how to reproduce this so it might take a bit to get this :(

It did occur to me that maybe sccache is playing a part here, I'll try to figure out if that is the case too.

(In reply to Mike Hommey [:glandium] from comment #1)

What's the content of the corresponding file in obj-browser-dbg-full/intl/l10n/.deps when that happens?

Hit this again, this time with browser/components/pwa/remote/parent/RemotePWA.mm. There are no files in the .deps directory. I stopped the sccache server, deleted its cache, rebuilt and it still ignored my changes.

Part of what the problem is here is that make knows:

Unified_file.o: Unified_file.cpp

but it doesn't know about:

Unified_file.cpp: file1.cpp file2.cpp file3.cpp

We expect that the first time you run, make uses the former piece of knowledge to compile the object file, and then the .deps directory is filled with bits that tell us about the second part, which make will then use to force compilation in subsequent rebuilds when you compile.

I think, but am not certain, that what's happening is something like:

1. You compile with sccache.
2. sccache hits, and deposits the object files in your directory, but not the files we expect in .deps
3. You go to modify files that don't have corresponding .deps files.
4. Nothing happens.

It's not clear to me why this doesn't happen more often.

So a partial solution would be to explicitly write out the dependencies for:

Unified_file.o: file1.cpp file2.cpp file3.cpp

so make always mostly understands what's going on.

This is still not perfect, of course, because what happens if you modify .h files? And I don't know what the right thing to do on the sccache side is either. We should be caching everything the compiler writes?

Ok I can reproduce this now.

• I found a different directory that had its .deps directory correctly populated.
• I added an include for a non-existent header to one of the header files in there.
• Ran the build and it failed and the .deps directory was now empty.
• Subsequent compiles succeeded with no changes.

OK, so additional dependencies would not fix that particular scenario. But I think the solution outlined under "Handling unusual situations" at http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ would...at least partially. Ideally that would prevent clang/gcc (who are the ones responsible for deleting those files, AIUI) from nuking perfectly good dependency files, since we wouldn't copy the temporary files over unless compilation definitely succeeded.

Mossop points out over IRC that there's also -MG, which would add missing headers to the dependency file.

I guess -MG requires -M. I added them both to my build flags and it didn't delete the deps file and it included the missing header. But that also caused the build to succeed and not log anything about a missing header.

Also apparently -M makes the compile uncacheable by sccache :(

People can find themselves stuck in situations where the compiler has
deleted the files in .deps/ telling make about the dependencies
between unified objects and their constituent source files. This
situation means that touching the source files has no impact as far as
make is concerned: make only sees that the unified object has a
dependency on the Unified_foo.cpp file, and that file remains
untouched.

To counter this, let's explicitly declare the dependencies clang -MD (or similar) would have found. We can't do this for everything (in
the above scenario, touching headers will still fail to compile things),
but this is something straightforward to do without resorting to
makedepend or similar tricks.

I think I have an idea that would work for includes too. Let me try it.

(In reply to Dave Townsend [:mossop] (he/him) from comment #5)

Ok I can reproduce this now.

• I found a different directory that had its .deps directory correctly populated.
• I added an include for a non-existent header to one of the header files in there.
• Ran the build and it failed and the .deps directory was now empty.
• Subsequent compiles succeeded with no changes.

Actually, I'm confused. I don't reproduce this at all. What happens for me is that for every .deps file that disappears, the corresponding .o disappears too. So obviously, a subsequent compile still needs to build the .o...

(In reply to Mike Hommey [:glandium] from comment #11)

(In reply to Dave Townsend [:mossop] (he/him) from comment #5)

Ok I can reproduce this now.

• I found a different directory that had its .deps directory correctly populated.
• I added an include for a non-existent header to one of the header files in there.
• Ran the build and it failed and the .deps directory was now empty.
• Subsequent compiles succeeded with no changes.

Actually, I'm confused. I don't reproduce this at all. What happens for me is that for every .deps file that disappears, the corresponding .o disappears too. So obviously, a subsequent compile still needs to build the .o...

sccache uses a separate invocation for the preprocessor, that might be the crucial difference here.

Ok, I was able to reproduce with sccache.... and I'd say this should be fixed in sccache (FWIW, ccache does remove the object file on its own when the preprocessor fails), although we /can/ work around it in the build system.

Filed https://github.com/mozilla/sccache/issues/517

(In reply to Mike Hommey [:glandium] from comment #13)

although we /can/ work around it in the build system.

And the attached work around demonstrates that, but I'm not 100% sure we should actually land it.

I've tested with a build of sccache with the fix and it solves this problem for me.

Looks like this should be resolved, since the associated sccache patch has landed.