Nathan Froyd [:froydnj] Reporter
	Description • 11 years ago

Encountered this morning after a fresh pull from m-c, x86-64 Linux:

In file included from /opt/build/froydnj/build-mc/memory/jemalloc/Unified_c_memory_jemalloc1.c:6:0:
/home/froydnj/src/mozilla-central-official/memory/jemalloc/src/src/util.c:1:0: warning: "assert" redefined [enabled by default]In file included from /opt/build/froydnj/build-mc/memory/jemalloc/Unified_c_memory_jemalloc0.c:9:0:
/home/froydnj/src/mozilla-central-official/memory/jemalloc/src/src/ctl.c:116:13: error: conflicting types for ‘arena_purge’
In file included from /opt/build/froydnj/build-mc/memory/jemalloc/Unified_c_memory_jemalloc0.c:1:0:
/home/froydnj/src/mozilla-central-official/memory/jemalloc/src/src/arena.c:861:1: note: previous definition of ‘arena_purge’ was here

Removing --enable-dmd enables the build to complete successfully.

	Nathan Froyd [:froydnj] Reporter
	Comment 1 • 11 years ago

--enable-replace-malloc is really what's at fault here.  Updating bug title.

	Nathan Froyd [:froydnj] Reporter
	Comment 2 • 11 years ago

Here's a straightforward fix.

	Catalin Iacob
	Comment 3 • 11 years ago

On Windows, Nathan's patch is not enough; with --enable-dmd (I'm looking at bug 819839 to get DMD to work on Windows) I get:

 2:53.76 Unified_c_memory_build0.c
 2:53.76 c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(52) :
error C2375: 'jemalloc_stats' : redefinition; different linkage
 2:53.76         c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(52) : see declaration of 'jemalloc_stats'
 2:53.76 c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(53) :
error C2375: 'jemalloc_purge_freed_pages' : redefinition; different linkage
 2:53.76         c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(53) : see declaration of 'jemalloc_purge_freed_pages'
 2:53.76 c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(54) :
error C2375: 'jemalloc_free_dirty_pages' : redefinition; different linkage
 2:53.76         c:\users\catalin\moz\mozilla-central\memory\build\malloc_decls.h(54) : see declaration of 'jemalloc_free_dirty_pages'
 2:53.76 Native command 'mozbuild.action.cl main' returned value '2'
 2:53.76 c:\Users\Catalin\moz\mozilla-central\config\rules.mk:1027:0: command 'mozbuild.action.cl main  cl -FoUnified_c_memory_build0.obj -c  -DMOZ_MEMORY_IMPL -DMOZ_REPLACE_MALLOC -DNO_NSPR_10_SUPPORT -Ic:/Users/Catalin/moz/mozilla-central/memory/build -I.  -I../../dist/include  -Ic:/Users/Catalin/moz/mozilla-central/obj-i686-pc-mingw32/dist/include/nspr -Ic:/Users/Catalin/moz/mozilla-central/obj-i686-pc-mingw32/dist/include/nss       -MDd           -FI ../../dist/include/mozilla-config.h -DMOZILLA_CLIENT  -TC -nologo -W3 -Gy -Fdgenerated.pdb -wd4244 -wd4819 -we4553  -DDEBUG -D_DEBUG -DTRACING -Zi -O1 -Oy-  c:/Users/Catalin/moz/mozilla-central/obj-i686-pc-mingw32/memory/build/Unified_c_memory_build0.c' failed,
 return code 2

Note that jemalloc_stats, jemalloc_purge_freed_pages and jemalloc_free_dirty_pages are each declared twice with different linkage but the two different declarations come from the same header and line: malloc_decls.h(52) for jemalloc_stats. This is because the header gets included multiple times, what exactly it declares on each inclusion depends on the values of MALLOC_FUNCS and the definition of the MALLOC_DECL macro.

	Catalin Iacob
	Comment 4 • 11 years ago

Attached patch fixes the build for me by having mozmemory_wrap.c tell malloc_decls.h to not declare jemalloc functions.

The idea came from the similar code in replace_malloc.c which includes malloc_decls.h twice and tells it to define first malloc functions with MOZ_MEMORY_API and then jemalloc functions with MOZ_JEMALLOC_API

But I can't say I understand the code very well so please review carefully.

	(no longer active) Assignee
	Comment 5 • 11 years ago

Hmm, yeah <http://mxr.mozilla.org/mozilla-central/ident?i=MALLOC_DECL>.  We can't really build that directory in unified mode...

	(no longer active) Assignee
	Comment 6 • 11 years ago

Comment on attachment 8336127 [details] [diff] [review]
Windows part.patch

Review of attachment 8336127 [details] [diff] [review]:
-----------------------------------------------------------------

Let's not build this stuff in unified mode.  Patch coming up.

	(no longer active) Assignee
	Comment 7 • 11 years ago

Catalin can you please test this patch?

	Catalin Iacob
	Comment 8 • 11 years ago

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #7)
> Catalin can you please test this patch?

With your patch and Nathan's it does get further but it still doesn't finish the Windows --enable-dmd build. By the way, it's the same with my patch even though I said it fixes the build; it actually only allows it to move forward but it then runs into the same linker issues below.

Linking replace_jemalloc.dll fails with lots of errors:

20:35.25 jemalloc_config.obj
20:35.25 mozjemalloc_compat.obj
20:37.16 module.res
20:37.20 Creating Resource file: module.res
20:37.80 Microsoft (R) Windows (R) Resource Compiler Version 6.2.9200.16384
20:37.80
20:37.80 Copyright (C) Microsoft Corporation.  All rights reserved.
20:37.80
20:37.80
20:38.33 jemalloc_config.c
20:38.33 mozjemalloc_compat.c
20:38.33 replace_jemalloc.dll
20:39.72 Executing: link -NOLOGO -DLL -OUT:replace_jemalloc.dll -PDB:replace_jem
alloc.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86 @c:\Users\Catalin\moz\mozilla-central\
obj-i686-pc-mingw32\memory\replace\jemalloc\tmpnllxbq.list module.res -LARGEADDR
ESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH -DEBUG -DEBUGTYPE:CV kernel32.lib user3
2.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib secur32.lib netapi32.lib
20:39.72 c:\Users\Catalin\moz\mozilla-central\obj-i686-pc-mingw32\memory\replace
\jemalloc\tmpnllxbq.list:
20:39.72     jemalloc_config.obj
20:39.72     mozjemalloc_compat.obj
20:39.72     ..\..\jemalloc\arena.obj
20:39.72     ..\..\jemalloc\ctl.obj
20:39.72     ..\..\jemalloc\Unified_c_memory_jemalloc0.obj
20:39.72     ..\..\jemalloc\Unified_c_memory_jemalloc1.obj
20:39.72
20:39.72    Creating library replace_jemalloc.lib and object replace_jemalloc.ex
p
20:39.72
20:39.72 mozjemalloc_compat.obj : warning LNK4217: locally defined symbol _je_ma
llctl imported in function _replace_jemalloc_free_dirty_pages
20:39.72
20:39.72 mozjemalloc_compat.obj : warning LNK4217: locally defined symbol _je_ma
llctlnametomib imported in function _replace_jemalloc_stats
20:39.72
20:39.72 mozjemalloc_compat.obj : warning LNK4217: locally defined symbol _je_ma
llctlbymib imported in function _replace_jemalloc_stats
20:39.72
20:39.72 mozjemalloc_compat.obj : warning LNK4217: locally defined symbol _je_na
llocm imported in function _replace_malloc_good_size
20:39.72
20:39.72 arena.obj : warning LNK4217: locally defined symbol _moz_abort imported
 in function _arena_avail_adjac_pred
20:39.72
20:39.72 ctl.obj : warning LNK4049: locally defined symbol _moz_abort imported
20:39.72
20:39.72 Unified_c_memory_jemalloc0.obj : warning LNK4049: locally defined symbo
l _moz_abort imported
20:39.72
20:39.72 Unified_c_memory_jemalloc1.obj : warning LNK4049: locally defined symbo
l _moz_abort imported
20:39.72
20:39.72 arena.obj : error LNK2019: unresolved external symbol _malloc_mutex_loc
k referenced in function _arena_bin_nonfull_run_get
20:39.72
20:39.72 ctl.obj : error LNK2001: unresolved external symbol _malloc_mutex_lock
20:39.72
20:39.72 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _malloc_mutex_lock
20:39.72
20:39.72 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _malloc_mutex_lock
20:39.72
20:39.73 arena.obj : error LNK2019: unresolved external symbol _malloc_mutex_unl
ock referenced in function _arena_bin_nonfull_run_get
20:39.73
20:39.73 ctl.obj : error LNK2001: unresolved external symbol _malloc_mutex_unloc
k
20:39.73
20:39.73 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _malloc_mutex_unlock
20:39.73
20:39.73 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _malloc_mutex_unlock
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _sa2u referenced
in function _arena_ralloc
20:39.73
20:39.73 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _sa2u
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _choose_arena ref
erenced in function _arena_malloc
20:39.73
20:39.73 ctl.obj : error LNK2001: unresolved external symbol _choose_arena
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _bitmap_full refe
renced in function _arena_run_reg_alloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _bitmap_get refer
enced in function _arena_run_reg_dalloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _bitmap_sfu refer
enced in function _arena_run_reg_alloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _bitmap_unset ref
erenced in function _arena_run_reg_dalloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _tcache_get refer
enced in function _arena_dalloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _tcache_alloc_sma
ll referenced in function _arena_malloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _tcache_alloc_lar
ge referenced in function _arena_malloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _tcache_dalloc_sm
all referenced in function _arena_dalloc
20:39.73
20:39.73 arena.obj : error LNK2019: unresolved external symbol _tcache_dalloc_la
rge referenced in function _arena_dalloc
20:39.74
20:39.74 arena.obj : error LNK2019: unresolved external symbol _ipallocx referen
ced in function _arena_ralloc
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _ipallocx
20:39.74
20:39.74 arena.obj : error LNK2019: unresolved external symbol _iqallocx referen
ced in function _arena_ralloc
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _iqallocx
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _arenas_tsd_set ref
erenced in function _thread_arena_ctl
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _arenas_tsd_set
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _narenas_total_get
referenced in function _ctl_init
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _narenas_total_get
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _tcache_tsd_get ref
erenced in function _thread_arena_ctl
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _tcache_flush refer
enced in function _thread_tcache_flush_ctl
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _tcache_enabled_get
 referenced in function _thread_tcache_enabled_ctl
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _tcache_enabled_set
 referenced in function _thread_tcache_enabled_ctl
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _imalloc referenced
 in function _ctl_grow
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _imalloc
20:39.74
20:39.74 ctl.obj : error LNK2019: unresolved external symbol _idalloc referenced
 in function _ctl_grow
20:39.74
20:39.74 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _idalloc
20:39.74
20:39.74 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _idalloc
20:39.75
20:39.75 ctl.obj : error LNK2019: unresolved external symbol _iralloc referenced
 in function _ctl_grow
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _iralloc
20:39.75
20:39.75 ctl.obj : error LNK2019: unresolved external symbol _thread_allocated_t
sd_get referenced in function _thread_allocated_ctl
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2001: unresolved external sym
bol _thread_allocated_tsd_get
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _pow2_ceil referenced in function _rtree_new
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _set_errno referenced in function _malloc_conf_init
20:39.75
20:39.75 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _set_errno
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _get_errno referenced in function _malloc_conf_init
20:39.75
20:39.75 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _get_errno
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _arenas_tsd_boot referenced in function _malloc_init_hard
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _s2u referenced in function _iallocm
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _rtree_set referenced in function _chunk_alloc
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _hash referenced in function _ckh_string_hash
20:39.75
20:39.75 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _imallocx referenced in function _iallocm
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _icallocx referenced in function _iallocm
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2001: unresolved external sym
bol _icallocx
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _icalloc referenced in function _replace_calloc
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _ipalloc referenced in function _ckh_grow
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _isalloc referenced in function _imemalign
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _ivsalloc referenced in function _je_sallocm
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _iqalloc referenced in function _replace_free
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _irallocx referenced in function _je_rallocm
20:39.76
20:39.76 Unified_c_memory_jemalloc0.obj : error LNK2019: unresolved external sym
bol _thread_allocated_tsd_boot referenced in function _malloc_init_hard
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2019: unresolved external sym
bol _tcache_tsd_boot referenced in function _tcache_boot1
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2019: unresolved external sym
bol _tcache_tsd_set referenced in function _tcache_create
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2019: unresolved external sym
bol _tcache_enabled_tsd_boot referenced in function _tcache_boot1
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2019: unresolved external sym
bol _tcache_alloc_easy referenced in function _tcache_alloc_small_hard
20:39.76
20:39.76 Unified_c_memory_jemalloc1.obj : error LNK2019: unresolved external sym
bol _idallocx referenced in function _tcache_destroy
20:39.76
20:39.76 replace_jemalloc.dll : fatal error LNK1120: 46 unresolved externals
20:39.76
20:39.76 c:\Users\Catalin\moz\mozilla-central\config\rules.mk:949:0: command 'c:
/Users/Catalin/moz/mozilla-central/obj-i686-pc-mingw32/_virtualenv/Scripts/pytho
n.exe c:/Users/Catalin/moz/mozilla-central/config/expandlibs_exec.py --depend .d
eps/replace_jemalloc.dll.pp --target replace_jemalloc.dll --uselist -- link -NOL
OGO -DLL -OUT:replace_jemalloc.dll -PDB:replace_jemalloc.pdb -SUBSYSTEM:WINDOWS
-MACHINE:X86  jemalloc_config.obj mozjemalloc_compat.obj  ./module.res -LARGEADD
RESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV    ../../../memo
ry/jemalloc/jemalloc.lib   kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.l
ib advapi32.lib secur32.lib netapi32.lib    ' failed, return code 1120
20:39.76 <libs>: Found error

If I forget about the patches in this bug and instead backout the whole of https://hg.mozilla.org/mozilla-central/rev/c3d41ab I do get replace_jemalloc.dll to link but I didn't have time to do a full build (I would expect it to work though).

	(no longer active) Assignee
	Comment 9 • 11 years ago

Any chance you could please try reverting individual hunks in that patch to see which parts should get reverted in order to fix this?

	(no longer active) Assignee
	Comment 10 • 11 years ago

Actually that is kind of an unfair request!  I'll back out that patch for now, but if you could help figure out which parts of this we can re-enable over in bug 940218 that would be highly appreciated!

	(no longer active) Assignee
	Comment 11 • 11 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/f7e67f388134

	Ryan VanderMeulen [:RyanVM]
	Comment 12 • 11 years ago

https://hg.mozilla.org/mozilla-central/rev/c93167ec9c1f

	Catalin Iacob
	Comment 13 • 11 years ago

Comment on attachment 8336129 [details] [diff] [review]
Don't build memory/build in unified mode

Review of attachment 8336129 [details] [diff] [review]:
-----------------------------------------------------------------

Out of date because of the backout in https://hg.mozilla.org/mozilla-central/rev/c93167ec9c1f

I'll look at bug 940218 to find a fix that works for Windows --enable-dmd as well.

	(no longer active) Assignee
	Comment 14 • 11 years ago

That would be great, thank you!