emscripten cross-compile wasm-ld: error: duplicate symbol _Py_LibHacl_Hacl_Hash_*

[Lukasdoe]

Bug report

Bug description:

For the emscripten build, during building of the python executable, all the LIBHACL_*_LIB_SHARED are used as linker flags, however the LIBHACL_HMAC_LIB_SHARED flag contains objects that are already contained in other LIBHACL_*_LIB_SHARED flags. This happens, because LIBHACL_MD5_OBJS, LIBHACL_SHA1_OBJS, LIBHACL_SHA2_OBJS, LIBHACL_SHA3_OBJS and LIBHACL_BLAKE2_OBJS are all included in LIBHACL_HMAC_LIB_SHARED through LIBHACL_HMAC_OBJS via the Makefile.pre.in file.

The configure script then sets LIBHACL_HMAC_LDFLAGS=LIBHACL_HMAC_LIB_${LIBHACL_LDEPS_LIBTYPE} which for a shared emscripten build leads to duplicate dependencies in the final build command which leads to duplicate symbols.

Since I don't know the purpose of the object file references in the LIBHACL_HMAC_OBJS, I do not feel comfortable just removing them, even though that change fixes the emscripten build.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-133042: remove object file dependencies from LIBHACL_HMAC_* #133043
• gh-133042: disable HACL* HMAC on Emscripten #133064

[picnixz]

Mmh, EMScripten with shared dependencies is experimental so I should actually force static linking in this case. You will likely encounter the same issues with the other HACL* primitives as they would also have duplicated symbols. As you can see on your PR, the change is breaking for the regular builds.

I'll see what I can do (probably a similar issue as we had with WASI).

[Lukasdoe]

Thx for the quick reply! Yes, just removing the objects is just breaking the regular builds. However, this is the only part that fails my shared build.

One part of the configure script also caught my eye:

cpython/configure.ac

Lines 7960 to 7966 in 614d792

	if test "$ac_sys_system" = "WASI"; then
	LIBHACL_LDEPS_LIBTYPE=STATIC
	AC_MSG_RESULT([static])
	else
	LIBHACL_LDEPS_LIBTYPE=SHARED
	AC_MSG_RESULT([shared])
	fi

[picnixz]

WASI requires static linking but I'm unsure whether emscripten can have the corresponding built-in modules. If needed, I can just.. disable the HACL* primitives. I can't test locally because I have a "python.mjs" not found and I don't know how to fix it.

[picnixz]

Question: would forcing LIBHACL_LDEPS_LIBTYPE=STATIC be sufficient on your side to fix the EMscripten build?

[picnixz]

OK I think I know the issue. At the final makefile step, we have something like that:

-lm -lm -lm -lm -lm Modules/_decimal/libmpdec/libmpdec.a -sUSE_ZLIB -sUSE_BZIP2 -sUSE_ZLIB Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.o   Modules/_hacl/Hacl_HMAC.o Modules/_hacl/Hacl_Streaming_HMAC.o Modules/_hacl/Hacl_Hash_MD5.o Modules/_hacl/Hacl_Hash_SHA1.o Modules/_hacl/Hacl_Hash_SHA2.o Modules/_hacl/Hacl_Hash_SHA3.o Modules/_hacl/Hacl_Hash_Blake2s.o Modules/_hacl/Hacl_Hash_Blake2b.o Modules/_hacl/Lib_Memzero0.o

The first occurrences are of Hacl* files are coming from each individual modules, but the last ones are indeed coming from HMAC. It's not an issue for regular compilers but for emscripten it appears that it's an issue (though I don't know why??) I'll make it so that the configure script can figure out the duplicates.

[picnixz]

Ok it's much more complex than I thought. Question, can you check if the following would solve your issue:

-if test "$ac_sys_system" = "WASI"; then 
+if test "$ac_sys_system" = "WASI" -o "$ac_sys_system" = "Emscripten"; then 

I don't have time now to fix the build entirely (sorry) but I'll have a look again tomorrow. If this still fails, we should make it so that only unique .o are retained when linking the Python executable.

[picnixz]

I'm marking it as a release blocker because I'm leaving tomorrow and won't be back before the next beta release so we need to find a solution in the next 24 hours (otherwise it's annoying for those working with Emscripten). It's not a true release blocker though but it's to keep in mind that this must be addressed before releasing the next sources.

[freakboy3742]

Tagging @hoodmane for his input.

[picnixz]

Ok, static linking doesn't work either. It's really because the .a or .o are here twice. Some possibilities:

• Disable HACL* HMAC as it's the one that induces inter-dependencies.
• Recompute the Makefile rules for Emscripten. I tried and failed. I tried something like "HMAC depends on MD5.a and I don't want to have MD5.a included because I built _md5 and MD5.a included because I built _hmac" so I assumed that I can just not build either of the two but I don't know what to write.

For the next beta, let's just disable HACL* HMAC. It's a new feature already so it's fine if we don't support it yet for Emscripten.