Compile without Atomics / POSIX threads / pthreads

[Jaifroid]

The main reason why reading a ZIM archive with libzim fails with the current build in older browsers is that somewhere in the build chain one or more of the binaries has been compiled with pthreads (Atomics). As we will never write or modify data (the main reason for atomic operations), it should be possible to do without them at least for a specific JavaScript (ASM) build targeting older browsers (#17).

@mgautierfr Is this something you're aware of in the libzim build chain? Any clue as to where pthreads are being required?

In JavaScript, the pthreads translate into JavaScript Atomic operations, which are not available in a wide range of potential targets. E.g., although Atomics are supported in latest Chromium and Firefox (desktop), they fail on: Safari (both on Mac OS and on iOS, as well as any other browser on iOS), on EdgeHTML (used still in UWP app webview), on ESR versions of Firefox, and on Internet Explorer (despite the fact that we can compile to ASM, which would be supported by IE11 were it not for the Atomics requirement). Any Android browser other than those based directly on Chromium also fail.

[mgautierfr]

libzim is using pthread because it is multithreaded (at creation) and threadsafe (at reading).

We have internal cache that we modify at reading (this is a cache) and we want to protect against concurrent cache modification.

[Jaifroid]

OK, thanks for the info @mgautierfr. The WASM libzim worker is single-threaded (it operates on its own thread, separate from the app, but it's just one thread), so there would be no danger in compiling a non-pthreaded version for JavaScript, and obviously Emscripten will only compile the APIs we actually use and declare (i.e. read APIs). Do you think it might be possible, or is the pthreading too deeply embedded in the source code?

[kelson42]

@Jaifroid For me its important to better understand the user impact we have currently. We don't provide KiwixJS for Safari... so why this is a problem? Which of our software are suffering of this limitation and how many users are impacted?

[Jaifroid]

@kelson42 Sorry to have dropped this thread.

Remember that the PWA is designed to be universal. It runs on anything that has a browser capable of using Service Workers, and also (like any Web app), degrades gracefully. I'm quite proud that it degrades all the way down to IE11. Fun fact: it can even read static Zimit archives in IE11...

However, I can't use the WASM version of libzim as the backend on any platform other than latest desktop Chrome/Edge and Firefox, due to this issue. Safari was just one example. The PWA runs pretty well on Safari on iOS, but can't use the features provided by the WASM.

Now, you can argue that it's OK, we have an alternative, which is to fall back to reading the ZIM with our legacy backend, which is what indeed happens. However, I wanted to see if there might be a "quick fix" somewhere in the compilation of libzim-wasm/asm which would allow me to extend the compatibility of the WASM. Can I disable pthreading during compilation?

It looks like that might be more complex than I thought, if pthreading is baked in to libzim, and not something that can be disabled through compilation options, or by excluding modules that aren't needed (e.g. ZIM writing functions).

[kelson42]

@Jaifroid Thank you for the explanation, to be really honest on my POV: is not a priority, but indeed would be nice if there is a quick fix
@mgautierfr This seems to rejoin our discussion about multithreading for reader/writer, etc... we had in Paris. But I don't remember the context/outclomes nor ticket. Could you update us, maybe link this ticket to the base discussion in openzim/libzim?

[mgautierfr]

@mgautierfr This seems to rejoin our discussion about multithreading for reader/writer, etc... we had in Paris. But I don't remember the context/outclomes nor ticket. Could you update us, maybe link this ticket to the base discussion in openzim/libzim?

This was the ticket :)

The idea was that we could:

• Separate creator and reader part in two libraries (but keeping the code in the same repository).
• Add a compilation option to remove the thread safe protection from the reader library for such use case.

[Jaifroid]

I believe this would widen compatibility, but I also understand if it's not high priority. The fact is that we already have very wide compatibility with our existing back end, and we are currently only using libzim WASM for full-text search.

I will add using libzim WASM for article retrieval as an experimental option only (opt-in), mostly for testing purposes, benchmarking, etc. But I don't currently see it being viable for mainstream use until 1) I can widen the compatibility; or 2) enough time has passed that browsers and platforms that don't currently support Atomics have either caught up or have become irrelevant... (But that could be a while yet...)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[Jaifroid]

Interestingly, I've discovered that iOS, which reports that it doesn't support Atomic operations in JS, can nevertheless run the WASM version in iOS 15 and 16 (but not in iOS <= 14). It seems that supporting Atomics in JS is not always aligned with support for them in WASM. (iOS always fails on the ASM version, because that is pure JS.)