electron concurrency issue and zombie processes

[kiskele]

Hi

I am trying to troubleshoot an issue: I am running ace-electron in docker and when multiple ace processes are running parallel, it seems they are blocking each other and ACE is failing with various error messages, like:

error:   Ace processing error: Error: Failed to check Content Document '949.html': did-fail-load: -3 -  - acehttps://0.0.0.0/Ops/949.html?AXE_RUNNER=948 - true - 7 - 1

I was able to reproduce the issue even without docker. In the process tree I can see this:

 |   |   \-+= 99767 kiskele node /Users/kiskele/temp/ace/node_modules/.bin/electron ./packages/ace-axe-runner-electron/lib/cli.js --verbose --out out2 /Users/kiskele/Downloads/test.epub
 |   |       |--- 99771 kiskele /Users/kiskele/temp/ace/node_modules/electron/dist/Electron.app/Contents/Frameworks/Electron Helper (GPU).app/Contents/MacOS/Electron Helper (GPU) --type=gpu-process --user-data-dir=/Users/kiskele/Library/Application Support/Electron --gpu-preferences=UAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAEAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAA --shared-files --field-trial-handle=1718379636,r,6378836638113066318,16088629395246184720,262144 --enable-features=ScreenCaptureKitPickerScreen,ScreenCaptureKitStreamPickerSonoma --disable-features=MacWebContentsOcclusion,SpareRendererForSitePerProcess,TimeoutHangingVideoCaptureStarts --variations-seed-version --seatbelt-client=35
 |   |       \--- 99772 kiskele /Users/kiskele/temp/ace/node_modules/electron/dist/Electron.app/Contents/Frameworks/Electron Helper.app/Contents/MacOS/Electron Helper --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-GB --service-sandbox-type=network --user-data-dir=/Users/kiskele/Library/Application Support/Electron --standard-schemes=acehttps --secure-schemes=acehttps --cors-schemes=acehttps --fetch-schemes=acehttps --streaming-schemes=acehttps --shared-files --field-trial-handle=1718379636,r,6378836638113066318,16088629395246184720,262144 --enable-features=ScreenCaptureKitPickerScreen,ScreenCaptureKitStreamPickerSonoma --disable-features=MacWebContentsOcclusion,SpareRendererForSitePerProcess,TimeoutHangingVideoCaptureStarts --variations-seed-version --seatbelt-client=35

and this:

 |       \-+= 99677 kiskele node /Users/kiskele/temp/ace/node_modules/.bin/electron ./packages/ace-axe-runner-electron/lib/cli.js --verbose --out out /Users/kiskele/Downloads/test.epub
 |           |--- 99681 kiskele /Users/kiskele/temp/ace/node_modules/electron/dist/Electron.app/Contents/Frameworks/Electron Helper (GPU).app/Contents/MacOS/Electron Helper (GPU) --type=gpu-process --user-data-dir=/Users/kiskele/Library/Application Support/Electron --gpu-preferences=UAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAEAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAA --shared-files --field-trial-handle=1718379636,r,9488308668255699911,12629781689342009476,262144 --enable-features=ScreenCaptureKitPickerScreen,ScreenCaptureKitStreamPickerSonoma --disable-features=MacWebContentsOcclusion,SpareRendererForSitePerProcess,TimeoutHangingVideoCaptureStarts --variations-seed-version --seatbelt-client=37
 |           \--- 99682 kiskele /Users/kiskele/temp/ace/node_modules/electron/dist/Electron.app/Contents/Frameworks/Electron Helper.app/Contents/MacOS/Electron Helper --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-GB --service-sandbox-type=network --user-data-dir=/Users/kiskele/Library/Application Support/Electron --standard-schemes=acehttps --secure-schemes=acehttps --cors-schemes=acehttps --fetch-schemes=acehttps --streaming-schemes=acehttps --shared-files --field-trial-handle=1718379636,r,9488308668255699911,12629781689342009476,262144 --enable-features=ScreenCaptureKitPickerScreen,ScreenCaptureKitStreamPickerSonoma --disable-features=MacWebContentsOcclusion,SpareRendererForSitePerProcess,TimeoutHangingVideoCaptureStarts --variations-seed-version --seatbelt-client=37

What I can see as a possible issue, that both electron instances are using the same user-data-dir. Is there any way to enter a diferent user-data-dir from the cli?

[kiskele]

During execution the following processes are open:

    543 ?        Sl     0:00 node /usr/bin/ace-electron --silent --user-data-dir=/tmp/tmp.7a9fimWK7n /data/output/test.epub
    550 ?        Sl     4:08 @daisy/ace-cli-shared
    553 ?        S      0:00 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=zygote --no-zygote-sandbox
    554 ?        S      0:00 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=zygote
    557 ?        S      0:00 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=zygote
    602 ?        Sl     0:00 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=utility --utility-sub-type=network.mojom.NetworkServic
    634 ?        Sl     0:00 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=gpu-process --enable-crash-reporter=0a6e3701-6f53-4e27
   1056 ?        Rl     0:08 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=renderer --enable-crash-reporter=0a6e3701-6f53-4e27-aa
   1057 ?        Rl     0:07 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=renderer --enable-crash-reporter=0a6e3701-6f53-4e27-aa
   1058 ?        Rl     0:08 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=renderer --enable-crash-reporter=0a6e3701-6f53-4e27-aa
   1059 ?        Rl     0:08 /usr/lib/node_modules/@daisy/ace/node_modules/electron/dist/electron --type=renderer --enable-crash-reporter=0a6e3701-6f53-4e27-aa

Even with a successful run (no parallel processes), two zombie processes are left open after ace was exiting:

    553 ?        Z      0:00 [electron] <defunct>
    554 ?        Z      0:00 [electron] <defunct>

[kiskele]

It seems that user-data-dir can be set only on Mac, but even there wasn't helping to solve the concurrency issue

[kiskele]

I assume that did-fail-load: -3 occurs when first ace process is exiting and closing all the electron windows/app

[danielweck]

hello, your logs indicate MacOS but you're running on Docker which should be Linux?

[danielweck]

oh sorry, you tested in MacOS as well.

[kiskele]

I haven't seen zombie processes on mac, only in docker

[danielweck]

are you using some Electron headless runtime tricks from https://github.com/daisy/ace/blob/master/Dockerfile

[kiskele]

I was trying with Xvfb server running in the background, but also tried with ace-electron /data/output/test.epub --no-sandbox --headless inside Docker, or the same but also adding --disable-gpu

[kiskele]

Now it seems that I have sold some of the issues:

• zombie processes are gone when I changed entrypoint.sh line from: exec "$@" to "$@"
• electron processes were sending "trap int3" signals to kernel was solved by downgrading to Ubuntu 20.04, because Electron binaries are pre-built on 20.04. Would be great to build from source on Ubuntu 24.04 and publish the binaries
  kernel: traps: electron[395254] trap int3 ip:58596746d54e sp:7ffff2fc15a0 error:0 in electron[5859633ee000+897e000]
• concurrency issues may have solved by running command as ace-electron test.epub --no-sandbox --headless --disable-gpu --user-data-dir=$(mktemp -d)

I will do more tests, but right now result looks significantly better

[danielweck]

Great findings! Thank you for sharing.