Gramine + Kafka

[rodrigo-pedro]

Has anyone managed to get Kafka running on Gramine? Mine just exits immediately with no error messages.

[monavij]

Try with all log messages enabled and see what you get. Is this the case both with gramine-direct and gramine-sgx? Share your manifest file.

[rodrigo-pedro]

Thank you for the response. Here are the logs:

logs.txt

[dimakuv]

@rodrigo-pedro Looks like you're trying to put a very complex Bash script inside Gramine. I see this line in the log:

[P52:T74:bash] trace: ---- execve("/usr/sbin/useradd", [useradd,-N,kafka,], [SHELL=/bin/bash,COLORTERM=truecolor,]) ...

The invocation of /usr/bin/useradd (a utility to add users) inside of the Gramine environment makes no sense. It is an administrative task that should be done outside of Gramine. Gramine is tailored to run non-administrative single applications.

I suggest you to extract only the actual workload-running part from your entrypoint.sh bash script, and wrap this part inside Gramine.

[rodrigo-pedro]

I understand. This is not my code, it's the official kafka launch script. From what I can tell, the workload is actually this line:

/opt/bitnami/java/bin/java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/kafka-files/bin/../logs/zookeeper-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka-files/bin/../logs -Dlog4j.configuration=file:/kafka-files/bin/../config/log4j.properties -cp /kafka-files/bin/../libs/activation-1.1.1.jar:/kafka-files/bin/../libs/aopalliance-repackaged-2.6.1.jar:/kafka-files/bin/../libs/argparse4j-0.7.0.jar:/kafka-files/bin/../libs/audience-annotations-0.5.0.jar:/kafka-files/bin/../libs/commons-cli-1.4.jar:/kafka-files/bin/../libs/commons-lang3-3.8.1.jar:/kafka-files/bin/../libs/connect-api-3.4.0.jar:/kafka-files/bin/../libs/connect-basic-auth-extension-3.4.0.jar:/kafka-files/bin/../libs/connect-json-3.4.0.jar:/kafka-files/bin/../libs/connect-mirror-3.4.0.jar:/kafka-files/bin/../libs/connect-mirror-client-3.4.0.jar:/kafka-files/bin/../libs/connect-runtime-3.4.0.jar:/kafka-files/bin/../libs/connect-transforms-3.4.0.jar:/kafka-files/bin/../libs/hk2-api-2.6.1.jar:/kafka-files/bin/../libs/hk2-locator-2.6.1.jar:/kafka-files/bin/../libs/hk2-utils-2.6.1.jar:/kafka-files/bin/../libs/jackson-annotations-2.13.4.jar:/kafka-files/bin/../libs/jackson-core-2.13.4.jar:/kafka-files/bin/../libs/jackson-databind-2.13.4.2.jar:/kafka-files/bin/../libs/jackson-dataformat-csv-2.13.4.jar:/kafka-files/bin/../libs/jackson-datatype-jdk8-2.13.4.jar:/kafka-files/bin/../libs/jackson-jaxrs-base-2.13.4.jar:/kafka-files/bin/../libs/jackson-jaxrs-json-provider-2.13.4.jar:/kafka-files/bin/../libs/jackson-module-jaxb-annotations-2.13.4.jar:/kafka-files/bin/../libs/jackson-module-scala_2.13-2.13.4.jar:/kafka-files/bin/../libs/jakarta.activation-api-1.2.2.jar:/kafka-files/bin/../libs/jakarta.annotation-api-1.3.5.jar:/kafka-files/bin/../libs/jakarta.inject-2.6.1.jar:/kafka-files/bin/../libs/jakarta.validation-api-2.0.2.jar:/kafka-files/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/kafka-files/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/kafka-files/bin/../libs/javassist-3.27.0-GA.jar:/kafka-files/bin/../libs/javax.annotation-api-1.3.2.jar:/kafka-files/bin/../libs/javax.servlet-api-3.1.0.jar:/kafka-files/bin/../libs/javax.ws.rs-api-2.1.1.jar:/kafka-files/bin/../libs/jaxb-api-2.3.0.jar:/kafka-files/bin/../libs/jersey-client-2.34.jar:/kafka-files/bin/../libs/jersey-common-2.34.jar:/kafka-files/bin/../libs/jersey-container-servlet-2.34.jar:/kafka-files/bin/../libs/jersey-container-servlet-core-2.34.jar:/kafka-files/bin/../libs/jersey-hk2-2.34.jar:/kafka-files/bin/../libs/jersey-server-2.34.jar:/kafka-files/bin/../libs/jetty-client-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-continuation-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-http-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-io-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-security-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-server-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-util-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jetty-util-ajax-9.4.48.v20220622.jar:/kafka-files/bin/../libs/jline-3.21.0.jar:/kafka-files/bin/../libs/jopt-simple-5.0.4.jar:/kafka-files/bin/../libs/jose4j-0.7.9.jar:/kafka-files/bin/../libs/kafka-clients-3.4.0.jar:/kafka-files/bin/../libs/kafka-group-coordinator-3.4.0.jar:/kafka-files/bin/../libs/kafka-log4j-appender-3.4.0.jar:/kafka-files/bin/../libs/kafka-metadata-3.4.0.jar:/kafka-files/bin/../libs/kafka-raft-3.4.0.jar:/kafka-files/bin/../libs/kafka-server-common-3.4.0.jar:/kafka-files/bin/../libs/kafka-shell-3.4.0.jar:/kafka-files/bin/../libs/kafka-storage-3.4.0.jar:/kafka-files/bin/../libs/kafka-storage-api-3.4.0.jar:/kafka-files/bin/../libs/kafka-streams-3.4.0.jar:/kafka-files/bin/../libs/kafka-streams-examples-3.4.0.jar:/kafka-files/bin/../libs/kafka-streams-scala_2.13-3.4.0.jar:/kafka-files/bin/../libs/kafka-streams-test-utils-3.4.0.jar:/kafka-files/bin/../libs/kafka-tools-3.4.0.jar:/kafka-files/bin/../libs/kafka_2.13-3.4.0.jar:/kafka-files/bin/../libs/lz4-java-1.8.0.jar:/kafka-files/bin/../libs/maven-artifact-3.8.4.jar:/kafka-files/bin/../libs/metrics-core-2.2.0.jar:/kafka-files/bin/../libs/metrics-core-4.1.12.1.jar:/kafka-files/bin/../libs/netty-buffer-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-codec-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-common-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-handler-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-resolver-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-transport-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-transport-classes-epoll-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-transport-native-epoll-4.1.78.Final.jar:/kafka-files/bin/../libs/netty-transport-native-unix-common-4.1.78.Final.jar:/kafka-files/bin/../libs/osgi-resource-locator-1.0.3.jar:/kafka-files/bin/../libs/paranamer-2.8.jar:/kafka-files/bin/../libs/plexus-utils-3.3.0.jar:/kafka-files/bin/../libs/reflections-0.9.12.jar:/kafka-files/bin/../libs/reload4j-1.2.19.jar:/kafka-files/bin/../libs/rocksdbjni-7.1.2.jar:/kafka-files/bin/../libs/scala-collection-compat_2.13-2.6.0.jar:/kafka-files/bin/../libs/scala-java8-compat_2.13-1.0.2.jar:/kafka-files/bin/../libs/scala-library-2.13.10.jar:/kafka-files/bin/../libs/scala-logging_2.13-3.9.4.jar:/kafka-files/bin/../libs/scala-reflect-2.13.10.jar:/kafka-files/bin/../libs/slf4j-api-1.7.36.jar:/kafka-files/bin/../libs/slf4j-reload4j-1.7.36.jar:/kafka-files/bin/../libs/snappy-java-1.1.8.4.jar:/kafka-files/bin/../libs/swagger-annotations-2.2.0.jar:/kafka-files/bin/../libs/trogdor-3.4.0.jar:/kafka-files/bin/../libs/zookeeper-3.6.3.jar:/kafka-files/bin/../libs/zookeeper-jute-3.6.3.jar:/kafka-files/bin/../libs/zstd-jni-1.5.2-1.jar org.apache.zookeeper.server.quorum.QuorumPeerMain /kafka-files/config/zookeeper.properties

Unfortunately, I'm getting "[P1:T1:] error: libos_init() failed in init_process_cmdline: Cannot allocate memory (ENOMEM)" now:

gramine-manifest \
        -Dlog_level=error \
        -Darch_libdir=/lib/x86_64-linux-gnu \
        kafka.manifest.template > kafka.manifest
(libos_init.c:382:libos_init) debug: Gramine was built from commit: aefd5d0875c4fdf76edd67f5043952bdd32b0f7a
(libos_init.c:384:libos_init) debug: Host: Linux
(libos_context.c:70:libos_xstate_init) debug: LibOS xsave_enabled 1, xsave_size 0x440(1088), xsave_features 0x1f
(libos_vma.c:615:init_vma) debug: Initial VMA region 0xffffffffff600000-0xffffffffff601000 ([vsyscall]) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffdd000-0x7ffffffff000 ([stack]) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffdc000-0x7ffffffdd000 (stack guard) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffcb000-0x7ffffffdc000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffca000-0x7ffffffcb000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb9000-0x7ffffffca000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb8000-0x7ffffffb9000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb7000-0x7ffffffb8000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb6000-0x7ffffffb7000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb5000-0x7ffffffb6000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb4000-0x7ffffffb5000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb2000-0x7ffffffb4000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffffffb1000-0x7ffffffb2000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7fffffefe000-0x7ffffffb1000 (PAL internal me) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffff7ffd000-0x7ffff7fff000 ([heap]) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffff7ff8000-0x7ffff7ffd000 (/usr/lib/x86_64) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffff7fd0000-0x7ffff7ff8000 (/usr/lib/x86_64) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffff7fce000-0x7ffff7fd0000 ([vdso]) bookkeeped
(libos_vma.c:615:init_vma) debug: Initial VMA region 0x7ffff7fca000-0x7ffff7fce000 ([vvar]) bookkeeped
(libos_vma.c:646:init_vma) debug: ASLR top address adjusted to 0x6820880a6000
(libos_init.c:410:libos_init) debug: LibOS loaded at 0x7fffffefe000, ready to initialize
(libos_init.c:568:create_pipe) debug: Creating pipe: pipe.srv:0b47326dee2645900e63902b2dfb4cd0a6ba81fd45f8d5c85c232685a6f49cb2
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:." (chroot) under /
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "proc" (pseudo) under /proc
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "dev" (pseudo) under /dev
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "dev:tty" (chroot) under /dev/tty
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "sys" (pseudo) under /sys
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/usr/lib/x86_64-linux-gnu/gramine/runtime/glibc" (chroot) under /lib
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/lib/x86_64-linux-gnu" (chroot) under /lib/x86_64-linux-gnu
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/usr" (chroot) under /usr
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/etc" (chroot) under /etc
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/bin" (chroot) under /bin
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/opt/bitnami" (chroot) under /opt/bitnami
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "(null)" (tmpfs) under /tmp
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/bitnami" (chroot) under /bitnami
(libos_fs.c:787:mount_fs) [P1:T1:] debug: mounting "file:/kafka-files" (chroot) under /kafka-files
(libos_init.c:437:libos_init) [P1:T1:] error: libos_init() failed in init_process_cmdline: Cannot allocate memory (ENOMEM)

I might just have to give up on this.

[kailun-qin]

@rodrigo-pedro The cmdline passed in (~6KB) in your case is too long for (current) Gramine (which now has a static buffer of one page (4KB) in size) so its initialization failed w/ Cannot allocate memory (ENOMEM). Pls kindly see the notes here:

gramine/libos/include/libos_process.h

Lines 63 to 66 in dd1d3dc

	/* Complete command line for the process, as reported by /proc/[pid]/cmdline; currently filled
	* once during initialization, using static buffer and restricted to CMDLINE_SIZE. This is
	* enough for current workloads but see https://github.com/gramineproject/gramine/issues/79. */
	char cmdline[CMDLINE_SIZE];

and the corresponding issue: #79.

So you can try reducing the cmdline size (if possible) as a workaround and we should revisit the requirement and fix it.

[dimakuv]

@rodrigo-pedro Thanks for the interesting corner case. I submitted a PR to increase the cmdline limit: #1319

If you have still have the motivation, I enourage you to test this PR.