Running a basic Haskell binary in Gramine hangs

[Abhiroop]

I am attempting to run this very simple program Haskell program that says "Hello World" using Gramine

module Main where

import System.IO

main :: IO ()
main = do
  putStrLn "Hello World"
  hFlush stdout

(Added the flushing of stdout after reading this - #788)

However, when I try to run this application using gramine-sgx the application hangs. With the DEBUG build, I get the following:

...
[P1:T1:helloworld] debug: Creating pipe: pipe.srv:b862427738f780b5cbeac1d933e629fcc42a458bccb19fccc48a83ae66c27936
[P1:T1:helloworld] debug: Allocated stack at 0x3956000 (size = 0x40000)
[P1:T1:helloworld] debug: loading "file:helloworld"
[P1:T1:helloworld] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2
[P1:T1:helloworld] debug: loading "file:/usr/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:helloworld] debug: Creating pipe: pipe.srv:1
[P1:T1:helloworld] debug: Creating pipe: pipe.srv:1085b8c1fa29b1b9732659d9a9cbe2cd4b516a5e9f3d119a7feac00a54d43b80
[P1:libos] debug: IPC worker started
[P1:T1:helloworld] debug: LibOS initialized
[P1:T1:helloworld] debug: glibc register library /lib/libm.so.6 loaded at 0x0383c000
[P1:T1:helloworld] debug: glibc register library /lib/x86_64-linux-gnu/libgmp.so.10 loaded at 0x037b8000
[P1:T1:helloworld] debug: glibc register library /lib/x86_64-linux-gnu/libatomic.so.1 loaded at 0x037ae000
[P1:T1:helloworld] debug: glibc register library /lib/librt.so.1 loaded at 0x037a9000
[P1:T1:helloworld] debug: glibc register library /lib/libdl.so.2 loaded at 0x037a4000
[P1:T1:helloworld] debug: glibc register library /lib/x86_64-linux-gnu/libffi.so.7 loaded at 0x03798000
[P1:T1:helloworld] debug: glibc register library /lib/libpthread.so.0 loaded at 0x03791000
[P1:T1:helloworld] debug: glibc register library /lib/libc.so.6 loaded at 0x0359d000
[P1:T1:helloworld] warning: Unsupported system call rseq
[P1:T1:helloworld] warning: Per-process and per-thread CPU-time clocks are not supported in clock_gettime(); they are replaced with system-wide real-time clock.

Is the issue happening due to the usage of the clock_gettime() function? (The GHC runtime uses this).

My manifest file looks like this - https://gist.github.com/Abhiroop/36b36ab858965f3803755e598030eaf8

And the Makefile - https://gist.github.com/Abhiroop/c67815df12bf27ed305158d4118c6ed6 (The Makefile is a minor modification of the helloworld project https://github.com/gramineproject/gramine/tree/master/CI-Examples/helloworld and the manifest adds the shared libraries required by a ghc-produced binary).

Can someone help with debugging this? Thanks!

[boryspoplawski]

You can try removing undeeded libraries (most of them are provided by Gramine in gramine.runtimedir()) and then run it with loader.log_level = "trace" to see what exact syscalls are issued (especially around the hang)

[boryspoplawski]

Gramine's madvise is rather dummy implementation supporting only basic cases and you hit one that is not supported:

gramine/libos/src/bookkeep/libos_vma.c

Line 1301 in 6a13434

if (!(vma->prot & PROT_WRITE)) {

[Abhiroop]

Thank you! I am currently trying to port the GHC runtime to run on Gramine, however, there seem to be a number of Linux system calls that are obstructing this. The list so far has been - madvise (as discussed above), clock_gettime, gettimeofday, pselect6 (https://linux.die.net/man/2/pselect6). I am trying to patch the runtime to remove their usage one by one but I wonder if there is a list of Gramine syscalls with dummy implementations.

For instance, I am not too familiar with the implementation of pselect6 but I see that Gramine has one (

gramine/libos/src/sys/libos_poll.c

Lines 329 to 350 in 74e74a8

	long libos_syscall_pselect6(int nfds, fd_set* readfds, fd_set* writefds, fd_set* errorfds,
	const struct __kernel_timespec* tsp, void* _sigmask_argpack) {
	struct sigset_argpack* sigmask_argpack = _sigmask_argpack;
	if (sigmask_argpack) {
	if (!is_user_memory_readable(sigmask_argpack, sizeof(*sigmask_argpack))) {
	return -EFAULT;
	}
	int ret = set_user_sigmask(sigmask_argpack->p, sigmask_argpack->size);
	if (ret < 0) {
	return ret;
	}
	}
	if (tsp) {
	struct __kernel_timeval tsv;
	tsv.tv_sec = tsp->tv_sec;
	tsv.tv_usec = tsp->tv_nsec / 1000;
	return libos_syscall_select(nfds, readfds, writefds, errorfds, &tsv);
	}
	return libos_syscall_select(nfds, readfds, writefds, errorfds, NULL);
	}

), however, the GHC runtime makes a call to select that ends up hanging the application.

[Abhiroop]

When debugging, I see repeated appearance of these logs:

[P1:T1:helloworld] trace: ---- pselect6(2, 0xb730a10, 0xb730a90, 0, 0, 0) ...
[P1:T1:helloworld] trace: ---- return from pselect6(...) = 0x0

[Abhiroop]

I see from the implementation that the value of the last argument _sigmask_argpack is 0 and that will cause the issue.

[boryspoplawski]

Regarding pselect6: there's a PR pending with rewrite/cleanup: #1067, you could try that.
As far as I know clock_gettime and gettimeofday should be implemented sufficiently for must usages.

I am trying to patch the runtime to remove their usage one by one but I wonder if there is a list of Gramine syscalls with dummy implementations.

There is no such list and it wouldn't make sense. Most of the time it's certain combination of options/flags that is not supported.
If you are inclined to look into the Gramine code, the list of supported syscalls is here: https://github.com/gramineproject/gramine/blob/6a13434ee71cc0e420f6cf95a789d941f0b328c2/libos/src/arch/x86_64/libos_table.c