Fixed runtime for WebAssembly/JavaScript

[HeinrichApfelmus]

Hi Lennart! I’m developing threepenny-gui, a GUI library in Haskell that uses the web browser as a display. My goal is to provide a GUI library in Haskell with an extremely low barrier to entry. I’m currently working on compiling to WebAssembly / JavaScript via MicroHs — this removes the need to run a native process (web server, containing the Haskell GUI logic).

In order to lower the barrier even further, I would like to ask for the following feature:

• Currently, in order to compile a Haskell program to WebAssembly / JavaScript, users of my library would need to have Emscripten installed, which is called by mhs.
• Instead, I want to compile the MicroHs C runtime once using Emscripten, and then have this fixed runtime be able to load “combinator bytecode” ("skicode" ?) dynamically. In this way, users of my library can use MicroHs to compile Haskell to combinator bytecode, but they do not need to have Emscripten installed, instead they hook up the bytecode with a predefined runtime.js file.

I suppose the main difficulty here is to accept combinator bytecode that calls foreign import javascript. This probably needs some form of combinator bytecode linker.

I managed to simplify the problem in that I only need a specific, fixed set of foreign import javascript in the runtime; I do other calls to JavaScript dynamically using eval() .

[augustss]

This sounds like a great idea. It will be fantastic if we can compile Haskell code with a GUI to run fully in the browser.

If you only need a fixed set of foreign import then those can be added in eval.c (like many others already are). Then mhseval does exactly what you need, but I'm not sure how it will read the combinator code. I'm sure we can figure out a way.

[augustss]

If you compile a program that uses all the the foreign import javascript you need to a C file, you can simply copy&paste the stubs to eval,c and put them inside some #ifdef EMSCRIPTEN. I would accept a patch like that to make them available in mhseval.

[augustss]

I'm quite excited about porting threepenny-gui to MicroHs, so if you need any help please let me know.

[HeinrichApfelmus]

If you compile a program that uses all the the foreign import javascript you need to a C file, you can simply copy&paste the stubs to eval,c and put them inside some #ifdef EMSCRIPTEN. I would accept a patch like that to make them available in mhseval.

Yes, I'd be happy to do that.

The fixed set of imports that I need is given by

foreign import javascript "console.log(UTF8ToString($0))"
    ffiDebug :: CString -> IO ()
foreign import javascript "eval(UTF8ToString($0))"
    ffiRunEval :: CString -> IO ()
foreign import javascript "return stringToNewUTF8(JSON.stringify(eval(UTF8ToString($0))))"
    ffiCallEval :: CString -> IO CString

They are somewhat specific, but do not seem to be more specific than what is provided by all targets of EMSCRIPTEN, so these should be ok.

Oh, and I promptly forgot that I need a foreign export as well. 😅 This is necessary for handling GUI events such as mouse clicks, which call Haskell code from JavaScript.

foreign export javascript callbackHs
    :: CString -> CString -> IO ()

Essentially, this is about offering entrypoints other than main into the Haskell code. I'm not entirely sure where to put the implementation of this one, though. 🤔 The first argument is essentially the name of an event handler to be called, and the second argument represents the arguments to that handler. I wrote some stuff in Haskell where you can register event handlers and have them garbage collected when fall out of use (provided support for weak pointers, but that's a different issue). But these bits require containers and are somewhat specific to javascript-ffi-threepenny package.

[augustss]

Since MicroHs recently got foreign exports for C it should be possible to do for JavaScript as well. But I don't think it will work with your vision of just having the combinator bytecode, since the foreign export wrapper is specific to the compiled Haskell code.

I assume you could also use foreign import javascript "wrapper" instead of export? That might be easier.

[augustss]

Will the callback into Haskell happen while Haskell is calling out to JavaScript?

[HeinrichApfelmus]

Since MicroHs recently got foreign exports for C it should be possible to do for JavaScript as well. But I don't think it will work with your vision of just having the combinator bytecode, since the foreign export wrapper is specific to the compiled Haskell code.

Yes — foreign export javascript works when compiling with emscripten. But indeed — I would like to compile the runtime with emscripten and then use bytecode to feed it.

I assume you could also use foreign import javascript "wrapper" instead of export? That might be easier.

What is "wrapper"? 🤔 If you mean importing a function

foreign import javascript setCallback :: (CString → CString → IO ()) → IO ()

that assigns a Haskell function to a global variable — yeah, that would work, but I would need some way of marshaling the CString → CString → IO () function pointer.

Will the callback into Haskell happen while Haskell is calling out to JavaScript?

Almost no. This problem occurs in the remote procedure call model that threepenny-gui uses as well. My current design decision is that

• Yes, the callback into Haskell can happen while Haskell is calling out to JavaScript. (The browser may call event handlers that I have installed as part of some JavaScript functions that I am calling.)
• No, such callbacks into Haskell are not executed immediately. They are scheduled for execution at a later time, but they do not influence the control flow of the JavaScript that is being called out to.

[augustss]

By "wrapper" I mean the special Haskell FFI construct that allows you to turn Haskell functions into C function pointers.
See, e.g., FFI

It looks like this:

foreign import ccall "wrapper" createAddPtr :: (Int -> Int) -> IO (FunPtr (Int -> Int))

where "wrapper" is a magic word that triggers this. But it doesn't matter if you could use that, since MicroHs doesn't currently support that. It's not easy to implement; you need to generate the stub code at runtime since C doesn't have closures, but only code pointers.

I've just made some modifications so that foreign import functions are allowed to call back into foreign export functions. This doesn't directly help you, though.

Here is what I suggest. I will add a C function to eval.c, it will have the C type

char* apply_sp(uintptr_t, char*);

And it corresponds to a Haskell function with this type

apply_sp :: StablePtr (CString -> IO CString) -> CString -> IO CString

You call it from C/Javascript, it takes a StablePtr to a Haskell function and a CString and the Haskell function will be called with that CString. Whatever the Haskell function returns is then returned back to C/Javascript.
This is very easy for me to implement, and I think you can do what you need with that.

[augustss]

I hope the callbacks you want to make into Haskell code do not happen asynchronously (i.e., from a thread different from the MicroHs evaluator thread). If they do, the MicroHs runtime need to know how deal with threads and locks.

[HeinrichApfelmus]

Here is what I suggest. I will add a C function to eval.c, it will have the C type

  char* apply_sp(uintptr_t, char*);

And it corresponds to a Haskell function with this type

  apply_sp :: StablePtr (CString -> IO CString) -> CString -> IO CString

This is very easy for me to implement, and I think you can do what you need with that.

Yes, this sounds like it can do what I need! — I can use something along the lines of

  foreign import javascript "callbackHs = $0"
      setCallbackHs :: StablePtr (CString → IO CString) → IO ()

to store a StablePtr to a Haskell function in a global variable and then use apply_sp on that global variable to call the Haskell function from C/JavaScript.

One question, though: How does marshaling work for StablePtr? In order to write the foreign import above, the StablePtr needs to be convertible to some C type. Internally, it's defined as newtype StablePtr a = StablePtr Word, but that definition is not exported and I am unable to convert it to CUIntPtr outside of MicroHs internals. Presumably, the StablePtr type needs a line similar to

MHS_FROM(mhs_from_StablePtr, SETINT, uvalue_t);

in eval.c as well?

I hope the callbacks you want to make into Haskell code do not happen asynchronously (i.e., from a thread different from the MicroHs evaluator thread). If they do, the MicroHs runtime need to know how deal with threads and locks.

Fortunately not. They do not happen in source code sequence because that sequence is hard to keep track of when two processes make nested remote procedure calls with each other — but they are not assumed to happen concurrently, it's just some alternative sequence.

[augustss]

If you pull, you'll have apply_sp() and StablePtr works. There's a test.

[HeinrichApfelmus]

Thanks! 😍

Hm. 🤔 When using foreign import javascript, I cannot marshal StablePtr just yet:

line 31: foreign import javascript "Haskell.callback = $0"
line 32:    setHaskellCallback :: StablePtr (CString -> IO CString) -> IO ()

give the error message

mhs: uncaught exception: error: "./src/Foreign/JavaScript/MicroHs.hs": line 32, col 27:
  Not a valid Javascript return type: (StablePtr ((Ptr CChar) -> (IO (Ptr CChar))))

🤔 But I suppose that I can use foreign import ccall and let emscripten deal with the marshaling to JavaScript. I'll give it a try.

[augustss]

Oh, sorry. I forgot to add StablePtr to the JavaScript FFI. I'll fix it tomorrow.

[augustss]

I think I've fixed it.

[HeinrichApfelmus]

I think I've fixed it.

Yes, it's working, thanks! I need to work on the encoding of arguments and get back to you with a pull request that hardwires the fixed foreign imports above into eval.c.

After that, we need a way to pass combexpr (and combexprlen) dynamically from the JavaScript context to main.c somehow. I suppose that setting up an appropriate stdin or file in the JavaScript/Emscripten context would be that way to go, as this fits the UNIX/C worldview that mhs is based on. 🤔