Function, Function Pointer, Closure and C Function Pointer Interop

[tothambrus11]

As part of my work on mapping the C type system to Hylo, I was trying to map C function pointers, and I made some preliminary designs on the UX of interoperating between Hylo and C functions, and closures with empty empty environments. I would welcome feedback on both this design and potential implementation challenges.

My main inpirations were Swift's function pointer mapping strategy and Rust's extern "C" declarations.

Function - Closure Interop

In Hylo, closures with empty environments can be represented as a function pointer (1 word). It would be therefore convenient if we could save pointers to functions so that we can call them later at another place (converting a function to a lambda type). It would essentially make it so that wherever we can use a lambda, we could also pass a function with the correct signature. AFAIU, this would be a trivial operation, as the representation is already the same under the hood.

(Currently it's crashing the compiler)

public fun f(_ a: Int, _ b: Int) -> Int {
    a + b
}

fun use(_ f: [](Int, Int) -> Int) -> Int {
    f(1, 2)
}

public fun main() {
    print(f(_:_:)(1, 2)) // this works
    
    // passing a lambda
    _ = use(fun [](_ a: Int, _ b: Int) -> Int {
        a + b
    })
    
    // passing a function reference should also work here
    _ = use(f(_:_:)) // currently crashes
   
    // this operation should be possible
    let f2: [](Int, Int) -> Int = f(_:_:) // currently crashes, same problem as above
}

This conversion is supported in Swift: https://godbolt.org/z/fabzfq7cs

External C Functions

The current @extern annotation only declares a Hylo function that is implemented externally (but using Hylo calling conventions). We should have a way of representing external functions with different calling conventions. Knowing the calling convention of a function is necessary for the backend, so that it can emit different calling code. This could be spelled like @extern(C), @convention(C), @conv(C), @C. For consistency and clarity, I will stick with @conv(C) for now.

// FFI declaration for a C function
// that takes two `int32_t`s and returns an `int32_t`
@conv(C) @extern fun add(_ a: Int32, _ b: Int32) -> Int32

Note, calling C functions is a non-trivial task. Rust reimplemented the function calling and struct layouting algorithms for it supported platforms' ABIs, and Swift leverages Clang to provide the necessary information. I heard that there are some compiler-specific layouting irregularities in MSVC regarding bitfields, so it might be challenging to link to C code compiled with any compiler. (I'll update after my discussion with a Rust Bindgen contributor)

Now such external function can be called from Hylo because we can take into account its corresponding calling convention:

precondition(add(3, 4) == 7)

C Function Pointers

Now that we can call C functions, we should be able to save a pointer to the C function using a closure with an empty environment. This closure type should now contain the calling convention:

let addRef: @conv(C) [](Int32, Int32) -> Int32 = add(_:_:)
precondition(addRef(1, 2) == 3)

The call should again succeed, as the callsite knows exactly what calling convention the function being pointed to has.

Passing C Function Pointers to a C function

Given the following external C function that takes a function pointer and two int32_ts:

@conv(C) @extern fun apply_in_c(_ a: Int32, 
                                _ b: Int32,
                                _ f: @conv(C) [](Int32, Int32) -> Int32
                               ) -> Int32

We should be able to pass in a pointer to a C function that already has the C calling convention:

let result = apply_in_c(1, 2, add(_:_:)) // note, add is a C function from before
precondition(result == 3)

We can then also save it into a C closure and call with that:

let addRef: @conv(C) [](Int32, Int32) -> Int32 = add(_:_:) // note, addRef is a pointer to a C function
let result = apply_in_c(1, 2, addRef)
precondition(result == 3)

Converting Between Hylo and C Function Pointers

Assume we have a Hylo function sub:

fun sub(_ a: Int32, _ b: Int32) -> Int32 {
    return add(-a, b)
}

We want to pass this function to our apply_in_c function, but that accepts a function with C calling convention (the C compiler doesn't know anything about Hylo).

let r1= apply_in_c(1, 2, sub(_:_:)) // note, sub is a Hylo function
precondition(r1== -1)


let subRef: [](Int32, Int32) -> Int32 = sub(_:_:)  // note, subRef is a pointer to a Hylo function

let r2= apply_in_c(1, 2, subRef) // implicitly converted to a C calling convention function
precondition(r2== -1)

To make the above code work, we can generate a wrapper function under the hood that can be called from C but is able to call the Hylo function, and make an implicit conversion between the received Hylo function pointer and the expected C function pointer:

@conv(C) public fun sub_callable_from_c(_ a: Int32, _ b: Int32) -> Int32 {
    sub(a, b) // call the Hylo function
}

Converting the other way could be also desirable, when we have a C function pointer that we want to pass to a Hylo function expecting a closure without environment:

fun use(_ f: [](Int32, Int32) -> Int32) { f(1, 2) }

precondition(use(add(_:_:))) // Note, add is a C function

When generating these wrapper functions, we shouldn't generate them wastefully: only generate wrappers that are actually being used, and only once per conversion direction (rather than per usage).

Calling C Function Pointers

A C function may return a C function pointer that we may want to call from Hylo:

// FFI declararation for a C function that returns a function pointer
@conv(C) @extern fun get_add_function() -> (@conv(C) [](Int32, Int32) -> Int32)

Calling A C function pointer is no different than calling an @extern @conv(C) function, we can again directly call it since there is type information about the calling convention of the closure:

let addRef: @conv(C) [](Int32, Int32) -> Int32 = get_add_function()
precondition(addRef(1, 2) == 3)

+1: Mapping Hylo Functions With a Non-Empty Environment

It could be possible to map a Hylo function with a non-empty environment to a struct with a pointer to the environment and a C function pointer to a special wrapper function. The wrapper function's first argument would be the pointer to the environment, and the rest being the Hylo closure's parameters.

This is not particularly useful for interacting with existing C libraries but rather if we are writing C code ourselves. Careful consideration will need to be taken for designing this conversion and the lifetime guarantees for the environment, and what requirements we want to set on the environment's type for this conversion.

[tothambrus11]

Regarding the (offline) discussion about not allowing the lambda types to become C convention qualified:

I would not worry at all about making functions generic over which calling convention they support. It is impractical (and would probably only work when monomorphizing) and we wouldn't win much from that. Generating wrapper functions that in turn call our original function with its correct calling convention are always almost free, since LLVM can easily optimize away such calls with LTO. One catch is that we should probably only allow conversions between C an Hylo calling conversions when all argument and return types are C-representable.

In case of a wrapper, we need to bless the compiler so that it knows that that type is actually callable, and in particular, lower the call according to the specified calling convention, so it might also complicate things a bit. Error diagnostics might be also slightly better if we treat function pointers to Hylo vs C functions similarly, and type names in error messages would also be shorter and easier to comprehend.

There is precedence in both Rust and Swift for this pattern:

Rust:

fn main() {
    // A captureless lambda
    let my_lambda = |x: i32| println!("Lambda called with: {}", x);

    // Coercing the lambda to a C function pointer
    let c_fn_ptr: extern "C" fn(i32) = my_lambda;

    // You can then pass this pointer to C code or call it directly
    c_fn_ptr(42);
}

Swift: https://github.com/swiftlang/swift/blob/main/docs/HowSwiftImportsCAPIs.md#function-pointers

It would make it nice and symmetric if everything that is a Hylo function declaration we could 1:1 map to closures with empty environment, but if it makes the type checking more complex for the general case, I would be fine with having some wrapper (I would need some convincing still that it's surely the case). Maybe we can see in the Swift compiler how much complexity this is adding.

[tothambrus11]

I realized that the conversion between arbitrary C and Hylo function pointers is not generally possible. The issue is that we must know the exact function that needs to be wrapped, and we can either do that at compile time, or at runtime.

Motivating Example

The following Swift code shows why it is challenging to perform the conversion given an arbitrary captureless closure:

let tobecalled: (Int32, Int32) -> Int32
if true {
    tobecalled = { a, b in
        return a + b
    }
} else {
    tobecalled = { a, b in
        return a - b
    }
}

c_taking_lambda(tobecalled, 2, 3)
// ^ error: A C function pointer can only be formed from a reference to a 'func' or a literal closure.

Approaches to Runtime Conversion

A runtime wrapper would need to look like this:

fun binary_op_callable_from_c(hylo_function f: (Int,Int) -> Int, a: Int, b: Int)) {
    f(a, b);
}
// ^ Not viable because we modify the signature by adding an additional argument, C doesn't expect this.

or

fun wrap<E>(hylo_function f: (Int, Int) -> Int) -> (@conv(C) [E](Int, Int) -> Int){
  return fun [f](_ a: Int, _ b: Int) -> Int {
    f(a,b)
  }
}
// ^ Not viable because C function pointers cannot carry an environment.

The only possible runtime solution for calling arbitrary lambdas would be to emit a wrapper for all Hylo functions, store these in a global map, and upon conversion, retreive the matching function from the global map. This is not really suitable because it causes the binary to significantly grow in size, and the conversions are quite costly (map lookup).

Approaches to Compile-Time Conversion

The limitation Swift put on compile-time conversions makes a lot of sense: "A C function pointer can only be formed from a reference to a 'func' or a literal closure". In practice, I think this limitation makes a lot of sense.

What this essentially means is that if you intend to have dynamic choosing between a set of Hylo functions to pass to a C function as a function pointer, you will need to explicitly convert each function to a C function pointer before erasing the compile-time information about them about which function they refer to:

func add(a: Int32, b: Int32) -> Int32 {
    return a + b
}
func sub(a: Int32, b: Int32) -> Int32 {
    return a - b
}

let tobecalled: @convention(c)(Int32, Int32) -> Int32
if true {
    tobecalled = add(a:b:) // comptime conversion
} else {
    tobecalled = sub(a:b:) // comptime conversion
}

taking_lambda(tobecalled, 2, 3)

[kyouko-taiga]

I didn't read the proposal in detail but it seems to rest on a faulty premise. Swift doesn't have a concept of "capture-less" closures AFAIK. (T) -> U can represent any function from T to U, including those with non-empty captures.

That is not the case in Hylo. [](T) let -> U can only represent functions without any captures and so I believe it solves most of the issues you have raised.

[tothambrus11]

@kyouko-taiga Thanks for the reply. I was aware of Swift not having captureless closures, but the point is that the issue still persists in Hylo, where we do have captureless closures. The main issue is when we have a Hylo lambda value (not literal), then we essentially have a function pointer coming at runtime. Writing a C-callable static wrapper function that calls that arbitrary function pointer that we know only at runtime is impossible because we would need to somehow pass the runtime function pointer value into the c-callable function, without modifying the signature or introducing an environment (then our wrapper would no longer be C-callable).

[kyouko-taiga]

It is easy enough to perform eta expansion at the language boundary manually. At least we can do that until more features are added in the language.

[dabrahams]

Well, there's @convention(thin) which is not officially supported but documented in the SIL docs.