STIC benchmark for Toit, uLisp and CircuitPython

[davidg238]

Toit, uLisp and CircuitPython running a simple STIC benchmark.

[floitsch]

Thanks for running the benchmark.

There was one mistake in the benchmark and I uploaded a PR (davidg238/performance_stic#1). The array-write benchmark wasn't actually writing to the array. It increased Toit's runtime, but Toit is still faster than uLisp or CircuitPython.

Most of the numbers are close to what I would have expected.

A few things stick out, though: Python has optimized their maps/dictionaries much more. That's to be expected, since their object model depends on it. They probably have bytecodes to work with dictionaries, whereas Toit is mostly pure Toit code. We could definitely improve that.

The floating point benchmark could also be better. Python might fuse some common operations together. Doesn't take much to significantly increase the speed this way.

I did a small experiment and inlined the local constants:

  float-math-speed-test-inlined:
    e := 5.0
    runtime := Duration.of:
      ITER_300K.repeat :
        10.repeat:
          e = (e * 87.0 + 53.0) * -87.0 + 42461.0
    return runtime.in-ms / 1000.0

That alone increased the speed by a factor of 2... Probably an optimization we should add.

Toit starts to shine when method calls are involved.

I think this reflects our priorities: make Toit work well with real-world programs, and not just benchmarks. Clearly, we would like to eventually tune our implementation to be at least as fast on these benchmarks as CircuitPython, but it's not as important to us as making sure typical programs run reliably and fast. In other words: if you are doing an FFT you are likely off-loading that task to a C function anyway; in contrast, object-oriented programming is what remains as part of your high-level language.

[floitsch]

Actually: the inlining doesn't seem to have a noticeable impact when running on the ESP32. I initially did my experiment on the host where it was clearly faster. On the device there wasn't much difference.

[davidg238]

Thanks for catching that, in aligning the code with CircuitPython, I forgot the rule about not playing with the collection in an iterator. With your fix as context, more concretely:

class Circle:

    radius/int := 2

    bigger -> none:
        radius = radius * 2

    stringify -> string:
        return "cirle: $radius"

main:

    groucho := [5, 5, 5]
    print "groucho before: $groucho"
    total := 0
    groucho.do:
        it = 6  // nope 
        total = total + it  // `it` is no longer bound to the current element
    print "groucho after: $groucho, with total: $total"

    harpo := ["ONE", "TWO", "THREE"]
    print "harpo before: $harpo"
    harpo.do:
        it.to-ascii-lower
        // nope, strings are immutable
    print "harpo after: $harpo"

    zeppo := [ Circle, Circle, Circle]
    print "zeppo before: $zeppo"
    zeppo.do:
        it.bigger  // updates each element, does not modify the collection 
    print "zeppo after bigger: $zeppo"

Toit is fast enough for what I am doing and the life-cycle tooling is superior. But some folks ask for benchmarks.
Looking at the desktop benchmark of Toit vs Zig (in the revised README), I did not see a strong reason to benchmark MicroZig