Performances degradataion in 1.9.0

[jibidus]

Hi @jlink,
I noticed a performance degradation since jqwik 1.9.0 in some of my tests.
I have many tests which heavily use Domain feature, which duration have increased by a factor 5. For example, here are durations for a single test:

Jqwik version	Test duration
1.8.5	7s
1.9.0	36s
1.9.1	36s
1.9.2	36s

I will try to provide a project to illustrate this issue.

[jlink]

@jibidus Thanks for reporting.

When you distill an illustrating example you might want to focus on which part is taking longer. I assume it happens during sample generation so typically it would be large collections of objects, some kind of recursion or heavily nested flat mapping.

[jlink]

Example from @jibidus: https://github.com/jibidus/jqwik-perfs-demo/blob/main/src/test/java/org/example/Test.kt

import net.jqwik.api.Arbitraries
import net.jqwik.api.Arbitrary
import net.jqwik.api.Combinators.combine
import net.jqwik.api.ForAll
import net.jqwik.api.Property
import net.jqwik.api.Provide
import net.jqwik.api.constraints.UseType
import net.jqwik.api.domains.Domain
import net.jqwik.api.domains.DomainContext
import net.jqwik.api.domains.DomainContextBase
import net.jqwik.kotlin.api.any
import net.jqwik.kotlin.api.anyForSubtypeOf
import net.jqwik.kotlin.api.anyForType

@Domain(MyDomain::class)
@Domain(DomainContext.Global::class)
class Test {

    @Property
    fun test(@ForAll model: @UseType Model) {
    }
}


class MyDomain : DomainContextBase() {
    @Provide
    fun model(): Arbitrary<Model> = anyForSubtypeOf<Model>()

    // Workaround
//    fun model(): Arbitrary<Model> = Arbitraries.oneOf(anyForType<Model1>(), anyForType<Model2>(), anyForType<Model3>())
}

sealed interface Model

data class Model1(val attr1: Int, val attr2: Int, val attr3: Int, val attr4: Int) : Model
data class Model2(val attr1: Int, val attr2: Int, val attr3: Int, val attr4: Int) : Model
data class Model3(val attr1: Int, val attr2: Int, val attr3: Int, val attr4: Int) : Model

[jibidus]

Sorry, I deleted my comment because this does not illustrate this issue (test duration is identical between 1.8.5 and 1.9.0 in this example). It illustrates another performance issue with anyForSubtypeOf(). I still working on an example about this issue.

[jlink]

Sorry, I deleted my comment because this does not illustrate this issue (test duration is identical between 1.8.5 and 1.9.0 in this example). It illustrates another performance issue with anyForSubtypeOf(). I still working on an example about this issue.

At least it shows that it takes much longer than the workaround.

[jibidus]

Here is a simple example which illustrate the observed behaviour:

import net.jqwik.api.Arbitrary
import net.jqwik.api.Combinators
import net.jqwik.api.ForAll
import net.jqwik.api.Property
import net.jqwik.api.Provide
import net.jqwik.api.domains.Domain
import net.jqwik.api.domains.DomainContextBase
import net.jqwik.kotlin.api.any
import net.jqwik.kotlin.api.anyForSubtypeOf
import net.jqwik.kotlin.api.combine

@Domain(MyDomain::class)
@Domain(AnotherDomain::class)
class Test {

    @Property
    fun test(@ForAll model: Model) {
    }
}

class MyDomain : DomainContextBase() {
    @Provide
    fun model(): Arbitrary<Model> = anyForSubtypeOf<Model>()

    @Provide
    fun attribute() = combine {
        val value by Double.any()
        combineAs {
            Attr(value)
        }
    }
}

class AnotherDomain : DomainContextBase() {
    @Provide
    fun attribute() = combine {
        val value by Double.any()
        combineAs {
            Attr(value)
        }
    }
}

sealed interface Model

data class Model1(val attr1: Attr) : Model
data class Model2(val attr1: Attr) : Model

data class Attr(val value: Double)

This involves 3 elements:

• duplicated arbitrary providers in different domains (there is no duration difference without this)
• these providers uses combine() kotlin DSL with a single arbitrary (there is no duration difference with Double.any().map{} instead)
• usage of anyForSubtypeOf<>() (the duration difference is smaller with Arbitraries.oneOf(anyForType<SubClass1>().enableRecursion(), …))

The demo project is available here.

So, to summarize, the removal of duplicated arbitrary provider, which is not desired i guess, solves the "issue". So we may probably close this issue.

[jlink]

So, to summarize, the removal of duplicated arbitrary provider, which is not desired i guess, solves the "issue". So we may probably close this issue.

It would still be interesting to see, why overridden methods make a difference from 1.8.5 to 1.9.0.

I will have a look at it, but I cannot promise to do it this week.

[jibidus]

Thanks

[jlink]

I spent a couple of hours trying to trace the differences in code execution from 1.8.5 to 1.9.0.
There don't seem to be any :-/

The two things that have changed are:

1. Introduction of org.jspecify annotations to better align with Kotlin type system: This should not have any runtime implications and was mainly done to satisfy compilation issues with Kotlin 2.0.
2. Move from Kotlin 1.9 to 2.0: This could have runtime implications but I don't know how exactly.

The thing that I don't understand at all is that in version >= 1.9.0 the code with two @Domain annotations is not just slower by a linear factor but seems to get slower quadratically with the number of tries (or something similar).

Cutting a long story short: I'm out of ideas - or maybe just dumb.

[jlink]

One possible next step could be to get rid of the Kotlin part in the example and see if it has anything to do with Kotlin.

[vlsi]

@jlink , have you profiled the execution? I wonder if the profiler would show the key resource consumption, so we could compare it between the releases

[jlink]

Using

    @Provide
    fun model(): Arbitrary<Model> = Arbitraries.oneOf<Model>(
        anyForType<Model1>(),
        anyForType<Model2>()
    )

speeds the code up tremendously. I conclude that the issue is somehow related to anySubtypeOf.

Maybe there's a way to get rid of recursion there? Or simplify it in a different way?

[jlink]

@jlink , have you profiled the execution? I wonder if the profiler would show the key resource consumption, so we could compare it between the releases

Actually no. Wondering why I haven't. Sadly, I have to let it go for today :-(

[jibidus]

I wrote "same" test in Java :

@Domain(Test.MyDomain.class)
@Domain(Test.AnotherDomain.class)
public class Test {
    @Property
    void test(@ForAll Model model) {
    }

    static class MyDomain extends DomainContextBase {
        @Provide
        Arbitrary<Model> model() {
            return Arbitraries.oneOf(Arbitraries.forType(Model1.class).enableRecursion(), Arbitraries.forType(Model2.class).enableRecursion());
        }

        @Provide
        Arbitrary<Attr> attribute() {
            return Arbitraries.doubles().map(Attr::new);
        }
    }

    static class AnotherDomain extends DomainContextBase {
        @Provide
        Arbitrary<Attr> attribute() {
            return Arbitraries.doubles().map(Attr::new);
        }
    }

    interface Model { }

    static class Model1 implements Model {
        public Model1(Attr attr1) {
            this.attr1 = attr1;
        }

        private Attr attr1;
    }

    static class Model2 implements Model {
        public Model2(Attr attr1) {
            this.attr1 = attr1;
        }

        private Attr attr1;

    }

    static class Attr {
        private Double value;

        public Attr(Double value) {
            this.value = value;
        }
    }
}

I did not observed any difference with this test between 1.9.0 and 1.8.5.

But this test uses

• Arbitraries.oneOf(Arbitraries.forType().enableRecursion(), …)) instead of anyForSubtypeOf<>().
• Arbitraries.doubles().map() instead of combine()

When, in kotlin test, I replace anyForSubtypeOf<>() by Arbitraries.oneOf(), the performance is better, but still 3x compared to 1.8.5.
And then, when still in Kotlin test, I replace combine() by Arbitraries.doubles().map(), the performance is identical between 1.9.0 and 1.8.5.

[jibidus]

I run a profiler, and noticed this:

1.8.5

1.9.0

I hope this will help.

[jlink]

I run a profiler, and noticed this:

I hope this will help.

The percentage are aggregated time I assume.

What about number of calls? If they are similar in both cases then something strange is going on. In Memoize nothing has changed except for the addition of type annotations - which should have no runtime consequences.

[vlsi]

In any case, it does not sound right to have a lot of Memoize....computeIfAbsent...put call call stacks.
It means the shrinkable is not cached for some reason, so we should probably address this issue first, and only then compare the performance.