Replacement for ClientSession when not using transactions

[XSpielinbox]

When using Spring Boot 3 (Webflux) with mongock-springboot-v3 and mongodb-reactive-driver and transactions disabled the code

@ChangeUnit(id = "test", order = "3", author = "test")
@Slf4j
public class MongockExperiment {

    private final ReactiveMongoTemplate reactiveMongoTemplate;

    public MongockExperiment(ReactiveMongoTemplate reactiveMongoTemplate) {
        this.reactiveMongoTemplate = reactiveMongoTemplate;
    }

    /**
     * Add criteria and event type
     **/
    @Execution
    public void migrate() {
        reactiveMongoTemplate.findAll(UserDTO.class).collectList().delayElement(Duration.of(3, ChronoUnit.SECONDS)).flatMapMany(list -> {
            list.forEach(user -> user.setFirstName(user.getFirstName() + "test"));
            return Flux.fromIterable(list);
        }).flatMap(reactiveMongoTemplate::save).subscribe();
    }

produces the following error:

Ensuring the lock
<time> ERROR 19120 --- [     parallel-2] reactor.core.publisher.Operators         : Operator called default onErrorDropped

reactor.core.Exceptions$ErrorCallbackNotImplemented: io.mongock.driver.api.lock.LockCheckException: Lock cannot be ensured after being cancelled
Caused by: io.mongock.driver.api.lock.LockCheckException: Lock cannot be ensured after being cancelled
	at io.mongock.driver.core.lock.LockManagerDefault.ensureLockDefault(LockManagerDefault.java:166)
	at io.mongock.driver.api.lock.guard.proxy.LockGuardProxy.invoke(LockGuardProxy.java:50)
	at io.mongock.driver.api.lock.guard.proxy.LockGuardMethodHandler.invoke(LockGuardMethodHandler.java:25)
	at org.springframework.data.mongodb.core.ReactiveMongoTemplate_$$_mongock_3da_0.save(ReactiveMongoTemplate_$$_mongock_3da_0.java)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:386)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:250)
	at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:335)
	at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:294)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:150)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onSubscribe(MonoFlatMapMany.java:245)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8671)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:195)
	at reactor.core.publisher.MonoDelayElement$DelayElementSubscriber.run(MonoDelayElement.java:170)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
<time> DEBUG 19120 --- [<db url>] org.mongodb.driver.protocol.command      : Sending command '{"hello": 1, "$db": "admin

I suppose it is due to interweaving of multiple operations. All online Examples I could finde of Mongock use ClientSession that seems to do something locking-related, but ClientSession only seems to work with transactions. How can one do the migration without using transactions?

[dieppa]

Hi @XSpielinbox ,

As @osantana85 mentioned in the issue you raised, we are adding the Reactive driver for Spring data v4 (as well as the documentation for Springboot 3.x).

So, in the meanwhile(and corroborated by your comment in the issue's description When using the Builder approach instead of the annotation everything works - even with mongock-springboot., you can use the Builder approach and it works, right?

If that's true, I am a bit confused

1. Is this a new problem because you are doing something different from what you were doing when said When using the Builder approach instead of the annotation everything works - even with mongock-springboot.?
2. Is a new problem you just figured out now, without changing any approach?
3. Do you currently have any workaround?

Thanks

[XSpielinbox]

thanks for the reply. Any help is greatly appreciated!

I'm sorry, if I wasn't clear. I chose the workaround for now to explicitly define the ConnectionDriver Bean despite the Annotation approach. Simple migrations work absolutely fine with that. I can test again, whether switching to the Builder approach solves above issue.

As the errors of using ClientSession (for a different migration) were independent of that and just related to enabling or disabling transactions my above question seemed independent of the general Springboot-v3/reactive support, though I might be wrong about that. I therefore just mentioned the frameworks used as a general reference just in case, knowing that you don't officially support it yet.

For above question I don't have any workaround at the moment. I don't know whether It is just an error in my implementation and I just have to use different methods/fields/approaches for it to work or whether this is an mongock-internal bug related to the springboot-v3 support (though that seemed unlikely from what I know at the moment).

I hope I could clarify your question.

[XSpielinbox]

I just did some further tests:
Yes, I have the same problem also with the builder approach:

Above code results in this error when applying it. It does not change any data, but nevertheless writes an successful execution to the mongockChangeLog in the Database. When rerunning the migration it just passes over it (without error) as it was saved in the database as applied successfully (even though it wasn't).

[dieppa]

Hi again. lets divide this into smaller issues:

Error Lock cannot be ensured after being cancelled

This is better explained in the official documentation. It just says that although Mongock's nature is synchronous, you can still use reactive. However, you must block the sync call in the changeUnits(obviously not need to do so in the rest of your application). You can see an example here

The reason behind this is that the distributed lock is acquired before starting the process, then each changeUnit is applied synchronously(at least that's the expectation) and then the lock is released. Keep in mind that Mongock provides a very robust lock mechanism, so every I/O call you make within a changeUnit, Mongock ensures the lock is acquired(this is transparent for you).
However, in your scenario, the changeUnit operates asynchronously, so by the time it gets the response asynchronously, and then it tries to do another I/O operation(for which Mongock checks the lock is still taken), Mongock may have already released(cancelled) the lock.

In summary, your problem should get fixed just by blocking it with get(), await(), etc.

ClientSession

The ClientSession is just for the transaction, if you are working without transaction, you shouldn't be bother 😉

Reactive Spring data compatibility

Currently you can use the MongoDB Java Reactive Streams driver(brought as transitive dependency with Spring data) in your ChangeUnits, so it's not a critical priority. However, as we are receiving more and more demand, we'll add the Spring data reactive driver soon.

I hope this answer your questions and helps you to progress. If you still get stuck, please let us know.

[XSpielinbox]

Yes, that answers my question. Thank you! I have read that documentation page, but I somehow couldn't bring that all together as the example on the bottom of the page was a bit confusing to me.

[dieppa]

Hi @XSpielinbox , thanks to you!! your questions and suggestions help us to make Mongock better 😃

Do you happen to have any suggestion(on the documentation, examples or whatever) that you think that it would have made easier for you.

PD: On top of the fact that we need to add the Springboot 3.x section to the documentation.

Thanks!!!

[XSpielinbox]

Hi @dieppa,
I'm glad to hear that. :)

Yeah, when thinking about it I would have many suggestions:

1. As long as Spingboot-v3 isn't really documented and only the Springdata v4 Driver is officially is supported, at least a small notice of that for Spring Boot 3 one should look at this other artifact and it only has very limited support would help dramatically.
2. Further, more details on the implementation in the Documentation would have been very helpful to me:
   a) Even small things like you just pointed out above with blocking with get() or await() as concrete examples to how to do this needed blocking. Then one has clarification/confirmation on what is meant with that, how one should do that and people not so familiar with the topic have at least a good starting point to look further.
   b) More details on ClientSession or an example on how to do it in a non-transaction way/help on how to work around transactions errors when using e.g. Flapdoodle Embedded Mongo for tests would help too.
   c) Explanation of when I need to define a MonoClient Bean, ConnectionDriver Bean or the like would also help. The example projects give a great insight into the multitude of possibilities, but a clear separation of what is just a matter of preference here or really needed for this concrete example is very difficult. This applies e.g. when explicitly setting values that actually are just defaults and therefore not necessary to explicitly set. For sure can everyone find out that

public MongoTransactionManager transactionManager(MongoTemplate mongoTemplate) {
    TransactionOptions transactionalOptions = TransactionOptions.builder()
            .readConcern(ReadConcern.MAJORITY)
            .readPreference(ReadPreference.primary())
            .writeConcern(WriteConcern.MAJORITY.withJournal(true))
           .build();
    return new MongoTransactionManager(mongoTemplate.getMongoDatabaseFactory(), transactionalOptions);
}

and

public ReactiveMongoTransactionManager transactionManager(ReactiveMongoDatabaseFactory mongoDatabaseFactory) {
    return new ReactiveMongoTransactionManager(mongoDatabaseFactory);
}

are functionally equivalent (except the reactive part, but just can be transformed between reactive and non-reactive be adding/removing the word "Reactive"), but it requires quite some digging. It doesn't always require code, but at least a small explicit comment to point that out would be helpful. It highly depends on the use-case after all. If I want/need to customize something it is invaluable to see the explicit code-example to see, how and what I can customize, but when just wanting to get started, it is extremely unsettling what would be the best way or what is needed for one's minimal use case at the beginning. Without this explanation it is unclear why this code is needed know (what it does) or whether it actually isn't needed. Another example of this would be the @EnableMongock annotation. I would assume from my testing that it sets up an MongockInitializingBeanRunner as this is more likely the right one/more useful one/whatever, but an explicit statement that it set's up this runner or that one or none at all would be nice.
d) Always having examples on the possible option (example to implement a MongockApplicationRunner vs. to implement a MongockInitializingBeanRunner vs. to use the annotation, example with and without transactions, etc.) would help with that tremendously of course, but would also be an extraordinary amount of work.
e) I couldn't find any documentation on/list of the options one can set in the applications.properties (or equivalent). If you actually have one, you should definitely present it way more, because the Javadoc/code complete in the IDE does more harm than good there as 4/5 options are highlighted as non-existent, but nevertheless have an effect and from many options I only out of chance get to know in some GitHub issue comment and for sure still don't know many of the great options.
f) On the note of transactions: the TransactionManager is still a mystery for me. When disabling transactions the use of ClientSession give's me errors that there is no such object or the like. For the TransactionManager this is different: adding it even when transactions are disabled works. Then again when having them enabled, I can run my code with and without it. I could find examples on the internet using transactions with and without defining this TransactionManager but also some sources stating, that one has to define one. What is the case now? Do I need it? Is it only needed when I actually use transactions and whether I set them to enabled doesn't matter? Does it get generated automatically in newer versions, but wasn't in older ones? Are the examples not defining it wrong? For sure one could say, that it being there shouldn't hurt and one can just add it as long as the application runs. But when then running into errors this is another of a at first seemingly endless possibilities of what could be wrong.
g) Related to the last point: a "common error" page might be very helpful to with that.
3. A Changelog/Release notes would be nice too that one does not have to try to infer it from the commits between the releases without knowing the codebase.
4. Support and Documentation on Java Modules and there use would be awesome too. I have a modularized application and using non-module dependencies is always a large amount of work to figure out how the automatic module names are, what possibly obscure dependencies/uses of reflection, etc. exist, one has to configure in the module-info.java, etc.

Above may look like Mongock is a bad library, but more of the opposite is true. It is the best for automatic MongoDB migrations in Java I know of and I now had one of the perhaps most awful usecases to integrate it in an application working sort of "against the system" with modules, Spring Boot 3 and reactive all at once...

I hope you can do something with above feedback/suggestions. If you have any further questions or want concrete examples/additions, just reach out to me. I would be glad to try to help my future self and others facing these problems.

[dieppa]

Hi @XSpielinbox

Thanks a lot for your contributions.

We'll address all those suggestions and then we'll forward them to you, as we would be very happy if you can give us your 👍🏼 (but it's totally optional, of course).

Anyway, as this is a important way of collaboration, we'd like to add you in our release notes as contributor(if you don't have anything against it), so you get your credit 😉

[XSpielinbox]

I would be very excited to see these improvements! The Mongock-examples project could also see some dependency updates.

If you want to name me as a contributor, you can for sure do that. 😄