Multi-tenant

[pelletier197]

I have a feature request for mongock that could be interesting. I see mongock is able to scan packages looking for changesets and automatically instantiate classes, which is good. However, in some use-cases, changesets could be manually instantiated, and in my use-case I would need to provide those instances directly to mongock, instead of letting the framework instantiate them.

Something like this:

val changeSet = MyChangeSet("custom instantiation parameters")

MongockStandalone.builder()
    .setDriver(MongoSync4Driver.withDefaultLock(client, "db"))
    .addChangeSet(changeSet)
    .buildRunner()
    .execute()

I haven't looked deep into the architecture, but I believe this would be quite simple to implement. Let me know what you think!

[dieppa]

Hello @pelletier197 ,
first at all thanks for your contribution and for using Mongock.

That's a good idea. We actually have it in our roadmap for the coming version 5(we are not adding any new feature to current version).

We are actually going further in Spring, we will provide the ability to load the changeLogs from the Spring context.

However, this has to be approached with caution. All the parameters Mongock pass to. a changeSet are proxies(unless the developers specifies the opposite) to ensure that the lock is wrapping the i/o operations to the database, at least the writes. If you construct your changeLog on your own, this is going to be a problem, as a lot of developers won't be aware of this fact.

Can you tell me why do you really need this?

[pelletier197]

Thanks for your quick reply, and thanks to you for providing such a great framework to the community!

All the parameters Mongock pass to. a changeSet are proxies(unless the developers specifies the opposite) to ensure that the lock is wrapping the i/o operations to the database, at least the writes.

I unserstand perfectly your concern. However, this arises a small question : why do you put a lock on every parameter of the changelog? Wouldn't it be simpler to wrap the call to the changeset methods in a transaction and revert if any of the changelog fail? Sorry for that question out of no-where, but it feels a little funny the whole proxy thing on every parameter, especially since this doesn't fit well with kotlin where everything is final by default.

Can you tell me why do you really need this?

Yes, so I have a special use case for Mongo to support multi-tenancy, meaning each customer uses the same instance of Mongo, but will see his data stored in a completely different database. This is not well supported by Mongo either (we use spring), so we had to develop a little custom code to make this work.

However, this means I have to run my migrations on multiple databases, which does not really fit into the use case for spring (or maybe there's a magical way to do this I don't know of) . So instead, I though of having an Interface MongoChsngeSet, and annotate the implementations with@Configuration so I could input the changesets manually into Mongock standalone, while still letting spring do the wiring/scanning part, but I realized Mongock didn't allow that.

Si here I am. Maybe there's a better alternative I didn't think of? Thanks for your help!

[dieppa]

However, this arises a small question : why do you put a lock on every parameter of the changelog? Wouldn't it be simpler to wrap the call to the changeset methods in a transaction and revert if any of the changelog fail? Sorry for that question out of no-where, but it feels a little funny the whole proxy thing on every parameter, especially since this doesn't fit well with kotlin where everything is final by default.

That would be a much easier approach. Mongock acquires the lock before the process starts, lets say for 3 minutes, and before every changeLog runs, it makes sure the lock is acquired for 3 minutes, right? What happens if a changeLog takes 4 minutes? The changeLog will be partially executed without lock. A solution it's to reserve the lock longer, lets say 5 minutes. Now, if another changeLog takes 6 minutes?...well, put something safe, an hour, as we know that mongock will release the lock once the process is finished, it won't probably take the hour ever. Well, that's ok, but what happens if the jvm is interrupted and Mongock cannot release the lock. The lock will be "unreleaseable" for an hour, which in most of cases will mean that the service running Mongock on that database won't be able to start. That's a unacceptable risk to take in production, specially in containerised environments 😉

I guess you get my point, basically that's not a robust approach. Instead, we have provided a more sophisticated mechanism hat guarantees the process is fully synchronised, based on a pessimistic lock which ensures every access(at least every write) is synchronised. Performance-wise doesn't mean a big impact as it's keep it in local memory and hits the database just when is required to refresh the lock.

You may suggest using demon, but that's not that reliable and threads my not fit all the ecosystems.

Regarding your scenario, I have a couple of questions

1. Do you know which databases you will be working with beforehand?
2. When do you run the Mongock process? At starting(standard) or in the middle of the application, triggered by something(non standard)

As part of version 5 we are addressing this scenario, as it's starting to be a common one, specially for consultancies. That's what I made this questions, I'd like to understand your needs so we can provide the best solution :)

[pelletier197]

Yeah, I understand the problems with the lock. This is actually the first system I see that has a lock with a duration. Usually, I believe they put the lock in the database at the beginning of the process and remove it at the end (either the changelog crashes or not). This is an interesting approach, but i honestly don't see the advantages of it right away. You surely designed it this way for a reason, so I trust your judgment 😉

Anyway here's the answer to both questions above :

1. We have a different set of credentials for every tenant, and every set of credential has access to only a specific database. It's a standard for highly secure applications (not for most applications I think), so the database is known beforehand, but not hard-coded in a file (if that's your question). We fetch the credentials and the database names for each tenant at startup from an external service. We also have a different MongoClient per tenant, each using a specific credential set.
2. We run the migrations at startup and they must all complete before the application finishes booting. We usually only add indexes, so they are instant most of the time. We haven't used Mongo long enough to need to change a large dataset.

Hope this helps 🙂

[dieppa]

I see your point. This is something that doesn't really shines and may not be needed in most of the cases for you, but it's a must for a robust system, specially in containerised environments where automation is key. Actually the reason we started this project was because we experienced a big issue, that happened sometimes, but when it happened, was terrorific. The service couldn't get recovered and we had to remove the lock collection and deploy it manually again. It was reeeeally hard to find.

It may look a bit over-engineered, but we are obsessed with robustness, so the tool gives the less problems possible and you can focus on your business.

Coming back to you scenario.
Apart from the fact that you are managing multiple databases, you are doing it as it's supposed to be done. That's good!! Let me think about your case and will come back to discuss the best solution for you, ideally using the spring module 😉
But this will be provided in the version 5, which will be released around end of May, but we can provide you with a beta version

Just out of curiosity? Which company are you? If you can't give the name, could you at least provides the industry?

Cheers

[dieppa]

OK. and I guess you then create different Mongock instances passing the MongoClient and the tenant-db, is that right?

Have you considered creating a MongoClient per tenant?

[pelletier197]

Yeah we do have a MongoClient per tenant 😉 sorry If I didn't explain that well. I said the code I sent you is similar, but we have multiple MongoClient.

We do as you said above. We execute Mongock for each tenant, passing its MongoClient and the tenant's database. 😉

[salerokada]

@dieppa Did you added multi tenancy support in the meantime?
Or @pelletier197, did you managed to resolve multi tenant configuration for Mongock?

[pelletier197]

Yeah, we figured something out. I don't remember exactly what we did since it was a long time ago, but I believe I did not use package scanning and I instantiate and run migrations manually by creating a different migration runner for each tenant.

Sorry I can't provide more details, I'm not on this project anymore.

[dieppa]

@dieppa Did you added multi tenancy support in the meantime? Or @pelletier197, did you managed to resolve multi tenant configuration for Mongock?

We added multienancy to the professional distribution. However, the version 6 will bring multiple features and suprises, which can affect this and make it more accessible