Incorrect instructions in celery tutorial

[jdavcs]

The "Configuring Galaxy to use Postgres" section in the Alternative Celery Deployment for Galaxy tutorial ) is incorrect and may lead to Galaxy not starting up.

See galaxyproject/galaxy#20809. I'll make the edits once/if we determine we are not making an exception for Celery tables (as per issue description).

Ping @hexylena (FYI)

[jdavcs]

@hexylena after looking at the tutorial, I think we should drop that tutorial. Given that we can't recommend using the Galaxy database as a result backend for Celery (as per discussion in #20809), the tutorial would have to instruct the user to setup a separate database, which, I think, would not "significantly simplify operational complexity", as per tutorial's key points. Would that be OK?

[hexylena]

It isn't a 'mainline' tutorial to start with, but I'll confess I feel a bit disappointed with the result.

I understand it's preferable to y'all to make such a strict assumption on an empty database.

1. Is there a rationale for galaxy not just ignoring tables it doesn't know anything about?
2. are there any other options? Where e.g. galaxy does the configuring, so galaxy "knows" about those tables, and we can still use galaxy's DB for that (which I think really is a sensible default, vs mandating redis.)

because I conclude from your alternative of "another database" that's the alternative to "setup redis"

[jdavcs]

Here's the relevant part of the linked issue description:

On every startup Galaxy tries to intelligently determine the state of the database and act accordingly. (here's a helpful diagram). This logic can be traced here. Long story short, if the database specified in the database_connection in your config exists, Galaxy checks what's inside. If it finds tables that are not known exceptions (kombu and sqlite), it will consider the database initialized and will look for the alembic_version table, which in this case does not exist. Not finding it, Galaxy will exit.

There are other reasons for not adding additional tables to the galaxy database; we've discussed them when moving from sqlalchemy migrate to alembic.

[jdavcs]

To follow-up, Galaxy makes this strict assumption on an empty database because it can't tell the difference between an empty database and a non-empty database that has not been properly initialized (due to whatever reason). We can't go over the entire schema to make sure it's what's in the model (in a reasonable way), so we make the assumption that if there is a table, it's been initialized - so we check its version, etc, etc. We can keep the tutorial if it does not recommend reusing the existing database.

[hexylena]

Here's the relevant part of the linked issue description:

yes, I read that before today. And saw the state diagram, it's a nice flow chart.

I understand the current behaviour of the system, what I was (and continue to be) missing is the why. That quoted text wasn't a reason, that was just a description of the system as it it functions today.

Is it because you might want to generate a table with that name? That'd be a reasonable rationale I suppose?

But since this is also a core feature, more and more basic functionality is locked behind having a task queue (sure, fair! I support it!), that's why I brought up point 2, I wanted to push towards a discussion about what can we do to make this required feature easier to use?

Speaking as an admin, it's easier to use if I don't have to setup another service, I can just use the same database that's already being used for the rest of galaxy, to store these galaxy tasks. Conceptually this is "all galaxy's data", as far as I'm concerned. Do you as a developer see it differently?

The conclusions as I see them:

1. Can galaxy change to permit this core feature of galaxy to use it's own database?
2. Is another database strictly required? Are you, speaking for the developers, saying it is simply too complex to support (that's an absolutely acceptable answer, I just would like to know.)
3. Or am I obligated to setup another service I need to monitor? (I will confess I'm

[hexylena]

Ah we post at the same time, I did not see it before sending. Thank you for the additional clarification.

We can't go over the entire schema to make sure it's what's in the model (in a reasonable way

this was useful to understand, a technical limitation.

That said, I still wonder if an exception can't be made for celery tables.

[jdavcs]

Can it be done? Sure. The reason is that it is not good design; and I mean not good as in brittle. We can agree that a table named "celery_backend" (or whatever) is an exception - that creates a special case. Such special cases make it more difficult to reason about the data model and what's going on in the database, which leads to more time spent on comprehending the code base for dev or debugging, higher likelyhood of bugs, etc. Edge cases that break things in new ways will pop up - the referenced issue took weeks to debug (in that particular case, those tables were created in a blank database before the model was initialized, which led to errors that further obscured the root cause of the problem). This is a little far-fetched, but this would be similar to adding ad-hoc python files to the lib directory to augment the code base with logic that doesn't belong to core, but is useful in some cases and doesn't get in the way in others - we'd never do that. Likewise, in my opinion, we really shouldn't treat the db as a catch-all storage for stuff we may need to persist.

[hexylena]

Thanks for the explanation and hypotheticals @jdavcs, that makes sense. Then let's disable this tutorial for now. I'll reconsider the options. Adding a second database is probably going to be the easiest ansible option (though I can imagine it'll be annoying for some admins who have more tenuous relations with their DBAs.)

Likewise, in my opinion, we really shouldn't treat the db as a catch-all storage for stuff we may need to persist.

sure, that's fair. one could argue that I'm going too far in the direction of "use postgres for everything", but here it really reduces my cognitive load which is important. Using the same DB means my existing backups are sufficient, that I don't need to worry about any additional folders to backup, don't need to monitor services and ensure their access credentials are ok. It would really simplify things for end users, at the cost of devs' energy.

[jdavcs]

Yes, it's a trade off, to be sure! Thank you for seeing this from both perspectives - I do very much appreciate it!. And, FWIW, I do think that removing/disabling a tutorial for whatever reason is always unfortunate: I realize how much thought and effort is put into each one of those, no matter whether the tutorial is major or supplementory).

[jdavcs]

@hexylena FYI, ongoing discussion here galaxyproject/galaxy#20809