Self-hosted Infra/Docker Simplification

[PrplHaz4]

Before you start (required)

Is this a bug?

No

Is this a feature request?

Yes

Is this related to Docker and/or hosting for self hosting?

Yes

---

Issue description

The current default infra/docker config is meant for running at scale in production. There are lots of opportunities for simplification, these would cut the number of containers we need to manage in half:

1. Change the default database to sqlite
2. Remove redis as a dependency
3. Update docker-compose.yml to reflect changes

Note: this would break existing installs without some significant migration work, but may be worth getting it in for the first release of a new product

[jjmata]

That's a really good idea, something that could be self-contained and easier to deploy. Love fig / docker-compose but to the average user it's a mess.

[jjmata]

A mono-user install could easily be just one single image where the rails + worker/redis processes run, sqlite as you mention ... I know the microservices Gods would strike upon me for even suggesting that - but would it be a reasonable "packaging" decision?

[jjmata]

The more I think about it, the more I like the idea. That slimmer packaging + devcontainer as runtime is good enough for me to run as a "local app" install. What am I missing?

[PrplHaz4]

The more I think about it, the more I like the idea. That slimmer packaging + devcontainer as runtime is good enough for me to run as a "local app" install. What am I missing?

Plaid as a local app will get ugly with linking and webhooks - but that's probably the only thing. And agree on the microservices Gods, but I value convenience over scale when selfhosting...

[jjmata]

Yeah, I'm right off-the-bat assuming Plaid and GoCardless are non-starters for local installs (since you need to have webhooks/https exposed services) but what about the n8n/my-LLM-cleans-my-data for me approach? I am not sure we can fit an n8n runtime in the same image or it will bloat, but extracting the workflow into API calls would be nice, see @gianniskotsas' workflow: https://n8n.io/workflows/6284-automate-multi-bank-transaction-sync-and-reporting-with-gocardless-and-maybe-finance/

[gianniskotsas]

Yeah, I'm right off-the-bat assuming Plaid and GoCardless are non-starters for local installs (since you need to have webhooks/https exposed services) but what about the n8n/my-LLM-cleans-my-data for me approach? I am not sure we can fit an n8n runtime in the same image or it will bloat, but extracting the workflow into API calls would be nice, see @gianniskotsas' workflow: https://n8n.io/workflows/6284-automate-multi-bank-transaction-sync-and-reporting-with-gocardless-and-maybe-finance/

That will work. Converting the workflow into api calls or code in general is the way to go. Also, I changed the workflow a bit because I ran in some inconsistencies mainly coming from the changing payment statuses. When you fetch the bank transactions from GoCardless, they are either marked as pending or booked depending on their actual status. however, a pending transaction can be later be converted into booked after the fetch request was triggered or even during the execution, so our scheduled trigger / cron job won't detect it. I think I have now a more clean approach that I can help make it more robust, but then again I am not Rails developer.

[jjmata]

Well, personally I would avoid specifying it in Ruby and try to make the LLM do the dirty work. Isn't that the promised land? End-to-end AI systems?

[PrplHaz4]

I have no idea what current AI capabilities of Sure are but n8n/llm workflow sounds to me like something that belongs in a sidecar similar to the SimpleFin bridge https://github.com/steveredden/simplefin_to_maybe

Unless the n8n "core" would be pluggable for different providers and have clear boundaries/interface...etc...

[jjmata]

100% percent with you! The n8n workflow is just a nice way to prototype possible implementations to add to native LLM flows (assuming the economics on # of tokens add up for this job to be done, or a local ollama-based LLM could be used!)

Really doubt the n8n "core" as you describe would work well for that, and their licensing makes me even less inclined to look into it.

[jjmata]

This idea could have many different implementations, so let's discuss first more openly before having it as an issue.

[antedebaas]

Just my 2 cents. But id refrain from using sqlite here. I get the direction and agree with it but i would not suggest sqlite for a product like this. At least add a solid database

[jjmata]

Don't want to start a vim/Emacs type religious war, but sqlite is quite solid and in use so many places you would be surprised. Having said that, the thought of DuckDB (the current darling of in-memory columnar databases) came to mind.

If stability is what we seek, I can't think of something much more stable than sqlite though.

[antedebaas]

No lets not make it that kind of discussion.
I know how well sqlite is being used, but personally i have a actual database server for storing my data.
but if its a changeable default im already happy :)