Has anyone played with SeaweedFS and Talos? Other small-scale storage solutions?

[alexpdp7]

My use case is two single-node Talos clusters. I know single-node k8s does not make a ton of sense, but this is personal infra. I just use k8s as a small PaaS to deploy some applications. Also for learning purposes and because increasingly, more software is easier to deploy on k8s.

For now, I've been running a storageless cluster, which has been fine for my purposes. However, I want to run a piece of software that supports object storage (Takahe). So I was looking at setting up Minio or some other "easy" object storage outside the cluster.

But SeaweedFS caught my attention. I got a POC very quickly (single command), and I read some encouraging articles. It apparently can do object storage and provide PVs to Kubernetes clusters.

Anyone using this with Talos? I would run it alongside my Talos clusters, in Proxmox. Talos is a VM, while I would run this as an LXC container.

[bitsondatadev]

@alexpdp7 did you ever try this? I am looking at this now with a similar use case.

[bitsondatadev]

That's great! The way you are running SeaweedFS services independent of Talos is exactly how I would want to do it though I am tempted to give the seaweedfs-operator a shot. I likely wouldn't want the SeaweedFS services running in k8s to depend on itself as the filesystem. I'd rather them just use local since it avoids the circular dependency.

I am setting up a homelab that can run some services for both personal workloads and scale up for business uses over time. This service needs to support both block and object storage patterns. My main concern is to scale/manage services and storage independently (like any cloud provider) but do it with various sizes of workloads, most likely smaller to begin with which makes Ceph less desirable. The key for me is flexibility of use cases over dominance over the larger use cases.

Thanks for sharing your COPRs. I have very little experience with Fedora/RHEL distros. If I don't end up using the kubernetes operator, do you suggest Fedora for this use case or are you just familiar with it?

Thanks!

[alexpdp7]

Thanks for sharing your COPRs. I have very little experience with Fedora/RHEL distros. If I don't end up using the kubernetes operator, do you suggest Fedora for this use case or are you just familiar with it?

Oh, the COPR only provides packages for RHEL 9 and clones. Fedora is great, but I prefer to run distributions with longer support cycles for my infrastructure.

[bitsondatadev]

Ah, is RHEL pretty accessible to a Debian-based user?

I used it for a couple years in industry and there were just so many nuances I wasn't used to since I never used it outside of work. Which taxonomies of use cases classes you recommend RHEL and/or enterprise support for?

1. Personal homelab setup
   1. high experimentation
   2. low security
   3. low-high latency/bandwidth
   4. random arbitrary downtime expected
2. Personal homelab servicing friends and family
   1. med-low experimentation
   2. med-high security with many family members data
   3. low-high latency/bandwidth
   4. occasional planned downtime with higher risk of unplanned downtime
3. Small business for friends/family/customer data
   1. low experimentation (likely some testing environment and cicd)
4. high security
5. med-high latency/bandwidth
6. occasional planned downtime with lower risk of unplanned downtime

My hope is that with new automation frameworks and decentralized and modular services, you enable smaller businesses to handle most of their own enterprise use cases or male it possible for more distributed community based support.

I do still think the most solid way is centralized support but I think that trend is to customize and localize pain points where more complexity is needed. In this case, grenade launchers like RHEL might be overkill if you have a small business, but always depends on specific I guess.

[alexpdp7]

I suspect this is veering off-topic; I'll answer briefly, feel free to email me if you want further discussion.

I use RHEL clones out of inertia and minor factors, but I'd always default to LTS distros (Ubuntu LTS, Debian Stable, RHEL clones, etc.) because on rolling distros, any update could require manual intervention to update configuration. With an LTS distro, I know that if I configure a system, it should not require configuration updates for a couple of years. I do not use paid enterprise support.

[bitsondatadev]

Thanks Alex that was quite helpful and mostly what I needed! Thanks for offering to chat offline!

I'll report back once I have Talks and SeaweedFS set up.

I think I may give that k8s operator a go on Talos a go.

Edit: I'm running RPi5 and ran into this issue: #7978. I'm gonna use k3s for now but will come back to talos once rpi5 is ready.

[saolof]

You may also be interested in garage if your main usecase is ease of self hosting. Due to not having or supporting anything that can lock up it needs less orchestration and you can get away with just using a statefulset.