Azure Deployment

[aderici]

Hi,

Starting this thread to help Azure people like me to deploy a simple self hosted Budibase into Azure App Services.

App services is the platform as a service offering of MS that is the go-to service if you simply want to run your app and not deal with anything else. It supports running codes directly (like a node.js app from github) or it can run a single docker container as well as simple docker-compose files (with a lot of limitations which makes running budibase impossible)

Another interesting limitation is in appservice if we want to use a custom docker image wa have to write our important data into /home folder otherwise azure wont persist any other files.

If we want to use direct ssh into docker feature we have to install openssh and provide a static user pass (app runs behind reverse proxy and ssh port is never exposed to public)

Given all info and limitations, i pursued the single docker path. Thankfully the single docker folder under hosting in the repo was a helpful start but definitely not working out of the box. After a many sleepless nights i was able to get it to work.

One thing i couldnt understand yet is if i build and run the app and worker from the repo, app doesnt work and complain "global db not found" i had to grab official app docker and copy the already compiled app directory into my single docker. Any idea why that might be happening?

2 of my clients have been using single docker budibase on azure happily ever since. Upgrades will be painful though.

I'd love to help single docker efforts to provide something usable for the community, please guide me how,

[mike12345567]

Hi @aderici - the single docker image is on our roadmap for official support soon, right now it was just put together as an experiment to see how it would work and what it might look like, but isn't officially released yet!

In terms of the build failing, I'm not sure what branch of Budibase you're using to build with, but master is the branch you should be using, there may be issues with the develop branch as it is still in development - we have it as the default branch so that external contributions will be added to it so we can test them before release.

As I say the single docker image is something that we want to support soon, we're still working out the details of documenting it, everything we want to expose via environment variables and putting together some testing for it, once this is done we'll have it released.

[PekkaJalonen]

@aderici I'm also looking an option to run on appservice the single container, can you give some high level insights on what type of customization did you need for the build to get it working?

[aderici]

@PekkaJalonen i pretty much had to go through every file on hosting/single folder on this repo and make changes. Nginx config, dockerfile, the text indexing plugin config file, startup script and others. 2 main issues i had to solve, one if them is persisting files into /home folder and other is just getting it to work. Also installed openssh to support app service ssh. I hope some day we will have a standard single docker supported by budibase which also fulfills azure appservice requirements,

[jonnymccullagh]

Hi @aderici, we have added a step in our build process that creates a single image with some Azure App Service customizations. It is tagged as budibase-aas if you want to try it out and give us some feedback:

docker pull registry.hub.docker.com/budibase/budibase-aas

[aderici]

Hey thats great to hear! Really happy to see continuous improvement here! I'll try it out and will let you know

[PekkaJalonen]

It would be nice to get some configuration info also. I have set minio, couch, redis, jwt, internal api key related env variables and tested mounting /data to file storage and also without mount using the native persistent storage, and I get nginx error 502 Bad Gateway, so no luck. Tested with v1.2.2

[PekkaJalonen]

Also seems that SSH is not working, so either it has configuration issue or the container is down. Last log couchdb cluster stable.

[PekkaJalonen]

Tested the budibase-aas image on Azure App Service. Could not get the container working when deployed using Single container, only when using Docker Compose to deploy single container which is in preview in Azure.

From functionality perspective, App Service SSH is not connecting and also, minio works only when not using persistent storage. Each time trying to mount /data to Azure File Storage or using the App Service native storage(home folder), I start getting errors from minio about file access denied. I can see couch, minio and .env generated, but minio not storing anything.

[jonnymccullagh]

Hi guys, I have merged pull requests to help support running Budibase in Azure App Service. This included the SSH service on port 2222, /home for data and some manipulation of environment variables as AAS already set a variable named PORT.
Install instructions are here:
https://docs.budibase.com/docs/azure-app-service
Note the docker image address of budibase/budibase-aas
We would appreciate any testing or feedback you might have.

@aderici @PekkaJalonen

[aderici]

Hi, @jonnymccullagh just started using it for one of the projects a few days ago. all seems to be working just fine other than SSH. See #7745. I just checked docker hub and the latest image seems from 2 days ago so i assume your merge didnt automatically generate new images in docker hub?

[jonnymccullagh]

Good point. Triggered it now.

[aderici]

@jonnymccullagh just switched to the latest and seems to be working fine including the ssh! Thank you!

[PekkaJalonen]

budibase-aas release works fine when running on Azure App Service in non-persistent mode, but when enabling data persistence by setting WEBSITES_ENABLE_APP_SERVICE_STORAGE=true, minio fails to store files due to fact that App Service used file storage does not comply well with applications which require file handles/locks as described in https://learn.microsoft.com/en-us/azure/app-service/configure-connect-to-azure-storage?pivots=container-linux&tabs=portal "It's not recommended to use storage mounts for local databases (such as SQLite) or for any other applications and components that rely on file handles and locks."

But when running on example Azure Container Apps and mounting file storage, minio works fine. So most likely this is App Service limitation which you cannot fix as minio has no configuration of disabling such locks. I recommend using Container Apps or changing budibase to use additional options to replace minio with example https://github.com/gaul/s3proxy which would allow defining different S3 targets including Azure Blob Storage.

[PekkaJalonen]

Hi, I started another discussion around the Azure before I realized that this was here already, so I'm just linking my finding here: #8040

[PekkaJalonen]

@aderici just to check, you mention couch folder and permissions, but the issue is with minio. Can you check minio folder permission?

[jonnymccullagh]

@aderici my Azure UI looks very different from your screenshot above. I recorded a video of what I did here: https://www.youtube.com/watch?v=wgOU6lgeUvU (apologies for the quality).
When I ran the container in the above video I was able to upload a PDF file and viewed it on the filesystem via SSH. However, after restarting the container all the content was gone. I am interested in how @aderici is using AAS if differently from me. And interested in how @PekkaJalonen is getting minio errors if he is following the same process as in the video.

[aderici]

@jonnymccullagh the UI is actually the same. my screenshot is taken after deploying the web app, browsing to the deployment center on the left hand menu. One thing i noticed is that the owner of the folders look different.

@PekkaJalonen sorry i said couch but i meant minio.

Here's what my ls-al on prod budi app assets looks like :

I'll record a loom video deploying an appservice from scratch today

[PekkaJalonen]

@jonnymccullagh did you set the WEBSITES_ENABLE_APP_SERVICE_STORAGE to true on App Service configuration? My step were the same with addition to set this mentioned setting to true which was false by default. And I also noticed you used ssh to connect to budibase. This does not tell you if the /home folder is persistent. I can do the exact same steps as you did as the service uses temporary volumes when the previously mentioned setting is false. This is why you also loose your data. When setting it to true, /home is mounted in App Service used storage which will persist you data. Here is when the minio gets broken for me.

[jonnymccullagh]

No I didn't set WEBSITES_ENABLE_APP_SERVICE_STORAGE to true. At what point are you setting that? And I will try the same.

[aderici]

@jonnymccullagh @PekkaJalonen after some research i have good and bad news.

It seems that minio recently introduced some changes, which determines how the files are stored in the disk. Previously if you have a single server your files would be stored as is, meaning if you upload a.jpg you'd see an a.jpg in your disk. But now, if you save a.jpg you see a folder called a.jpg in your disk and within that folder there are files that specify the content.

This behavior is determined by the content of /path-to-minio-data/.minio.sys/format.json. See how in an earlier successful deployment, "format" property is "fs".

{"version":"1","format":"fs","id":".....","fs":{"version":"2"}}

If you have previously deployed a budibase and have been upgrading, minio will see your existing folder / format.json and will continue with the fs format and that will work fine with azure. But it wont allow you to start a new instance with fs format anymore

See how the new format.json looks on a recent deployment, format is now "xl-single"

{"version":"1","format":"xl-single","id":".....,"xl":{"version":"3","this":"....","sets":[["....."]],"distributionAlgo":"SIPMOD+PARITY"}}

Minio won't allow you to manually change a new deployments format, it will throw an error. It will only allow you to migrate an old folder with fs format. It is their way of fighting agaist people using fs mode i guess

I have been building my custom images earlier, when we didnt have budibase-aas standard image. When i checked still have an old image that works out of the box and the minio version on that one is "RELEASE.2022-05-23T18-45-11Z"

@jonnymccullagh would it be possible to use this specific version of minio i mentioned in azure images?

Are you guys using minio for budibase cloud? Or are you using s3 and if we somehow have an s3 compatible proxy for azure, can we ditch minio completely by setting some environment variables?

[PekkaJalonen]

Thank you for solving the root cause. This seems to be limitation with App Service then, because I can use Azure Container App successfully with Minio when mounting to Azure File Storage share.

I also tested the older version with Minio gateway to blob storage which works well with Azure Blob Storage in App Service. Bad news is that Minio has dropped the support and futher development of the gateways, so it is not sustainable solution.

[PekkaJalonen]

@aderici I started think about the permissions of the folders you mentioned earlier that your ls -al looks different and the fact that Minio works with ACA (so it cannot be Azure File Storage limitation). Could it be also that Minio changed permissions somehow for the new deployments. I think that we could try to test permission fix and see if that helps. It could be then fixed in budibase if that might be the issue.

[aderici]

Unfortunately this seems to be a filesystem issue as you previously highlighted. I've tried chmod 777 or chown but no luck.

[PekkaJalonen]

Yes, App Service mounts with owner nobody / nogroup which cannot be changed. In ACA mount is owner root / root which seems to be the reason why it works there.

[aderici]

@PekkaJalonen there is a discussion about file locking which is very similar to the error messages that i see. But this is an old message and we have been using it happily on fs mode. minio/minio#9212

[jonnymccullagh]

@PekkaJalonen @aderici I have been playing with Azure Container Instances as an alternative to AAS. I created a video here: https://www.youtube.com/watch?v=40EzlgftPo0
Not sure that this totally ticks all the boxes for you but it does appear to persist the storage and minio uploads working fine.
I'll try to also get a look over ACA as I get time.

[aderici]

Thank you @jonnymccullagh nice demonstration, happy to see at least another path exists. @PekkaJalonen i see that mounting a storage into appservice is possible (into a custom local folder, not /home) and when doing so, you can either mount a blob storage or file storage. Shouldnt "file storage" work fine with appservice and minio as it is exactly the same type of storage that you use for container instances? note that if you mount a "blob" type of storage its not going to work because blob storages are read only when mounted to appservice

[PekkaJalonen]

Unfortunately this does not help. I tested it with single container as well compose and having some custom folder path with budibase/budibase-aas release as well with the budibase/budibase mounting to /data. Both gives the same issue. App Service still mounts with user/group = nobody/nogroup as where when using ACA the mount is root/root which I assume is the reason for the problem.

[PresidentCamach0]

Hi, is there any more updates to this thread?

We want to deploy on Azure Web Apps.
When we set WEBSITES_ENABLE_APP_SERVICE_STORAGE to TRUE, BudiBase works for about a day or two. Once the service restarts, it keeps the logins and apps. When you try to open an app it shows the error: Error initialising app: Not Found

Is this still an issue with Azure Web Apps or the version of CouchDB being incompatible?

[PresidentCamach0]

Hi, is there any more updates to this thread?

We want to deploy on Azure Web Apps.
When we set WEBSITES_ENABLE_APP_SERVICE_STORAGE to TRUE, BudiBase works for about a day or two. Once the service restarts, it keeps the logins and apps. When you try to open an app it shows the error: Error initialising app: Not Found

Is this still an issue with Azure Web Apps or the version of CouchDB being incompatible?

[mike12345567]

Hi @OriginalLimited - this sounds like a problem with the MinIO storage being wiped out after the service restarts. We encountered issues with this when using Azure Web Apps, sadly we were unable to find a good solution to this.

[jonnymccullagh]

Azure offer approx 3 container services each with different pros & cons. I think the best option is Azure Container Instances for a persistent storage volume.

[PresidentCamach0]

@jonnymccullagh Thank you, and thanks for your YouTube guides on setting this up. The only thing with the container instances is the additional work behind setting up SSL and custom domains - which, out of the box on Azure Web Apps is very simple with a couple of clicks.

In the end, we just spun up an Ubuntu VM and installed Budibase on there - works much better and easier to setup SSL/Custom Domains.

[PresidentCamach0]

@mike12345567 Thanks! That's a shame!

[aderici]

I remember being able to run budibase on azure appservice. Its been more than a year though. Just be careful where individual services save their files. When you enable persistent storage only home folder is persisted. So you have to make sure all important files are saved underneath /home/ path

[PresidentCamach0]

Re-visiting this thread, it appears recently that the budibase/budibase-aas container has been updated for MinIO for it to run using persistent storage using Azure Blob.

I can confirm it works, however the setup information is not correct on the Budibase setup guides here: https://docs.budibase.com/docs/azure-app-service

Currently, it only works using the experimental tag, and at version 3.4.13. Using stable does not work.

But otherwise, so far so good. It loads fine even after the app service is restarted/stopped.

[ff-lkubin]

Hi. Fairly new to this platform so I apologize if I am missing something simple here. Deploying via Azure App Services as well, the application deploys but once inside budibase deploying / importing apps returns "Unable to write to object store bucket." I am misinterpreting something in the instructions here?

[PresidentCamach0]

The deployment instructions on the BudiBase website do not work. Currently, it only works using the experimental tag, and at version 3.4.13.

I've had bad test results using Azure Web Apps.
It might keep the data persistently when the web service restarts, but I often find that the BB server does not start and you get a bunch of errors in the log stream. I would only use it in development and not production.