Directory /var/cloudstack not created (cloudstack 4.18.2.4 / Ubuntu)

[mwaag]

Hi,
iam trying to rotate our old management-server, based on outdated os and replace them with ubuntu-based management-hosts.

Here is what i do so far (using https://docs.cloudstack.apache.org/en/4.18.2.4/installguide/management-server/index.html)

• Prepare OS
• Configure Package Repo https://download.cloudstack.org for ubuntu jammy (because i still need 4.18 atm)
• Install packages cloudstack-management, -usage, -common (Version 4.18.2.4)
• copy db.properties of one of the other Servers and replace (instead of using cloudstack-setup-databases) the generic one
• execute cloudstack-setup-management

cloudstack-management then starts fine and the web-interface is reachable, but there are some errors in /var/log/cloudstack/management/management-server.log:

2025-06-13 11:17:54,634 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.7b38fd2
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.6650a7d
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.74a4a218
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e6a7e1a
2025-06-13 11:17:54,635 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.5e0936f5
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.3effc941
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.a93d98b
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.72ec9776
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.2cdd8c2d
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount: /var/cloudstack/mnt/345049466209.19f01fe3
2025-06-13 11:17:54,636 ERROR [o.a.c.s.NfsMountManager] (BackgroundTaskPollManager-6:ctx-74c87b2f) (logid:2752ce59) Unable to create mount point for nfs://172.27.249.130/bi_nfs_sec_01

When i look at the given path, there is no /var/cloudstack directory at all. Is this relevant for clean functionality?

We use a separate nfs-server for secondary Storage which is reachable and the export is mountable, just to be sure i can test it, but following the log messages there is a problem with creating the mount-directories not with reaching the nfs-export.

ping xxx.xxx.xxx.xxx
PING xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 56(84) bytes of data.
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=63 time=0.331 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=2 ttl=63 time=0.455 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=3 ttl=63 time=0.264 ms

mount -t nfs xxx.xxx.xxx.xxx:/bi_nfs_sec_01 /mnt/
ls /mnt/
2501-311-a4ff37ec-efdd-3ed5-b03a-927c705974d6  snapshots  systemvm  template  volumes

Checking on the other two hosts i see following directory-structure / permissions:

ls -lah /var/cloudstack/
total 4.0K
drwxr-xr-x.  4 root  root    35 Apr  4  2022 .
drwxr-xr-x. 20 root  root   285 Apr  4  2022 ..
drwxrwx---.  4 cloud cloud   38 Oct 10  2024 management
drwxrwx---. 25 root  cloud 4.0K Jun 10 15:00 mnt

The errors disappear when i manually create these directories as well but i expect the package cloudstack-management to create those dirs.
Checking the package on the old hosts indicates this:

rpm -ql cloudstack-management | grep var
/var/cache/cloudstack/management
/var/cloudstack/management
/var/cloudstack/mnt
/var/log/cloudstack/ipallocator
/var/log/cloudstack/management
/var/run/cloudstack-management.pid

The cloudstack-java-process is running as user 'cloud' which is expected, because it's the same on the old management-hosts.
Additionally i cannot remember needing to create those directories manually in the past.

What could be wrong here?

Thanks in advance
Markus

[mwaag]

Ok, i think i found the culprit already. I think its because of the global setting mount.parent, which is /mnt/cloudstack/mnt but the new package brings /var/lib/cloudstack/mnt. /var/lib/cloudstack is succesfully created by cloudstack-packages on the new server.

I assume it would be preferred to use the new path? Will it work to change the path in global settings of the existing installation and restart the management-servers to apply? I may have to make the new directory first on the old servers.

[DaanHoogland]

@mwaag , as you put it now it still seems like a bug in the upgrade procedure. Would be nice to fix it for future potential victims. Do you still have an old DB dump to find out whether you had changed it before?

[mwaag]

@DaanHoogland
Not from that particular database, but i have very old ones from an old productive instance.
It seems your default never has changed but we edited it at some point. Attached a screenshot of an old 4.11-database from around 2022 compared to a fresh 4.18.2.4-database from 2025.

Plus this beeing the special case where one want to add a freshly installed management-host to existing ones (which may have lived a long time already). As i understand, the directory /var/cloudstack will be created at installation time, there is no knowledge about the information from a potential existing database at this stage.

Ofc it would be nice, if cloudstack fix it by itself through creating the customized path (after a service-restart for example).

[DaanHoogland]

would that mean that there is an exception handler that can see that the mount point is not available, check the configuration to see if it should and then do the java equivalent of mkdir -p <the configuration value> . The downside of this is that the operator can make a mistake and cloudstack will forgive them instead of pointing them to their error. Is there maybe a way to make the "Unable to create mount:” message more clear instead?

[DaanHoogland]

@mwaag , do you want to create an issue for this or shall we close?

[mwaag]

No. Pointing me to the misconfiguration was fine for me at the end. You can close.
Thanks!