HTTP ERROR 503 Service Unavailable #9880
-
ISSUE TYPE
COMPONENT NAMECLOUDSTACK VERSIONCONFIGURATIONOS / ENVIRONMENTDebian 12.7 Stable SUMMARYSTEPS TO REPRODUCEEXPECTED RESULTSACTUAL RESULTS |
Beta Was this translation helpful? Give feedback.
Answered by
penguin2716
Nov 6, 2024
Replies: 24 comments 29 replies
-
|
can you search |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@gobiursus can you re-do all steps and share the output ? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
How do I check the connection? How do I fix?
…On October 18, 2024 8:35:27 AM UTC, Wei Zhou ***@***.***> wrote:
@gobiursus
from my experience, it is mostly caused by db misconfiguration.
in short, the mgmt server cannot connect to the db
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
There was no error during installation. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I captured some extra stuff, I am not sure if they are related: |
Beta Was this translation helpful? Give feedback.
0 replies
-
Logs look ok |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I loaded the full log earlier in the link.
…On October 18, 2024 7:58:39 PM UTC, Wei Zhou ***@***.***> wrote:
> I captured some extra stuff, I am not sure if they are related:
> ```
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; preset: enabled)
> Active: active (running) since Fri 2024-10-18 14:33:44 EDT; 1s ago
> Main PID: 20191 (java)
> Tasks: 30 (limit: 9471)
> Memory: 320.8M
> CPU: 1.919s
> CGroup: /system.slice/cloudstack-management.service
> └─20191 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMem>
>
> Oct 18 14:33:44 manage systemd[1]: Started cloudstack-management.service - CloudStack Management Server.
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Initializing server daemon on null, with http.enable=true, http.port=8080, https.enable=false, https.port=8443, context.path=/>
> Oct 18 14:33:45 manage java[20191]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> Oct 18 14:33:45 manage java[20191]: SLF4J: Defaulting to no-operation (NOP) logger implementation
> Oct 18 14:33:45 manage java[20191]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details
> ```
Logs look ok
Can you load the full management-server.log ?
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
<https://pastebin.com/R3CT2v5Y>
There you go
…On October 18, 2024 7:58:39 PM UTC, Wei Zhou ***@***.***> wrote:
> I captured some extra stuff, I am not sure if they are related:
> ```
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; preset: enabled)
> Active: active (running) since Fri 2024-10-18 14:33:44 EDT; 1s ago
> Main PID: 20191 (java)
> Tasks: 30 (limit: 9471)
> Memory: 320.8M
> CPU: 1.919s
> CGroup: /system.slice/cloudstack-management.service
> └─20191 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMem>
>
> Oct 18 14:33:44 manage systemd[1]: Started cloudstack-management.service - CloudStack Management Server.
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Initializing server daemon on null, with http.enable=true, http.port=8080, https.enable=false, https.port=8443, context.path=/>
> Oct 18 14:33:45 manage java[20191]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> Oct 18 14:33:45 manage java[20191]: SLF4J: Defaulting to no-operation (NOP) logger implementation
> Oct 18 14:33:45 manage java[20191]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details
> ```
Logs look ok
Can you load the full management-server.log ?
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I have the same issue after upgrading from 4.19.0.0 to 4.19.1.2 on Ubuntu 22.04. MySQL and the management server are running on the same node. After the following steps, the management console works now (although it may not be a good solution since the password is saved in plaintext). The problem may be related to somewhere around encryption / decryption logic?
--- db.properties.bak 2024-10-25 08:51:07.179157143 +0900
+++ db.properties 2024-10-25 08:50:10.391852007 +0900
@@ -22,7 +22,7 @@
# CloudStack database settings
db.cloud.username=cloud
-db.cloud.password=ENC(<encrypted_text>)
+db.cloud.password=plain_password_text
db.cloud.host=localhost
db.cloud.driver=jdbc:mysql
db.cloud.port=3306
@@ -53,7 +53,7 @@
# usage database settings
db.usage.username=cloud
-db.usage.password=ENC(<encrypted_text>)
+db.usage.password=plain_password_text
db.usage.host=localhost
db.usage.driver=jdbc:mysql
db.usage.port=3306
@@ -100,4 +100,4 @@
db.usage.secondsBeforeRetrySource=3600
db.usage.queriesBeforeRetrySource=5000
db.usage.initialTimeout=3600
-db.cloud.encryptor.version=V2
+#db.cloud.encryptor.version=V2 |
Beta Was this translation helpful? Give feedback.
11 replies
-
|
When I run cloudstack-management 4.19.1.2 with java17, I got the following error |
Beta Was this translation helpful? Give feedback.
-
So the problem was that I followed the |
Beta Was this translation helpful? Give feedback.
-
|
right. the doc needs to be fixed. however, when you downgrade to java11, cloudstack should work well. the db.cloud.password and db.cloud.password are both decrypted as "password", are they correct ? |
Beta Was this translation helpful? Give feedback.
-
Yes, the db.cloud.password and db.usage.password is both set to "password".
After setting default Java vertion to 11, the management server works well. |
Beta Was this translation helpful? Give feedback.
Answer selected by
weizhouapache
-
It worked for me as well but SSO via SAML is not working after the upgrade. |
Beta Was this translation helpful? Give feedback.
-
@Avneetdabas Can you upgrade to 4.19.1.3 ? |
Beta Was this translation helpful? Give feedback.
-
I was having the exact same issue on AlmaLinux 9.3 and this resolved the problem as well. |
Beta Was this translation helpful? Give feedback.
0 replies
-
@purteppers |
Beta Was this translation helpful? Give feedback.
0 replies
-
Yes, specifically 8.0.36 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I suggest we shouldn't use MySQL from its official repo, we should use MariaDB from repo of Debian Stable.
…On October 30, 2024 4:02:10 PM UTC, purteppers ***@***.***> wrote:
> > I was having the exact same issue on AlmaLinux 9.3 and this resolved the problem as well.
>
> @purteppers do you use mysql 8.0 as well ?
Yes, specifically 8.0.36
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I tested upgrade on debian 12 and alma9 with mysql 8.0, both worked |
Beta Was this translation helpful? Give feedback.
-
|
I did not find any valuable information in the log. is there anything wrong in the db upgrade ? cc @gobiursus @purteppers |
Beta Was this translation helpful? Give feedback.
8 replies
-
Do you have the original db.properties ? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately since I was just testing the install I do not, I can run through the install again on a fresh Alma install if you want and provide the results. |
Beta Was this translation helpful? Give feedback.
-
Ok I suspect it is caused by char # in the password |
Beta Was this translation helpful? Give feedback.
-
|
The management service is currently working with the password with '#' and no encryption, so I'm not sure what changing the password will accomplish. Do you want me revert back to the encryption settings and change the password? |
Beta Was this translation helpful? Give feedback.
-
|
ignore it, thanks anyway @purteppers |
Beta Was this translation helpful? Give feedback.
-
|
How did you install your MySQL?Please don't use the MySQL from MySQL's official repo.
On Thursday, October 31, 2024 at 03:30:46 p.m. GMT, Wei Zhou ***@***.***> wrote:
I have tested debian/ubuntu/alma/rocky with different versions, cannot reproduce the issue unfortunately.
@purteppers
can you share the output of some commands ?
egrep 'secret|password' /etc/cloudstack/management/db.properties
cat /etc/cloudstack/management/key
# the logs when the issue is not fixed
egrep 'c.c.u.c.CloudStackEncryptor|c.c.u.Database' /var/log/cloudstack/management/management-server.log
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
@gobiursus |
Beta Was this translation helpful? Give feedback.
-
|
I installed MariaDB 10.11.6 from Debian 12 Stable repo byapt install mariadb-server
On Thursday, October 31, 2024 at 03:37:05 p.m. GMT, Wei Zhou ***@***.***> wrote:
How did you install your MySQL?Please don't use the MySQL from MySQL's official repo.
@gobiursus
I installed mysql 8.0.36 from the alma9/rocky9 official repo by
dnf install -y mysql-server
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
I tried mariadb 10 on debian12 and ubuntu 24, also mysql 8.0.x on alma9/rocky9, no issues found Probably it is not related to mariadb/mysql, but some value in db.properties which cannot be decrypted. |
Beta Was this translation helpful? Give feedback.
-
|
Java 17 is the default version of Debian 12.
Also the default database in Debian 12 is MariaDB not MySQL.
…On November 6, 2024 2:45:30 PM UTC, Wei Zhou ***@***.***> wrote:
@penguin2716
if you run with cloudstack 4.19 or prior, JAVA 17 is not supported.
JAVA 17 is supported since 4.20 (will be released in the coming weeks): #8497
--
Reply to this email directly or view it on GitHub:
#9880 (reply in thread)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
yes @gobiursus |
Beta Was this translation helpful? Give feedback.
-
|
@gobiursus @penguin2716 @purteppers |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
It looks reasonable. I just confirmed that the management server starts without 503 error on Debian 12.8 / Java 11.0.25+9 (Temurin) / MariaDB 10.11.6. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
closed this for lack of interaction, please reopen or open a new item if applicable |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I encounter this problem upgrading from 4.19.1.1 to 4.20.0. After upgrade I tried to access UI but got the 503 error. After downgrading by selecting java 11, I was able to access the UI again. I thought 4.20 needed java 17 to work?? Still searching with a solution. I revert my mgm server back to 4.19.1 java11 and all my KVM host are running on 4.20. All my system VM are also on 4.19.1. |
Beta Was this translation helpful? Give feedback.
5 replies
-
4.20 works with both java 11 and java 17. java 17 is recommended. All mgmt server and kvm hosts should use the same ACS version. If java 17 does not work with ACS 4.20.0, can you share more logs ? |
Beta Was this translation helpful? Give feedback.
-
|
/etc/default/cloudstack-management: Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements. See the NOTICE filedistributed with this work for additional informationregarding copyright ownership. The ASF licenses this fileto you under the Apache License, Version 2.0 (the"License"); you may not use this file except in compliancewith the License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing,software distributed under the License is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANYKIND, either express or implied. See the License for thespecific language governing permissions and limitationsunder the License.JAVA_OPTS="-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx6G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:ErrorFile=/var/log/cloudstack/management/cloudstack-management.err " CLASSPATH="/usr/share/cloudstack-management/lib/:/etc/cloudstack/management:/usr/share/cloudstack-common:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management:/usr/share/java/mysql-connector-java.jar:/usr/share/cloudstack-mysql-ha/lib/" BOOTSTRAP_CLASS=org.apache.cloudstack.ServerDaemon ################################################################################################ This option here should be used with 'systemmd' based operating systems such as CentOS7, Ubuntu 16, and so on.#JAVA_DEBUG="-agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n" On the other hand, this option is used by CentOS6.#JAVA_DEBUG="-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n" |
Beta Was this translation helpful? Give feedback.
-
|
sorry, I have no idea why it came out with different front size. I just copy and paste. |
Beta Was this translation helpful? Give feedback.
-
|
After upgrading to 4.20 with java17 root@xx-mgm-01:/var/log/cloudstack/management# tail -f management-server.log |
Beta Was this translation helpful? Give feedback.
-
|
I also created an issue about this #10960 where I pasted more logs. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Yes, the db.cloud.password and db.usage.password is both set to "password".
After setting default Java vertion to 11, the management server works well.