QueuePool limit of size 150 overflow 30 reached, connection timed out, timeout 30.00

[starboyate]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.4.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

2025-06-24 10:55:59,187.187 ERROR [Thread-595 (worker_with_context)] [app_generator.py:500] - Unknown Error when generating
Traceback (most recent call last):
  File "/app/api/core/app/apps/advanced_chat/app_generator.py", line 471, in _generate_worker
    conversation = self._get_conversation(conversation_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/app/apps/message_based_app_generator.py", line 260, in _get_conversation
    conversation = db.session.query(Conversation).filter([Conversation.id](http://conversation.id/) == conversation_id).first()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2728, in first
    return self.limit(1)._iter().first()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
    conn = self._connection_for_bind(bind)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in _connection_for_bind
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
    conn = bind.connect()
           ^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
    raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 150 overflow 30 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
2025-06-24 10:55:59,396.396 INFO [ThreadPoolExecutor-138_0] [_client.py:1038] - HTTP Request: POST http://sandbox-ap-southeast-1-production/v1/sandbox/run "HTTP/1.1 200 OK"
2025-06-24 10:56:01,421.421 INFO [ThreadPoolExecutor-169_0] [_client.py:1038] - HTTP Request: POST http://sandbox-ap-southeast-1-production/v1/sandbox/run "HTTP/1.1 200 OK"
2025-06-24 10:56:01,677.677 INFO [ThreadPoolExecutor-167_0] [_client.py:1038] - HTTP Request: POST http://sandbox-ap-southeast-1-production/v1/sandbox/run "HTTP/1.1 200 OK"
2025-06-24 10:56:02,957.957 ERROR [Thread-285 (worker_with_context)] [node.py:285] - error while executing llm node
Traceback (most recent call last):
  File "/app/api/core/workflow/nodes/llm/node.py", line 212, in _run
    prompt_messages, stop = self._fetch_prompt_messages(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/workflow/nodes/llm/node.py", line 704, in _fetch_prompt_messages
    model = ModelManager().get_model_instance(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 430, in get_model_instance
    provider_model_bundle = self._provider_manager.get_provider_model_bundle(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 253, in get_provider_model_bundle
    provider_configurations = self.get_configurations(tenant_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 98, in get_configurations
    provider_name_to_provider_records_dict = self._get_all_providers(tenant_id)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 401, in _get_all_providers
    providers = session.scalars(stmt)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2470, in scalars
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
    conn = self._connection_for_bind(bind)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in _connection_for_bind
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
    conn = bind.connect()
           ^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
    raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 150 overflow 30 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[starboyate]

At first, my database only supported a maximum of 400 connections, but this error occurred. I tried to increase the number of connections to 800, but still encountered this exception. How should I solve this problem

[junbinding]

At first, my database only supported a maximum of 400 connections, but this error occurred. I tried to increase the number of connections to 800, but still encountered this exception. How should I solve this problem

In my actual usage scenario, I checked all possible places to initiate a connection and finally made the following environment variable settings. It may be useful to you.

// api & worker = pg 20 + pgvector 5
SQLALCHEMY_POOL_SIZE=20
SERVER_WORKER_AMOUNT=6
CELERY_WORKER_AMOUNT=3

// max total connections = server_worker 6 * 25 + celery_worker 3 * 25 + celery_session 5 * 3 + dify_plugin 30

[FyhSky]

Has this problem been resolved? I have also reproduced it.

[FyhSky]

I have set these parameters, but they still don't work.

QueuePool limit of size 5 overflow 10 reached：connection timed out, timeout 30.00（Background on this error at: https://sqlalche.me/e/20/3o7r）

@dousubot

[saurabhy27]

Hi maintainers 👋

I’d like to work on this issue and submit a PR to make SQLALCHEMY_POOL_TIMEOUT configurable via environment variable (so users can adjust how long to wait for a connection before a timeout).

Could you please assign this issue to me?

Thanks! 🙏