feat: add in-memory repo to improve workflow performance

[xinlmain]

Summary

Addresses #20147
See also #10982, See also #19325

The default workflow-run api is very slow on simple tasks, making it unacceptable for real time interactions. The api calls spend most of the time interacting with DB. In a scenario where the api is called by another service, there is not much sense to save each call into DB.

So here I added a fast mode for workflow api, it can be turned on by setting WORKFLOW_NODE_EXECUTION_REPO_MODE=memory. This enables deploying a dedicated api service for external api calls, with the tradeoff of not able to see execution detail in application logs from the web page.

Checklist

Important

Please review the checklist below before submitting your pull request.

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

[Copilot]

Pull Request Overview

This PR introduces an in-memory repository mode to improve workflow API performance by avoiding unnecessary database interactions. Key changes include:

• Adding a new environment variable WORKFLOW_NODE_EXECUTION_REPO_MODE and its documentation.
• Implementing an in-memory workflow node execution repository and integrating it via a new repository factory.
• Updating application configuration and app generator code to use the new repository factory.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
docker/docker-compose.yaml	Adds WORKFLOW_NODE_EXECUTION_REPO_MODE environment variable with default sql.
docker/.env.example	Updates the environment file with the new repository mode option.
api/core/workflow/repository/workflow_node_execution_repository.py	Introduces the ExecRepoMode enum for repository mode selection.
api/core/repositories/workflow_node_execution_factory.py	Implements a factory to choose between SQL and in-memory repository modes.
api/core/repositories/in_memory_workflow_node_execution_repository.py	Provides the in-memory repository implementation for execution records.
api/core/repositories/init.py	Exports the new in-memory repository.
api/core/app/apps/workflow/app_generator.py	Refactors to use the repository factory instead of directly instantiating SQL.
api/configs/feature/init.py	Adds new configuration field for repository mode.
api/.env.example	Updates the environment file with the new repository mode setting.

[laipz8200]

Great PR! However, the two issues(PRs) you’ve referenced are not currently open. Could you please create a proposal for this implementation? We’d like to discuss some implementation details there. You can refer to issue #19429 for guidance.

[xinlmain]

Great PR! However, the two issues(PRs) you’ve referenced are not currently open. Could you please create a proposal for this implementation? We’d like to discuss some implementation details there. You can refer to issue #19429 for guidance.

@laipz8200 Thanks, I have created this one #20147