An error occurred: save data failed: allocated size is greater than max storage size

[Rajendrazepto]

Problem:
Getting below error for an new slack message event received.

Error:
An error occurred: save data failed: allocated size is greater than max storage size Traceback (most recent call last): File "/app/storage/cwd/solaoi/slack-thread-bot-0.0.5@a8ac7d3e711b86e6bd5fa08b52e9308efac7d10612eed800ea368f683cb07095/endpoints/slack.py", line 546, in _invoke self.session.storage.set( File "/app/storage/cwd/solaoi/slack-thread-bot-0.0.5@a8ac7d3e711b86e6bd5fa08b52e9308efac7d10612eed800ea368f683cb07095/.venv/lib/python3.12/site-packages/dify_plugin/invocations/storage.py", line 23, in set for data in self._backwards_invoke( File "/app/storage/cwd/solaoi/slack-thread-bot-0.0.5@a8ac7d3e711b86e6bd5fa08b52e9308efac7d10612eed800ea368f683cb07095/.venv/lib/python3.12/site-packages/dify_plugin/core/runtime.py", line 315, in _full_duplex_backwards_invoke yield from self._line_converter_wrapper(reader.read(timeout_for_round=1), data_type) File "/app/storage/cwd/solaoi/slack-thread-bot-0.0.5@a8ac7d3e711b86e6bd5fa08b52e9308efac7d10612eed800ea368f683cb07095/.venv/lib/python3.12/site-packages/dify_plugin/core/runtime.py", line 217, in _line_converter_wrapper raise Exception(event.message) Exception: save data failed: allocated size is greater than max storage size

How to reproduce?
Send bulk slack message events to slack-thread-bot endpoint/plugin, after some time it will lead to above error as the It reach the max storage limit.

[djjimba]

Hi @solaoi , first of all thank you so much for creating this amazing plugin!

We are using Dify Cloud version and get the same error as explained in this issue.
The exact error message we get is

Sorry, I'm having trouble processing your request. Please try again later. Error: save data failed: allocated size is greater than max storage size

Is my understanding correct that, due to how Dify Plugin's Persistent Storage works, and because there is no Key Deletion mechanism currently in the plugin (I checked slack.py), eventually, when the storage reaches 1MB as shown below, the plugin won't be able to return answers to Slack?

Is there any chance a Persistent Storage clean-up system may be introduced to the plugin while we wait for Dify's update on a more flexible and powerful storage system?

I see a garbage collection type of logic around CACHE_DURATION, but wasn't entirely sure that was sufficiently cleaning up old cache sufficiently as we still get the error.

Currently, a KV database is provided to meet storage needs, and more flexible and powerful storage interfaces may be introduced in the future based on actual usage.

ref

[djjimba]

I have made a PR for a partial solution to prune stale threads' cache from Dify Plugin's Persistent Storage.
This solution is not complete, as it will not touch (delete) old keys that were not mentioned.

However, I thought that as a partial solution, it could make sense and wanted to know what you think.

As another temporary solution, would it be possible to increase the 1MB limit for Maximum Storage?

[hirune924]

This PR perfectly resolves this issue. The used key-value pairs will be managed in persistent storage, with appropriate garbage collection performed.
#19