Utility to Discover and Delete Old Entities Based on Age

[EtaCassiopeia]

Short description

In Akka persistence Cassandra, while we have a cleanup utility requiring specific persistent ids, it's cumbersome for large deployments. Setting a TTL upfront isn't always desired. Proposing a tool to auto-discover entities by age and periodically delete them post their lifespan. Keen to understand the community's perspective and explore if there's traction for such a feature

Details

While working with Akka persistence Cassandra, I realized that while there's a cleanup utility provided for removing specific entities (by supplying persistent ids), this approach can be cumbersome especially when managing a large number of entities over time. In particular, situations where entities are created frequently and have a certain lifespan, it becomes necessary to periodically remove them.

One might argue that the de-facto approach for such requirements is setting a TTL. However, there are certain scenarios where setting an upfront TTL may not be suitable or desired. What if, instead, we could have a comprehensive tool that automatically discovers entities based on their creation age and purges them using a moving window approach?

The Challenge:

1. Discovering Old Entities: The current cleanup utility requires explicit persistent ids, which may not be feasible for larger deployments or when old entities need to be identified based on their age.

2. Periodic Cleanup: Without a TTL, old entities continue to exist, potentially occupying unnecessary space and resources. A tool that could continually check the database and delete expired entities after they've passed their lifespan would be immensely valuable.

Potential Solutions:

While I have worked on an in-house solution for this particular challenge and am inclined to share it, I wanted to start a discussion here to gauge the community's feedback and see if others also face a similar challenge. It would be interesting to know:

• How others in the community manage old entities in Akka persistence Cassandra?
• Is there any interest or traction for such a feature within the project?

If there's consensus on the need for such a feature, I'd be more than happy to share our solution, collaborate on refinements if needed, and potentially see it being integrated into the project.

I believe this tool would add value for users who want a more dynamic approach to cleaning up old entities, especially when TTL is not the preferred approach. Looking forward to the community's feedback!

[patriknw]

I think it's a reasonable request. Isn't the main problem that there is no efficient way of finding "old entitits", since the timestamp isn't part of the partition key? One way could be to have a projection that would store timestamp and persistence id in a separate table that can be queried to find old entities. Then use the cleanup tool with the found persistence ids.