Server-wide media retention policy

[babolivier]

A feature that's both interesting to have and fairly well requested is the ability to configure a media retention policy at the server level.

A first approach would have been to base the TTL of a media on its date of upload, but then we'd likely delete still-in-use avatars, medias used in community descriptions, etc.

Therefore, a preferred approach is to base that TTL on the date it was last accessed at, to ensure we don't delete media that are still being used. FTR, that date of last access is stored by Synapse for both remote and local media, so it's technically doable.

Another thing to consider is that we currently don't have any way in deleting a media in Synapse, so that'd need to be added in.

Also, we'd need to figure out how this feature would handle quarantined media.

cc @rxl881

Related: #6459, #3479, https://github.com/matrix-org/matrix-doc/issues/790

[richvdh]

there are open issues/MSCs around deleting media: we should hunt them down and link them

[babolivier]

there are open issues/MSCs around deleting media: we should hunt them down and link them

I've updated the issue's description with the few issues I could find. I didn't find any MSC relating to that though.

[richvdh]

Thanks. MSC2278 seems to be the MSC I was thinking of.

[babolivier]

After chatting with @neilisfragile and @lampholder about the quarantine concern, the conclusion is that a media retention policy should ignore quarantined media.

[IF-Adin]

I would like to add that avatars and room avatars should probably be stored in a different way. They mostly seem to be cached, so they are probably not accessed all that often in smaller groups, which mean they would be purged, too.

May add a tag or something to all user and room Avatars?

Or was that what you mean by quarantined data?

[IF-Adin]

Nevermind, as it turns out i can not read, you have already thought of that.

[MamasLT]

Great points made by babolivier.

There are few options for dealing with media:
https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_remote_media.rst
https://github.com/matrix-org/synapse/blob/master/docs/admin_api/media_admin_api.md

But they don't seem to do the main thing - to delete it on local server. I would agree that there should be a server-wide media retention policy available, that could be set in config in the same way as message retention is now.
The quarantine, as I understand should be done just before deleting the particular media files, so there won't be any issues for clients.
Doing the quarantine and then deleting all media manually (not sure if there is any other way at this time) sounds a bit over complicated. Cheers!

[NHAS]

Not sure if this is the correct place to note this.
However when a general "event" retention policy is applied to a server it doesnt remove media such as images or uploaded files.
This means there are a large number of files that have no event referencing them. This issue would help somewhat to solve that. However the best solution would be for the server retention policy to delete said media.