UniqueViolation('duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"

[matthewbyrne]

Self-Hosted Version

22.12.0

CPU Architecture

x86_64

Docker Version

20.10.23

Docker Compose Version

1.29.2

Steps to Reproduce

We did a backup and restore of our data from v22.12.0 to v22.12.0

Expected Result

No internal errors.
All events to be stored into Sentry

Actual Result

We're getting a lot of internal project errors on our sentry instance, and reports that some events are not being stored (test events being the most obvious and reproducible to report)

UniqueViolation('duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\nDETAIL:  Key (project_id, short_id)=(12, 142817) already exists.\n')
SQL: INSERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "message", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first_release_id", "resolved_at", "active_at", "time_spent_total", "time_spent_count", "score", "is_public", "data", "short_id", "type") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sentry_groupedmessage"."id"

ESC[36mworker_1                                    |ESC[0m SQL: INSERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "message", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first
_release_id", "resolved_at", "active_at", "time_spent_total", "time_spent_count", "score", "is_public", "data", "short_id", "type") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sen
try_groupedmessage"."id"
ESC[36mworker_1                                    |ESC[0m 11:25:26 [ERROR] celery.app.trace: Task sentry.tasks.store.save_event[f5bfedfd-b7e4-47e7-be2e-a42837ea2a9b] raised unexpected: IntegrityError('UniqueViolation(\'duplicate key valu
e violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\\nDETAIL:  Key (project_id, short_id)=(11, 4414) already exists.\\n\')\nSQL: INSERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "mes
sage", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first_release_id", "resolved_at", "active_at", "time_spent_total", "time_spent_count", "score", "is_public", "data", "short_id", "type") VALUES
 (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sentry_groupedmessage"."id"') (data={'hostname': 'adcaa4897694', 'id': 'f5bfedfd-b7e4-47e7-be2e-a42837ea2a9b', 'name': 'sentry.tasks.store.sa
ve_event', 'exc': 'IntegrityError(\'UniqueViolation(\\\'duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\\\\nDETAIL:  Key (project_id, short_id)=(11, 4414) already exists.\\\\n\\\')\
\nSQL: INSERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "message", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first_release_id", "resolved_at", "active_at", "time_spent_tot
al", "time_spent_count", "score", "is_public", "data", "short_id", "type") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sentry_groupedmessage"."id"\')', 'traceback': 'Traceback (mo
st recent call last):\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 91, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/bas
e.py", line 86, in execute\n    return self.cursor.execute(sql, clean_bad_params(params))\npsycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\nDETAIL:
 Key (project_id, short_id)=(11, 4414) already exists.\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils
.py", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner\n    raise_the_exception(self.db, e)\n  File "/usr/local/lib/pyth
on3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner\n    return func(self, *
args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner\n    raise exc_info[0](msg).with_traceback(exc_info[2])\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgre
s/decorators.py", line 91, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute\n    return self.cursor.execute(sql, clean_bad_params(params
))\npsycopg2.errors.UniqueViolation: UniqueViolation(\'duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\\nDETAIL:  Key (project_id, short_id)=(11, 4414) already exists.\\n\')\nSQL: I
NSERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "message", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first_release_id", "resolved_at", "active_at", "time_spent_total", "ti
me_spent_count", "score", "is_public", "data", "short_id", "type") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sentry_groupedmessage"."id"\n\nThe above exception was the direct ca
use of the following exception:\n\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.8/s
ite-packages/sentry_sdk/integrations/celery.py", line 207, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 57, in reraise\n    raise value\n  File "/usr/local/lib/python3.8/si
te-packages/sentry_sdk/integrations/celery.py", line 202, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__\n    return self.run(*args, **kwargs)
\n  File "/usr/local/lib/python3.8/site-packages/sentry/tasks/base.py", line 64, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/tasks/store.py", line 807, in save_event\n    _do_save
_event(cache_key, data, start_time, event_id, project_id, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/tasks/store.py", line 702, in _do_save_event\n    manager.save(\n  File "/usr/local/lib/python3.8/site-packages/sen
try/utils/metrics.py", line 283, in inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/event_manager.py", line 536, in save\n    group_info = _save_aggregate(\n  File "/usr/local/lib/python3.8/site
-packages/sentry/event_manager.py", line 1463, in _save_aggregate\n    group = _create_group(project, event, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/event_manager.py", line 1651, in _create_group\n    return Group
.objects.create(\n  File "/usr/local/lib/python3.8/site-packages/sentry/silo/base.py", line 152, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 8
2, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/silo/base.py", line 152, in override\n    return original_method(*args, **kwargs)\n  File "/usr/lo
cal/lib/python3.8/site-packages/django/db/models/query.py", line 422, in create\n    obj.save(force_insert=True, using=self.db)\n  File "/usr/local/lib/python3.8/site-packages/sentry/models/group.py", line 477, in save\n    super().save(*
args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 743, in save\n    self.save_base(using=using, force_insert=force_insert,\n  File "/usr/local/lib/python3.8/site-packages/sentry/silo/base.py",
 line 152, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 780, in save_base\n    updated = self._save_table(\n  File "/usr/local/lib/python3.8/site-
packages/django/db/models/base.py", line 873, in _save_table\n    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 910, in _do_inser
t\n    return manager._insert([self], fields=fields, return_id=update_pk,\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwa
rgs)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert\n    return query.get_compiler(using=using).execute_sql(return_id)\n  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql
/compiler.py", line 1377, in execute_sql\n    cursor.execute(sql, params)\n  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 563, in execute\n    return real_execute(self, sql, params)\n  Fil
e "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File "/usr/local/lib/python3.8/site-packages/django/db
/backends/utils.py", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute\n    return self.cursor.execute(sq
l, params)\n  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils
.py", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 77, in inner\n    raise_the_exception(self.db, e)\n  File "/usr/local/lib/pyth
on3.8/site-packages/sentry/db/postgres/decorators.py", line 75, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 18, in inner\n    return func(self, *
args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/decorators.py", line 95, in inner\n    raise exc_info[0](msg).with_traceback(exc_info[2])\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgre
s/decorators.py", line 91, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/sentry/db/postgres/base.py", line 86, in execute\n    return self.cursor.execute(sql, clean_bad_params(params
))\ndjango.db.utils.IntegrityError: UniqueViolation(\'duplicate key value violates unique constraint "sentry_groupedmessage_project_id_short_id_29374dd7_uniq"\\nDETAIL:  Key (project_id, short_id)=(11, 4414) already exists.\\n\')\nSQL: IN
SERT INTO "sentry_groupedmessage" ("project_id", "logger", "level", "message", "view", "num_comments", "platform", "status", "times_seen", "last_seen", "first_seen", "first_release_id", "resolved_at", "active_at", "time_spent_total", "tim
e_spent_count", "score", "is_public", "data", "short_id", "type") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "sentry_groupedmessage"."id"\n', 'args': '()', 'kwargs': "{'cache_key'
: 'e:9135cd98f6784f0897f6c7413018e199:11', 'data': None, 'start_time': 1674818724.0, 'event_id': '9135cd98f6784f0897f6c7413018e199', 'project_id': 11}", 'description': 'raised unexpected', 'internal': False})

Event ID

No response

[hubertdeng123]

Closing this as this is a duplicate of getsentry/sentry#36868

Our team will be working on this in the upcoming months.

[matthewbyrne]

Is there anything to be done for my corrupt database now though? Does fixing the backup and restore fix bad databases caused by the broken backup/restore?
Ironically we wanted to clean up our self-hosted install in order to get more precise metrics so we could price up a Sentry SaaS offering, but now 1. We're getting lots of internal errors so our metrics are incorrect, 2. We don't have confidence all the data being sent to sentry is being ingested because of internal errors/corrupt database/test errors which are not showing in Sentry.

[hubertdeng123]

Sorry to hear that you've had to experience this, I understand it's definitely not ideal. Did you by chance backup your postgres db before performing a backup/restore? I'm assuming you followed the steps here for a quick backup and restore

[matthewbyrne]

We have a backup of our data, but if a fix for the backup and restore is months away then we have months of uncertainty with our sentry instance, and months for that backup to become outdated and not fit for use (if/when a backup/restore fix is available).

We can't do a fresh install of sentry either to help fix things. We just did this to fix other issues, but now backup and restore has caused new ones.

It seems our options are to wait an unknown period of time for a possible fix (backup and restore fix is one thing, but the corrupt data it has caused is now another), or completly start our sentry instance afresh (causing toil on all our teams to recreate and setup their projects and integrations).

[hubertdeng123]

By backup, I meant a backup of the docker volumes that your old Sentry instance used. If that's the case, it can probably still be recovered. However, if you only have the backup.json, it is unlikely that the db can be restored 😕. I noticed that you said you tried a fresh install of Sentry recently to fix other issues? What were those? Unfortunately self-hosted Sentry is not built well for scale.

[hubertdeng123]

Otherwise, it looks like some community members have played around and came up with a script if you'd like to go down that route:
getsentry/sentry#36868 (comment)

[matthewbyrne]

By backup, I meant a backup of the docker volumes that your old Sentry instance used. If that's the case, it can probably still be recovered. However, if you only have the backup.json, it is unlikely that the db can be restored 😕. I noticed that you said you tried a fresh install of Sentry recently to fix other issues? What were those? Unfortunately self-hosted Sentry is not built well for scale.

We do not have a backup of docker volumes.
We're not running self-hosted sentry at scale.
Our install path is in line with the documentation.

We had issues with disk usage, and a significant number of internal events from Sentry itself consuming disk, which caused a feedback loop which consumed more disk. Some of our investigation on that is here but ultimately were unsuccessful in fixing our instance. We felt a fresh install was the best method to resolve those issues, but the broken backup/restore script has scuppered those efforts.

Otherwise, it looks like some community members have played around and came up with a script if you'd like to go down that route:
getsentry/sentry#36868 (comment)

Been there, done that, but unsure what's causing the issue mentioned in my first comment here.
https://gist.github.com/Cactiw/c2397561966e2f343b0563a6c5b7f17f?permalink_comment_id=4448446#gistcomment-4448446