Restore feature not working - django.core.serializers.base.DeserializationError

[mesenger]

Self-Hosted Version

22.12.0.dev0+99926859321bd1f82d1ac4b07bc8601e3d78b9c9

CPU Architecture

x86_64

Docker Version

20.10.21

Docker Compose Version

1.29.2

Steps to Reproduce

1. Take backup of existing Sentry installation - OK
2. Remove Sentry, purge volumes, install same version ./install.sh --skip-user-prompt - OK
3. try to restore previous backup: docker-compose run --rm -T web import /etc/sentry/backup.json - Error:
   docker-compose run --rm -T web import /etc/sentry/backup.json
   Starting sentry-self-hosted_symbolicator_1 ... done
   Starting sentry-self-hosted_postgres_1 ... done
   Starting sentry-self-hosted_zookeeper_1 ... done
   Starting sentry-self-hosted_clickhouse_1 ... done
   Starting sentry-self-hosted_memcached_1 ... done
   Starting sentry-self-hosted_smtp_1 ... done
   Starting sentry-self-hosted_redis_1 ... done
   Starting sentry-self-hosted_kafka_1 ... done
   Starting sentry-self-hosted_snuba-subscription-consumer-transactions_1 ... done
   Starting sentry-self-hosted_snuba-replacer_1 ... done
   Starting sentry-self-hosted_snuba-sessions-consumer_1 ... done
   Starting sentry-self-hosted_snuba-consumer_1 ... done
   Starting sentry-self-hosted_snuba-api_1 ... done
   Starting sentry-self-hosted_snuba-transactions-consumer_1 ... done
   Starting sentry-self-hosted_snuba-outcomes-consumer_1 ... done
   Starting sentry-self-hosted_snuba-subscription-consumer-events_1 ... done
   Creating sentry-self-hosted_web_run ... done
   Updating certificates in /etc/ssl/certs...
   0 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d...
   done.
   15:06:27 [INFO] sentry.plugins.github: apps-not-configured
   Traceback (most recent call last):
   File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer
   objects = json.loads(stream_or_string)
   File "/usr/local/lib/python3.8/json/init.py", line 357, in loads
   return _default_decoder.decode(s)
   File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
   File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
   raise JSONDecodeError("Expecting value", s, err.value) from None
   json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main
func(**kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return _callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import
for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError
ERROR: 1

Unable to restore

Expected Result

Restore would finish OK and project data would be restored correctly

Actual Result

backup is not restored, producing error:

15:06:27 [INFO] sentry.plugins.github: apps-not-configured
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer
objects = json.loads(stream_or_string)
File "/usr/local/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main
func(**kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return _callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import
for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError
ERROR: 1

Event ID

No response

[emmatyping]

Did you check the output of the generated JSON file? It seems that something expected is missing.

[mesenger]

I am not an expert on json syntax, it looks good enough to me. jq reports some errors. Here are some snippets :

root@sentry1:~/sentry# jq sentry/backup.json
jq: error: sentry/0 is not defined at <top-level>, line 1:
sentry/backup.json
jq: error: backup/0 is not defined at <top-level>, line 1:
sentry/backup.json       
jq: 2 compile errors

root@sentry1:~/sentry# head sentry/backup.json 
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
[
{
  "model": "sites.site",
  "pk": 1,
  "fields": {
    "domain": "example.com",

root@sentry1:~/sentry# tail sentry/backup.json
{
  "model": "sentry.projectoption",
  "pk": 7,
  "fields": {
    "project": 2,
    "key": "sentry:option-epoch",
    "value": 8
  }
}
]

[mesenger]

I removed the top part from backup.json, starting now with this:
[ { "model": "sites.site", "pk": 1, "fields": { "domain": "example.com",

now I get more specific error:

`13:44:04 [INFO] sentry.plugins.github: apps-not-configured
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 143, in Deserializer
data[field.name] = field.to_python(field_value)
File "/usr/local/lib/python3.8/site-packages/sentry/db/models/fields/picklefield.py", line 54, in to_python
return json.loads(value)
File "/usr/local/lib/python3.8/site-packages/sentry/utils/json.py", line 120, in loads
return _default_decoder.decode(value)
File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 392, in raw_decode
raise TypeError("Input string must be text, not bytes")
TypeError: Input string must be text, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 192, in main
func(**kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return _callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import
for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 145, in Deserializer
raise base.DeserializationError.WithData(e, d['model'], d.get('pk'), field_value)
django.core.serializers.base.DeserializationError: Input string must be text, not bytes: (sentry.option:pk=1) field_value was '1670942824.3678975'
ERROR: 1
`

I can see this entry in backup.json, but not sure what could be wrong with it:
{ "model": "sentry.option", "pk": 1, "fields": { "key": "sentry:last_worker_ping", "value": 1670942824.3678975, "last_updated": "2022-12-13T14:47:04.367Z" } },

[emmatyping]

Thanks for digging into that! In that case, I believe this might be the same as getsentry/sentry#36868 I will close this in favor of that issue. Unfortunately right now we don't have the bandwidth to fix the backup, but we likely will tackle it Q1.