Cannot get backup and restore to work

[drmrbrewer]

The restore section of the page relating to backups has the following statement:

This directory automatically gets mounted to /etc/sentry so you can run the following to restore your backup

But in my installation, this directory is not mounted to /etc/sentry... there is nothing at all mounted to /etc/sentry. So either there's a bug in the docs, or a bug in the installation.

[drmrbrewer]

If it means that the directory is mounted to /etc/sentry as part of the import process, and I just run that restore command as-is (with backup.json having been dumped into the sentry sub-folder within the main self-hosted repo folder ~/sentry/self-hosted), I just get the following errors:

~/sentry/self-hosted# docker compose run --rm -T web import /etc/sentry/backup.json
[+] Running 16/0
 ⠿ Container sentry-self-hosted-memcached-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-postgres-1                                  Running                                  0.0s
 ⠿ Container sentry-self-hosted-zookeeper-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-clickhouse-1                                Running                                  0.0s
 ⠿ Container sentry-self-hosted-kafka-1                                     Running                                  0.0s
 ⠿ Container sentry-self-hosted-symbolicator-1                              Running                                  0.0s
 ⠿ Container sentry-self-hosted-smtp-1                                      Running                                  0.0s
 ⠿ Container sentry-self-hosted-redis-1                                     Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-outcomes-consumer-1                   Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1  Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-sessions-consumer-1                   Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-consumer-1                            Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-replacer-1                            Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-api-1                                 Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-transactions-consumer-1               Running                                  0.0s
 ⠿ Container sentry-self-hosted-snuba-subscription-consumer-events-1        Running                                  0.0s
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
11:20:16 [INFO] sentry.plugins.github: apps-not-configured
/usr/local/lib/python3.8/site-packages/sentry/runner/initializer.py:551: DeprecatedSettingWarning: The SENTRY_URL_PREFIX setting is deprecated. Please use SENTRY_OPTIONS['system.url-prefix'] instead.
  warnings.warn(DeprecatedSettingWarning(old, "SENTRY_OPTIONS['%s']" % new))
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if key_extra_len is 0:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer
	objects = json.loads(stream_or_string)
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
	return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
	obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
	raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
	sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 188, in main
	func(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
	return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
	rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
	return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
	return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
	return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
	return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
	return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
	return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/backup.py", line 19, in import_
	for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 73, in Deserializer
	raise DeserializationError() from exc
django.core.serializers.base.DeserializationError

[AbhiPrasad]

Hey thanks for writing in. I'm going to transfer this issue to the self-hosted repo so the folks who work on that can help out!

[drmrbrewer]

Thanks, @AbhiPrasad.

@chadwhitacre I may be doing something wrong, but I'd expect that (as a test) I should be able to do a backup followed immediately by a restore of that backup, based on the steps as described on the Self-Hosted Backup & Restore page (using docker compose rather than docker-compose):

Backup:

docker compose run --rm -T -e SENTRY_LOG_LEVEL=CRITICAL web export > sentry/backup.json

Restore:

docker compose run --rm -T web import /etc/sentry/backup.json

But I just get the error messages as shown previously when I run the restore command.

[chadwhitacre]

But in my installation, this directory is not mounted to /etc/sentry... there is nothing at all mounted to /etc/sentry

Quick skim (and recollection based on experience) suggests this is unexpected, afaict we mount /etc/sentry for all containers that use the x-sentry-defaults config:

self-hosted/docker-compose.yml

Line 69 in 25381da

- "./sentry:/etc/sentry"

[chadwhitacre]

I should be able to do a backup followed immediately by a restore

The export/import functionality is somewhat undermaintained, so it wouldn't surprise me if it's broken. I'm not sure what testing we have for it over in sentry, a high-level integration test for the basic functionality would be good to have if we don't already.

[drmrbrewer]

this is unexpected, afaict we mount /etc/sentry for all containers that use the x-sentry-defaults config:

# ls /etc/sentry
ls: cannot access '/etc/sentry': No such file or directory

The sentry installation is otherwise working fine.

The export/import functionality is somewhat undermaintained, so it wouldn't surprise me if it's broken.

Would be great to get this working properly because in IMHO backup and restore is a key feature.

[chadwhitacre]

backup and restore is a key feature

Since it's only metadata and not actual event data, it's less critical than it would be if it were a full backup solution. For that we bump to Docker.

[drmrbrewer]

Since it's only metadata and not actual event data

For me, metadata is more important (to back up) than actual event data, because event data will just repopulate over time but metadata won't.

[drmrbrewer]

@chadwhitacre did you confirm this as a bug, or is it just something I'm misunderstanding about the backup and restore process?

[chadwhitacre]

Have not had time to confirm, no. You made it past the original /etc/sentry mount issue, yes? Now we're on JSONDecodeError during restore? Did you inspect the contents of the json file to see if there's any clues as to what is causing the decode error?