Arm version

[123billybob321]

Hi. Are there any plans of an Arm version of the dockers if it's possible?

Apologies if this has been asked before.

[biolds]

Hi. There is no plan yet for na Arm version as I don't have the hardware to test it. Though I guess it should be not too hard to build since it uses debian as a base. I can help if you encounter errors.

[123billybob321]

Thanks for the help. If I try to run docker compose as normal, this is what I get

.0s st platform (linux/arm64/v8) and no specific platform was requested 0.0s
� Volume "sosse_sosse_data" Created0.0s
� Container sosse_db Started0.5s
� Container sosse_app Started0.4s
! sosse The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Although the containers start, if I go to the site I get a error of

502 Bad Gateway
nginx/1.22.1

[biolds]

I've moved the issue to a Discussion.
This is weird it's starting at all :/
I think a rebuild is required, the most straight-forward way to check if it's feasible:

• update the architecture (https://stackoverflow.com/questions/47809904/how-to-set-architecture-for-docker-build-to-arm64), so chang the FROM line in docker/pip-base/Dockerfile to:

FROM --platform=linux/arm64 debian:bookworm

• then from the root directory, run make docker_git_build

This should build a stand-alone biolds/sosse:git image that runs the git version of Sosse. (building the official stable release may be a bit more complicated since it pulls the Python package of Sosse which may not be arm compatible either, trying with git is probably safer)

[123billybob321]

Thanks for your help.

I followed your guide and it gets so far before failing with this

[16/25] RUN /venv/bin/pip install ./ && /venv/bin/pip install uwsgi && /venv/bin/pip cache purge:
1.751 Processing /root/sosse
1.754 Installing build dependencies: started
14.48 Installing build dependencies: finished with status 'done'
14.48 Getting requirements to build wheel: started
14.79 Getting requirements to build wheel: finished with status 'error'
14.80 error: subprocess-exited-with-error
14.80
14.80 × Getting requirements to build wheel did not run successfully.
14.80 │ exit code: 1
14.80 ╰─> [9 lines of output]
14.80 running egg_info
14.80 creating Sosse.egg-info
14.80 writing Sosse.egg-info/PKG-INFO
14.80 writing dependency_links to Sosse.egg-info/dependency_links.txt
14.80 writing entry points to Sosse.egg-info/entry_points.txt
14.80 writing requirements to Sosse.egg-info/requires.txt
14.80 writing top-level names to Sosse.egg-info/top_level.txt
14.80 writing manifest file 'Sosse.egg-info/SOURCES.txt'
14.80 error: package directory 'se/deps/linkpreview/linkpreview' does not exist
14.80 [end of output]
14.80
14.80 note: This error originates from a subprocess, and is likely not a problem with pip.
14.80 error: subprocess-exited-with-error
14.80
14.80 × Getting requirements to build wheel did not run successfully.
14.80 │ exit code: 1
14.80 ╰─> See above for output.

ERROR: failed to solve: process "/bin/sh -c /venv/bin/pip install ./ && /venv/bin/pip install uwsgi && /venv/bin/pip cache purge" did not complete successfully: exit code: 1
make: *** [Makefile:91: docker_git_build] Error 1

[123billybob321]

I've fixed the above as I need to use the git clone to get the other submodules.
The file then builds and I think I'm running the new image. The log in page now loads and I get server error 500 when logging in.

Looking in the docker logs it says the connection to the dB has failed password authentication for user 'sosse_user'.
Also a message about could not open certificate file /root/.postgresql/postgresql.crt permission denied.

[biolds]

This is weird the default database user should be "sosse", it's created there:

sosse/Dockerfile

Line 31 in c32f7e9

psql --command "CREATE USER sosse WITH PASSWORD 'sosse';" && \

[123billybob321]

Got it working.

The docker compose file has sosse_user and sosse_db and sosse_password in it.

Changed all these to sosse and restarted and I've been able to log in

[123billybob321]

So the thing I cannot get to work now is persistent data for the database.
I have the volume and location defined as in the docker compose file supplied, but no data ever appears on the host.
The volume folders are created and it does work for the app too.

[biolds]

I think it's still using the local database. The docker image you built forces the database to localhost (see docker/pg_run.sh).
You can prevent this by modifying the Dockerfile at the bottom:

diff --git a/Dockerfile b/Dockerfile
index 380b3f5..08c63b0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -34,4 +34,4 @@ RUN /etc/init.d/postgresql start && \
     tar -c -p -C / -f /tmp/postgres_sosse.tar.gz /var/lib/postgresql
 
 USER root
-CMD ["/usr/bin/bash", "/pg_run.sh"]
+CMD ["/usr/bin/bash", "/run.sh"]

[123billybob321]

Thanks for your help again. When I've changed the file how do I create the image again?

I tried make docker_git_build and it seems to finish really quickly so I'm not sure if I've done it correctly?

Sorry if this is a really basic question, but this is btw for me 😄

[biolds]

No worries! Yes this is normal, since only the last line of the Dockerfile is modified, Docker uses a cache for the previous step and just rebuilds the last step.

[123billybob321]

Ok so I think that's done it. I've tried starting up the docker container, adding stuff and then removing the container. If I start it up again, it seems to keep the config.
I'll do a bit more testing when I can.

[123billybob321]

I'm still testing but it does look like the data is now persistent.

A couple of things I've noticed.

If I try to delete a large number of documents, for example a page of 100, after a few times it causes the entire machine to crash. Assuming it's too much for the Pi.

The other thing is when I crawl websites and then look in the archive, pictures are missing. I'm guessing this is my settings as they seem to be hosted externally so I'm still trying to figure that out.

[biolds]

The crash is maybe caused by a memory shortage, if that's the case, you could remove them from command line with:

docker exec -ti <container_name> /venv/bin/sosse-admin delete_documents https://google.com/.*

(see https://sosse.readthedocs.io/en/stable/cli.html#cli-delete-documents)

Seems weird for the pictures, these are inline pictures in webpages and they don't show up ? or links to external images that are not crawled?
(it could be file size limitation define in /etc/sosse/sosse.conf, defined with these https://sosse.readthedocs.io/en/stable/config_file.html#conf-option-max-file-size)

[123billybob321]

So this seems to be working as far as I can tell.

[biolds]

Good to know !

[123billybob321]

Hi. This has been working great until a recent update. If I pull the latest commit and start the container I get the log in screen but then when logging in get
Server error (500)

I'm not sure which commit caused it, but if I roll back to commit eb25ead and recreate it works ok.

[123billybob321]

It looks like when it fails I see this on the logs that didn't seem to be there when it works

Traceback (most recent call last):
File "/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedColumn: column se_document.retries does not exist

[biolds]

Seems like the DB is at an earlier stage, try to revert to 2f5c653 then run steps 2 and 3.

[123billybob321]

Brilliant that has fixed it!

I can run it in a different way so not the development version but this is the one with your help I managed to get working on ARM.

Happy to try a different way though 😃

[biolds]

Awesome!
You can switch to the stable branch if you want to follow the official releases (just run the docker exec ... before switching, to make the DB compatible). But feel free to stay on the main branch, I'll change the way DB upgrades are handled to make it work whatever the upgrade path to avoid these kind of problems, just let me know if you catch any early bug :)

[123billybob321]

Apologies I didn't realise I wasn't in the stable branch. I'm not really sure how to swap. But, if it doesn't cause you any issues with me using this one, I'm happy to stay on it and let you know if I get any problems.

Thanks for your help.

[biolds]

No worries, that's a legitimate use-case. Thanks for testing the dev branch!

[biolds]

Arg, sorry, it seems like I missed some notifications.. I'll answer now.