Development Environment / Contribution Guidelines

[danielperna84]

Hi,

are there any guidelines on how to setup a development environment? In particular I was thinking I might be able to help a bit with a Web UI. But I'm not really into Docker, so it's hard for me to judge what is really needed to get a simple instance (without the individual hardware-support) of Viseron running where I can easily work with the relevant files without building the container images on every change. I gave it a shot to install the requirements in a venv in WSL, but there OpenCV was missing. And im hesitant to see for myself how far down the rabbit hole goes. 😳

So essentially I'm asking for something like a Bash-script which installs the base requirements on a blank Linux (this could also be the starting point to set up Viseron within a Proxmox container), or possibly a VM image which already has everything installed (bare metal?) to get started quickly.

Maybe, if available, that could be part of the Wiki of this repository.

Thanks

[roflcoopter]

Hi!

I am glad you're asking! Would very much love some help on the UI side of things.
My plan has been to use tornado as a backend and then build a frontend in React, is that something you know?
Or what did you have in mind?

I have recently moved, thats why there havent been any new releases as of late.
However I did do some work on development guidelines some time ago, it is all present on the dev branch currently.
Is that something you have seen? Link

I always run Viseron in docker during development since there are a lot of dependencies as you said, which means frequent rebuilding of the container but that is pretty much instantaneous with the Docker caching.

If you want you could have a look at the link above and please let me know if something could be done better!

[roflcoopter]

Correct, the images are all based on Ubuntu, but the underlying OS that runs the container doesnt really matter i think.

There is a very high chance that i have overlooked something when it comes to Windows support however, so i can almost guarantee there will be some issues there 🤣
But with WSL its a different story, that can possibly work

[danielperna84]

Ok, then a Ubuntu VM it will be. 👍

[roflcoopter]

Sorry i totally missed what you wrote about the UI!

But yes that sounds great, a proper API will be needed whichever way we go.
I've looked around for some inspiration and unamnic could pose as an example.

To specify what I have in mind what this initial UI would provide:

• Dashboard with a list of the available cameras (possibly with some config details)
• With links to view the MJPEG stream directly
• A list of recordings with the option to download or delete the data
• Maybe, depending on the format, with an embedded video player
• Maybe a simple textarea to modify the configuration
• Oh, and authentication (for the UI and streams, maybe using JWTs as well)

Being able to view the recordings is probably the most needed feature in my opinion.
Having an embedded video player shouldnt be too difficult to achieve i think, i run a small web application now using videojs which works great

[danielperna84]

Regarding unamnic: I assume this would be required to convert the saved videos to a format that browsers can display properly? Which of course can be done. But wouldn't in make sense then to directly store videos in a compatible format? In my experience Chrome as well as Firefox have no problems with webp, which for ffmpeg is vp8 or vp9. I don't know however how / if hardware acceleration can be used for those. But I don't see a reason why not.

Since I don't have it running yet I don't know what data I'll have to work with. I assume there's no DB which stores everything. So my initial approach would be to just look at the file attributes for creation date etc. to get some metadata. In the long run that would be inefficient though. But one step at a time. I don't want to make architectual decision without having ever seen anything about this. 😅

Right now I have my Ubuntu set up and cloned my fork. I'm failing at the tox -e pylint though with the error ERROR: No pyproject.toml or setup.py file found.. Do you have such files in your local environment and they are just missing from the repo?

BTW: I'm taking notes of everything I'm doing to get this working. So I might be able to provide some steps which others then can follow.

Edit:
Or are those basic tox tests run inside of Docker as well?

[roflcoopter]

Regarding unamnic: I assume this would be required to convert the saved videos to a format that browsers can display properly? Which of course can be done. But wouldn't in make sense then to directly store videos in a compatible format? In my experience Chrome as well as Firefox have no problems with webp, which for ffmpeg is vp8 or vp9. I don't know however how / if hardware acceleration can be used for those. But I don't see a reason why not.

No unmanic was just an example of an app which uses a tornado API. We can create whatever file encoding thats necessarry from Viseron directly since its using ffmpeg.

Since I don't have it running yet I don't know what data I'll have to work with. I assume there's no DB which stores everything. So my initial approach would be to just look at the file attributes for creation date etc. to get some metadata. In the long run that would be inefficient though. But one step at a time. I don't want to make architectual decision without having ever seen anything about this. 😅

Correct, no database. Need some way to store data on events also that goes along with the recordings.

Right now I have my Ubuntu set up and cloned my fork. I'm failing at the tox -e pylint though with the error ERROR: No pyproject.toml or setup.py file found.. Do you have such files in your local environment and they are just missing from the repo?

BTW: I'm taking notes of everything I'm doing to get this working. So I might be able to provide some steps which others then can follow.

Edit:
Or are those basic tox tests run inside of Docker as well?

Yes the tests are run in Docker also. Running tox builds the containers and runs them automatically.

Regarding the setup.py, I did not know that was required for tox to work. I do have one locally which i created a long time ago but never got around to using it.
I dont have time right now to put it on GitHub but here is what it looks like if you wanna create it temporarily!

from setuptools import find_packages, setup

setup(
    name="viseron",
    packages=find_packages(),
    version="0.1.0",
    description="Viseron",
    author="Me",
    license="MIT",
)

[danielperna84]

Just a heads up on my current state:
Within your contribution-docs I'm at the state of building the individual containers. However 2 of the 6 fail.

1. amd64-ffmpeg:

---> Running in 101c34459126
freetype-2.5.5.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
The command '/bin/sh -c DIR=/tmp/freetype &&   mkdir -p ${DIR} &&   cd ${DIR} &&   curl -sLO https://download.savannah.gnu.org/releases/freetype/freetype-${FREETYPE_VERSION}.tar.gz &&   echo ${FREETYPE_SHA256SUM} | sha256sum --check &&   tar -zx --strip-components=1 -f freetype-${FREETYPE_VERSION}.tar.gz &&   ./configure --prefix="${PREFIX}" --disable-static --enable-shared &&   make &&   make install &&   rm -rf ${DIR}' returned a non-zero code: 1
ERROR: Service 'amd64-ffmpeg' failed to build : Build failed

2. amd64-viseron:

E: Version '15.0' for 'libedgetpu1-max' was not found
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends   gnupg   tzdata   python3   python3-pip   python3-sklearn   usbutils   libgomp1   libopenblas-base   && rm -rf /var/lib/apt/lists/*     && pip3 install /wheels/*.whl   && rm -r /wheels     && APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn apt-key adv --fetch-keys https://packages.cloud.google.com/apt/doc/apt-key.gpg   && echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" > /etc/apt/sources.list.d/coral-edgetpu.list   && echo "libedgetpu1-max libedgetpu/accepted-eula select true" | debconf-set-selections   && apt-get -qq update && apt-get -qq install --no-install-recommends -y   libedgetpu1-max=15.0     && chmod +x /tmp/s6-overlay-installer && /tmp/s6-overlay-installer /   && apt-get autoremove -y   && apt-get clean -y   && rm -rf /var/lib/apt/lists/*   && rm -rf /wheels   && rm -r /tmp/s6-overlay-installer   && useradd --uid 911 --user-group --create-home abc   && mkdir -p /home/abc/bin /segments' returned a non-zero code: 100
ERROR: Service 'amd64-viseron' failed to build : Build failed

I'm still figuring this all out, so I might have done something wrong. I assume the last part may fail because of the first failing build.

In any case I now first start with building a dummy Tornado-App to have some boilerplate to work with. But while we're at it: what would be the command to fire up the self-built viseron-container?

[roflcoopter]

Building all the containers will take a lot of time, especially the ffmpeg and OpenCV ones.
You dont really have to do that, since you are only modifying the Python code which is pulled into the last image, amd64-viseron.
The other images will be pulled down from DockerHub.
So you can stick with just running docker-compose --file ./azure-pipelines/docker-compose-build.yaml --env ./azure-pipelines/.env build amd64-viseron

If you need to add some extra requirements to requirements.txt you need to build the amd64-wheels image first however.

Working on fixing that build error, Google has a habit of breaking the EdgeTPU library installations at times...

In any case I now first start with building a dummy Tornado-App to have some boilerplate to work with. But while we're at it: what would be the command to fire up the self-built viseron-container?

You simply run it like normal with a command like this

docker run --rm \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
--name viseron \
roflcoopter/amd64-viseron:latest

[roflcoopter]

Just pushed to dev where the build error in amd64-viseron is fixed.

Will look at the ffmpeg error later!

[danielperna84]

The build has worked and I could start the container with motion detection and recording. I think so far that's all I need to get started. Thanks a lot. 👍

I can't give an exact ETA on when I can provide my results. I'm doing some renovation here. But I think within the next 2 weeks I can do some work in the evening hours. I'll get back at you then.

[danielperna84]

Oh, one more question out of curiosity: is it possible to restart / reload viseron from within the container? The reson I'm asking is my idea of modifying the config.yaml via the UI. To apply the new settings they obviously have to be reloaded. I don't feel that should require the whole container to be restarted. So some sort of viseron.reload() function could be useful. Haven't checked if such a function already exists.

Thinking out loud further along, reloading individual cameras and their settings would make sense as well if one day everything should be configurable via the UI. To clarify: If settings for camera A have been changed but those for camera B have not, then the recording / *-detection should not be interrupted by a reload. That part could be tricky though, depending on how the reload is implemented. For a start a global reload would be fine I guess.

[roflcoopter]

Reloading is not implemented as of now but it should be easy pretty easy to get that going.
We can probably also just kill the Viseron process in the container and it will automatically restart because of the s6-overlay that is used.
Could be an interim solution

[danielperna84]

I agree with the interim solution. This still can be improved later. 👍