Move from uwsgi to gunicorn (#735)

Co-authored-by: Tomek Chytry-Trzeciak <75318192+Repumba@users.noreply.github.com>

Author: psrok1

deploy/docker/Dockerfile

@@ -28,6 +28,7 @@ RUN mkdir -p /app/uploads/ && \
 
 ENV PYTHONPATH=/app
 ENV FLASK_APP=/app/mwdb/app.py
+# How many workers gunicorn should spawn by default
 WORKDIR /app
 
 CMD ["/app/start.sh"]

docker-compose-dev-karton.yml

@@ -26,8 +26,7 @@ services:
       # NOTE: use gen_vars.sh in order to generate this file
       - mwdb-vars.env
     environment:
-      UWSGI_PY_AUTORELOAD: 1
-      UWSGI_ENABLE_THREADS: 1
+      HOT_RELOAD: 1
       MWDB_MAIL_SMTP: "mailhog:1025"
       MWDB_MAIL_FROM: "noreply@mwdb.dev"
       MWDB_RECAPTCHA_SITE_KEY: "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI"

docker-compose-dev-remote.yml

@@ -25,8 +25,7 @@ services:
       # NOTE: use gen_vars.sh in order to generate this file
       - mwdb-vars.env
     environment:
-      UWSGI_PY_AUTORELOAD: 1
-      UWSGI_ENABLE_THREADS: 1
+      HOT_RELOAD: 1
       MWDB_MAIL_SMTP: "mailhog:1025"
       MWDB_MAIL_FROM: "noreply@mwdb.dev"
       MWDB_RECAPTCHA_SITE_KEY: "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI"

docker-compose-dev.yml

@@ -25,8 +25,7 @@ services:
       # NOTE: use gen_vars.sh in order to generate this file
       - mwdb-vars.env
     environment:
-      UWSGI_PY_AUTORELOAD: 1
-      UWSGI_ENABLE_THREADS: 1
+      HOT_RELOAD: 1
       MWDB_MAIL_SMTP: "mailhog:1025"
       MWDB_MAIL_FROM: "noreply@mwdb.dev"
       MWDB_RECAPTCHA_SITE_KEY: "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI"

docker-compose-e2e.yml

@@ -23,7 +23,6 @@ services:
       MWDB_BASE_URL: http://127.0.0.1
       MWDB_ENABLE_RATE_LIMIT: 0
       MWDB_ENABLE_REGISTRATION: 1
-      UWSGI_PROCESSES: 4
       MWDB_MAIL_SMTP: "mailhog:1025"
       MWDB_MAIL_FROM: "noreply@mwdb.dev"
     volumes:

docker-compose-oidc-dev.yml

@@ -25,8 +25,7 @@ services:
       # NOTE: use gen_vars.sh in order to generate this file
       - mwdb-vars.env
     environment:
-      UWSGI_PY_AUTORELOAD: 1
-      UWSGI_ENABLE_THREADS: 1
+      HOT_RELOAD: 1
       MWDB_MAIL_SMTP: "mailhog:1025"
       MWDB_MAIL_FROM: "noreply@mwdb.dev"
       MWDB_RECAPTCHA_SITE_KEY: "6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI"

docker/gunicorn.conf.py

@@ -0,0 +1,7 @@
+import os
+
+wsgi_app = "mwdb.app:app"
+bind = "0.0.0.0:8080"
+user = "nobody"
+reload = bool(int(os.getenv("HOT_RELOAD", "0")))
+workers = int(os.getenv("GUNICORN_WORKERS", "4"))

docker/start.sh

@@ -7,4 +7,4 @@ until psql "$MWDB_POSTGRES_URI" -c "\q" ; do
 done
 
 echo "Configuring mwdb-core instance"
-mwdb-core configure --quiet basic && exec uwsgi --ini /app/uwsgi.ini
+mwdb-core configure --quiet basic && exec gunicorn

docker/uwsgi.ini

@@ -1,16 +1,48 @@
 Setup and configuration
 =======================
 
-Prerequisites
--------------
+Installation and configuration with Docker Compose
+--------------------------------------------------
+
+The quickest way setup MWDB is to just clone the repository and use Docker-Compose with all batteries included.
+
+.. code-block:: console
+
+    $ git clone https://github.com/CERT-Polska/mwdb-core.git
+
+After cloning repository, the first step is to go to the ``mwdb-core`` directory and generate configuration using ``./gen_vars.sh`` script.
+Generated variables can be found in mwdb-vars.env.
+
+.. code-block:: console
+
+   $ ./gen_vars.sh 
+   Credentials for initial mwdb account:
+
+   -----------------------------------------
+   Admin login: admin
+   Admin password: la/Z7MsmKA3UxW8Psrk1Opap
+   -----------------------------------------
+
+   Please be aware that initial account will be only set up on the first run. If you already have a database with at least one user, then this setting will be ignored for security reasons. You can always create an admin account manually by executing a command. See "flask create_admin --help" for reference.
+
+Then build images via ``docker-compose build`` and run MWDB via ``docker-compose up -d``.
+
+Your MWDB instance will be available on default HTTP port (80): http://127.0.0.1/
+
+If you want to use Docker Compose for MWDB development, check out :ref:`Developer guide`.
+
+Standalone installation
+-----------------------
+
+Step 1.: Prerequisites
+~~~~~~~~~~~~~~~~~~~~~~
 
 MWDB was tested on Debian-based systems, but should work as well on other Linux distributions.
 
 For production environments, you need to install:
 
 
 * **PostgreSQL database** (minimum supported version: 12, https://www.postgresql.org/download/linux/debian/)
-* **python-ssdeep library dependencies for Python 3** (https://python-ssdeep.readthedocs.io/en/latest/installation.html#id9) 
 
 Optionally you can install:
 
@@ -41,8 +73,8 @@ It's highly recommended to create a fresh `virtualenv <https://docs.python.org/3
 
    The connection string is: ``postgresql://mwdb:mwdb@127.0.0.1:54322/mwdb``
 
-Installation & Configuration
-----------------------------
+Step 2.: Installation and configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The recommended installation method is pip:
 
@@ -87,14 +119,14 @@ Then, use ``mwdb-core configure`` to provide first configuration for your MWDB s
    3) Current directory
    : 3
 
-For first installation we recommend to install everything in current folder via ``3`` option. If you want to install MWDB system-wide or locally for user: choose ``1`` or ``2``. 
+For first installation we recommend to install everything in current folder via ``3`` option. If you want to install MWDB system-wide or locally for user: choose ``1`` or ``2``.
 
 Then, input the connection string for PostgreSQL database. The database must be online and reachable at the time of configuration. After that, you will be asked for path for uploads and instance base URL. If the default value is ok, press Enter:
 
 .. code-block::
 
    PostgreSQL database connection string [postgresql://localhost/mwdb]:
-   Uploads storage path [./uploads]: 
+   Uploads storage path [./uploads]:
    Base public URL of Malwarecage service [http://127.0.0.1]:
 
 Depending on the installation type, your configuration will be stored in ``mwdb.ini`` file and can be changed any time you want:
@@ -136,42 +168,30 @@ And you are done! ``run`` command will start the Flask server:
 
 Your MWDB instance will be available on port 5000 (use ``--port`` to change that): http://127.0.0.1:5000/
 
-.. warning::
-
-   Remember to run ``mwdb-core configure`` after each version upgrade to apply database migrations
-
-
-Alternative setup with Docker Compose
---------------------------------------
+Keep in mind that Flask server is meant to be used as development server and **is not suitable for production**.
+See also: https://flask.palletsprojects.com/en/2.2.x/server/
 
-The quickest way setup MWDB is to just clone the repository and use Docker-Compose. We recommend this method **only for testing** because it can be a bit more difficult to install extensions and integrate with other services.
-
-.. code-block:: console
-
-    $ git clone https://github.com/CERT-Polska/mwdb-core.git
-
-After cloning repository, the first step is to go to the ``mwdb-core`` directory and generate configuration using ``./gen_vars.sh`` script.
+.. warning::
 
-.. code-block:: console
+   In standalone setup, remember to run ``mwdb-core configure`` after each version upgrade to apply database migrations.
 
-   $ ./gen_vars.sh 
-   Credentials for initial mwdb account:
+Step 3.: Setting up gunicorn and nginx
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-   -----------------------------------------
-   Admin login: admin
-   Admin password: la/Z7MsmKA3UxW8Psrk1Opap
-   -----------------------------------------
+It's recommended to deploy Flask applications using dedicated WSGI server. We highly recommend Gunicorn as it's used
+in our Docker images and combine it with Nginx serving as proxy server for best security and performance
 
-   Please be aware that initial account will be only set up on the first run. If you already have a database with at least one user, then this setting will be ignored for security reasons. You can always create an admin account manually by executing a command. See "flask create_admin --help" for reference.
+.. seealso::
 
-Then build images via ``docker-compose build`` and run MWDB via ``docker-compose up -d``.
+    https://flask.palletsprojects.com/en/2.2.x/deploying/gunicorn/
 
-Your MWDB instance will be available on default HTTP port (80): http://127.0.0.1/
+    https://docs.gunicorn.org/en/latest/deploy.html#deploying-gunicorn
 
-If you want to use Docker Compose for MWDB development, check out :ref:`Developer guide`.
+Proper configuration files and templates used in our Docker images can be found in `docker directory on our Github repository
+<https://github.com/CERT-Polska/mwdb-core/tree/master/docker>`_
 
-Upgrade mwdb-core to latest version
------------------------------------
+Upgrading mwdb-core to latest version
+-------------------------------------
 
 For standalone installation (pip-based), upgrade mwdb-core package to the latest version.