Merge pull request #53 from omics-datascience/2.1.4

2.1.4

Author: Genarito

.github/workflows/prod-env-wf.yml

@@ -2,7 +2,7 @@ name: Prod enviroment workflow to build and push docker image
 on:
   push:
     branches:
-      - "prod"
+      - "main"
 jobs:
   docker-modulector:
     runs-on: ubuntu-latest

.github/workflows/prod-pr-wf.yml

@@ -2,7 +2,7 @@ name: Check if version exist on docker registry
 on:
   pull_request:
     branches:
-      - "prod"
+      - "main"
 jobs:
   version-check-repo:
     runs-on: ubuntu-latest

.gitignore

@@ -149,3 +149,4 @@ modulector/files/EPIC-8v2-0_A1.csv
 modulector/files/mirDIP_Unidirectional_search_v.5.txt
 *.sql.gz
 modulector/files/tmp_db.csv
+docker-compose.mauri_dev.yml

CONTRIBUTING.md

@@ -47,19 +47,19 @@ The entire contributing process consists in the following steps:
 
 ## Workflow
 
-We use gitlab environment git workflow. The default branch is `dev` and the publishing branch is `prod`. The working branches are created from `dev` and must respect the following steps and actions:
+We use gitlab environment git workflow. The default branch is `dev` and the publishing branch is `main`. The working branches are created from `dev` and must respect the following steps and actions:
 
 1. A new branch is created from `dev`.
 1. After finish working with it, a PR to `dev` must be created.
 1. Automatic Action/Workflow for PR is executed.
 1. The new branch is merged to `dev`.
 1. Automatic Action/Workflow for _Push_ events into `dev` is executed.
-1. When is ready to publish a new version of `dev`, a PR to `prod` is created.
+1. When is ready to publish a new version of `dev`, a PR to `main` is created.
 1. These Action/Workflow are executed:
     1. PR.
     1. Version checker (to avoid overwrite an existing image on Docker Hub repository).
-1. `dev` is merged into `prod`.
-1. Automatic Action/Workflow for _Push_ events into `prod` is executed to build a new Docker image for Modulector and publish it.
+1. `dev` is merged into `main`.
+1. Automatic Action/Workflow for _Push_ events into `main` is executed to build a new Docker image for Modulector and publish it.
 
 
 [**More information**](https://docs.google.com/presentation/d/1c1PXM89HLXJyF-zHAEpW_bcxb0iE_Fv2XEpEXYV2Tj4/edit?usp=sharing)

DEPLOYING.md

@@ -2,27 +2,27 @@
 
 Below are the steps to perform a production deploy.
 
-
 ## Requirements
 
 1. The entire deploy was configured to be simple from the tool Docker Compose. So you need to install:
     - [Docker](https://docs.docker.com/desktop/#download-and-install)
     - [Docker Compose](https://docs.docker.com/compose/install/)
 
-
 ## Instructions
 
 1. Create MongoDB Docker volumes:
+
     ```bash
     docker volume create --name=modulector_postgres_data
     ```
+
 1. Make a copy of `docker-compose_dist.yml` with the name `docker-compose.yml`.
 1. Set the environment variables that are empty with data. They are listed below by category:
     - Django:
         - `DJANGO_SETTINGS_MODULE`: indicates the `settings.py` file to read. In production, we set in `docker-compose_dist.yml` the value `ModulectorBackend.settings_prod` which contains several production properties.
         - `ALLOWED_HOSTS`: list of allowed host separated by commas. Default `['web', '.localhost', '127.0.0.1', '[::1]']`.
         - `ENABLE_SECURITY`: set the string `true` to enable Django's security mechanisms. In addition to this parameter, to have a secure site you must configure the HTTPS server, for more information on the latter see the section [Enable SSL/HTTPS](#enable-sslhttps). Default `false`.
-        - `CSRF_TRUSTED_ORIGINS`: in Django >= 4.x, it's mandatory to define this in production when you are using Daphne through NGINX. The value is a single host or list of hosts separated by commas. 'http://', 'https://' prefixes are mandatory. Examples of values: 'http://127.0.0.1', 'http://127.0.0.1,https://127.0.0.1:8000', etc. You can read more [here][csrf-trusted-issue].
+        - `CSRF_TRUSTED_ORIGINS`: in Django >= 4.x, it's mandatory to define this in production when you are using Daphne through NGINX. The value is a single host or list of hosts separated by commas. 'http://', 'https://' prefixes are mandatory. Examples of values: '<http://127.0.0.1>', '<http://127.0.0.1,https://127.0.0.1:8000>', etc. You can read more [here][csrf-trusted-issue].
         - `SECRET_KEY`: Django's secret key. If not specified, one is generated with [generate-secret-key application](https://github.com/MickaelBergem/django-generate-secret-key) automatically.
         - `MEDIA_ROOT`: absolute path where will be stored the uploaded files. By default `<project root>/uploads`.
         - `MEDIA_URL`: URL of the `MEDIA_ROOT` folder. By default `<url>/media/`.
@@ -34,7 +34,7 @@ Below are the steps to perform a production deploy.
         - `POSTGRES_PORT` : Database server listen port. By default, the docker image uses `5432`.
         - `POSTGRES_DB` : Database name to be used. By default, the docker image uses `modulector`.
     - Health-checks and alerts:
-        - `HEALTH_URL` : indicates the url that will be requested on Docker health-checks. By default, it is http://localhost:8000/drugs/. The healthcheck makes a GET request on it. Any HTTP code value greater or equals than 400 is considered an error.
+        - `HEALTH_URL` : indicates the url that will be requested on Docker health-checks. By default, it is <http://localhost:8000/drugs/>. The healthcheck makes a GET request on it. Any HTTP code value greater or equals than 400 is considered an error.
         - `HEALTH_ALERT_URL` : if you want to receive an alert when health-checks failed, you can set this variable to a webhook endpoint that will receive a POST request and a JSON body with the field **content** that contains the fail message.
 1. Go back to the project's root folder and run the following commands:
     - Docker Compose:
@@ -49,62 +49,57 @@ Below are the steps to perform a production deploy.
     1. Run: `python3 manage.py createsuperuser`
     1. Exit the container: `exit`
 
-
 ### Start delays
 
 Due to the database restoration in the first start, the container `db_modulector` may take a while to be up a ready. We can follow the status of the startup process in the logs by doing: `docker compose logs --follow`.
 Sometimes this delay makes django server throws database connection errors. If it is still down and not automatically fixed when database is finally up, we can restart the services by doing: `docker compose up -d`.
 
-
 ## Enable SSL/HTTPS
 
 To enable HTTPS, follow the steps below:
 
 1. Set the `ENABLE_SECURITY` parameter to `true` as explained in the [Instructions](#instructions) section.
-1. Copy the file `config/nginx/multiomics_intermediate_safe_dist.conf` and paste it into `config/nginx/conf.d/` with the name `multiomics_intermediate.conf`.
-1. Get the `.crt` and `.pem` files for both the certificate and the private key and put them in the `config/nginx/certificates` folder.
-1. Edit the `multiomics_intermediate.conf` file that we pasted in point 2. Uncomment the lines where both `.crt` and `.pem` files must be specified.
-1. Edit the `docker-compose.yml` file so that the `nginx` service exposes both port 8000 and 443. Also, you need to add `certificates` folder to `volumes` section. It should look something like this:
-
-```yaml
-...
-nginx:
-    image: nginx:1.23.3
-    ports:
-        - 80:8000
-        - 443:443
-    # ...
-    volumes:
-        ...
-        - ./config/nginx/certificates:/etc/nginx/certificates
-...
-```
+2. Copy the file `config/nginx/multiomics_intermediate_safe_dist.conf` and paste it into `config/nginx/conf.d/` with the name `multiomics_intermediate.conf`.
+3. Get the `.crt` and `.pem` files for both the certificate and the private key and put them in the `config/nginx/certificates` folder.
+4. Edit the `multiomics_intermediate.conf` file that we pasted in point 2. Uncomment the lines where both `.crt` and `.pem` files must be specified.
+5. Edit the `docker-compose.yml` file so that the `nginx` service exposes both port 8000 and 443. Also, you need to add `certificates` folder to `volumes` section. It should look something like this:
+
+    ```yaml
+    ...
+    nginx:
+        image: nginx:1.23.3
+        ports:
+            - 80:8000
+            - 443:443
+        # ...
+        volumes:
+            ...
+            - ./config/nginx/certificates:/etc/nginx/certificates
+    ...
+    ```
 
 6. Redo the deployment with Docker.
 
-
 ## Perform security checks
 
 Django provides in its official documentation a configuration checklist that must be present in the production file `settings_prod.py`. To verify that everything is fulfilled, you could execute the following command **once the server is up (this is because several environment variables are required that are set in the `docker-compose.yml`)**.
 
-```
+```bash
 docker container exec modulector_backend python3 manage.py check --deploy --settings ModulectorBackend.settings_prod
 ```
 
 Otherwise, you could set all the mandatory variables found in `settings_prod.py` and run directly without the need to pick up any service:
 
-```
+```bash
 python3 manage.py check --deploy --settings ModulectorBackend.settings_prod
 ```
 
-
 ## Restart/stop the services
 
 If the configuration of the `docker-compose.yml` file has been changed, you can apply the changes without stopping the services, just running the `docker compose restart` command.
 
 If you want to stop all services, you can execute the command `docker compose down`.
 
-
 ## See container status
 
 To check the different services' status you can run:
@@ -113,7 +108,6 @@ To check the different services' status you can run:
 
 Where  *\<service's name\>* could be `nginx_modulector`, `web_modulector` or `db_modulector`.
 
-
 ## Creating Dumps and Restoring from Dumps
 
 ### Export
@@ -124,23 +118,20 @@ In order to create a database dump you can execute the following command:
 
 That command will create a compressed file with the database dump inside.
 
-
 ### Import
 
 You can use set Modulector DB in two ways.
 
-
 ### Importing an existing database dump (recommended)
 
-1. Start up a PostgreSQL service. You can use the same service listed in the `docker-compose.dev.yml` file. Run `docker compose -f docker-compose.dev.yml up -d db_modulector` to start the DB service. 
+1. Start up a PostgreSQL service. You can use the same service listed in the `docker-compose.dev.yml` file. Run `docker compose -f docker-compose.dev.yml up -d db_modulector` to start the DB service.
 1. **Optional but recommended (you can omit these steps if it's the first time you are deploying Modulector)**: due to major changes, it's probably that an import thrown several errors when importing. To prevent that you could do the following steps before doing the importation:
     1. Drop all the tables from the DB: `docker exec -i [name of the DB container] psql postgres -U postgres -c "DROP DATABASE modulector;"`
     1. Create an empty database: `docker exec -i [name of the DB container] psql postgres -U postgres -c "CREATE DATABASE modulector;"`
 1. Download `.sql.gz` from [Modulector releases pages](https://github.com/omics-datascience/modulector/releases) or use your own export file.
 1. Restore the database: `zcat modulector.sql.gz | docker exec -i [name of the DB container] psql modulector -U modulector`. This command will restore the database using a compressed dump as source, **keep in mind that could take several minutes to finish the process**.
    - **NOTE**: in case you are working on Windows, the command must be executed from [Git Bash][git-bash] or WSL.
 
-
 ### Regenerating the data manually
 
 1. Download the files for the mirDIP database (version 5.2) and the Illumina 'Infinium MethylationEPIC 2.0' array. The files can be freely downloaded from their respective web pages.  
@@ -152,38 +143,35 @@ You can use set Modulector DB in two ways.
    **For the EPIC Methylation array**:
       - Go to the [Illumina product files web page](https://support.illumina.com/downloads/infinium-methylationepic-v2-0-product-files.html) and download the ZIP file called "*Infinium MethylationEPIC v2.0 Product Files (ZIP Format)*".
       - Unzip the file.
-      - Within the unzipped files you will find one called "*EPIC-8v2-0_A1.csv*". Move this file to the directory **"modulector/files/"**. 
+      - Within the unzipped files you will find one called "*EPIC-8v2-0_A1.csv*". Move this file to the directory **"modulector/files/"**.
       - **NOTE:** the total weight of both files is about 5 GB.
 
    **For the mirBase database**: this database is embedded as it weighs only a few MBs. Its data is processed in Django migrations during the execution of the `python3 manage.py migrate` command. So, you don't have to do manual steps to incorporate mirBase data inside Modulector.
-1. Start up a PostgreSQL service. You can use the same service listed in the _docker-compose.dev.yml_ file.
+1. Start up a PostgreSQL service. You can use the same service listed in the *docker-compose.dev.yml* file.
 1. Run `python3 manage.py migrate` to apply all the migrations (**NOTE:** this can take a long time to finish).
 
-
 ## Update databases
 
 Modulector currently works with the mirDIP (version 5.2) and miRBase (version 22.1) databases for miRNA data, and with information from the Illumina 'Infinium MethylationEPIC 2.0' array  for information about methylation sites.  
 If new versions are released for these databases, and you want to update them, follow these steps:  
 
- - For **mirDIP** and **Illumina EPIC array** you must follow the same steps described in the [Regenerating the data manually](#regenerating-the-data-manually) section, replacing the named files with the most recent versions that have been published on their sites .
- - For **miRBase**, follow the instructions below:
-   1. Go to the [_Download_ section on the website][mirbase-download-page].
-   1. Download the files named _hairpin.fa_ and _mature.fa_ from the latest version of the database.
-   1. Replace the files inside the _modulector/files/_ directory with the ones downloaded in the previous step.
-   1. Start up a PostgreSQL service. You can use the same service listed in the _docker-compose.dev.yml_ file.
+- For **mirDIP** and **Illumina EPIC array** you must follow the same steps described in the [Regenerating the data manually](#regenerating-the-data-manually) section, replacing the named files with the most recent versions that have been published on their sites .
+- For **miRBase**, follow the instructions below:
+   1. Go to the [*Download* section on the website][mirbase-download-page].
+   1. Download the files named *hairpin.fa* and *mature.fa* from the latest version of the database.
+   1. Replace the files inside the *modulector/files/* directory with the ones downloaded in the previous step.
+   1. Start up a PostgreSQL service. You can use the same service listed in the *docker-compose.dev.yml* file.
    1. Run the command `python3 manage.py migrate` to apply all the migrations (**NOTE:** this can take a long time to finish).
 
 **Note:** These updates will work correctly as long as they maintain the format of the data in the source files.
 
-
 ## Configure your API key
 
 When we notify user about updates of pubmeds they are subscribed to we interact with a ncbi api that uses an API_KEY, by default, we left a random API_KEY pre-configured in our settings file, you should replace it with your own.
 
-
 ## Cron job configuration
-For cron jobs we use the following [library](https://github.com/kraiz/django-crontab). In our settings file we configured our cron jobs inside the `CRONJOBS = []`
 
+For cron jobs we use the following [library](https://github.com/kraiz/django-crontab). In our settings file we configured our cron jobs inside the `CRONJOBS = []`
 
 [mirbase-download-page]: https://www.mirbase.org/ftp.shtml
 [csrf-trusted-issue]: https://docs.djangoproject.com/en/4.2/ref/csrf/

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.12-slim-bullseye
+FROM python:3.12-slim-bookworm
 
 # Default value for deploying with modulector DB image
 ENV POSTGRES_USERNAME "modulector"

ModulectorBackend/settings.py

@@ -13,7 +13,7 @@
 import os
 
 # Modulector version
-VERSION: str = '2.1.3'
+VERSION: str = '2.1.4'
 
 # Default primary key field type
 # https://docs.djangoproject.com/en/4.0/ref/settings/#default-auto-field
@@ -163,6 +163,9 @@
 MEDIA_ROOT = os.getenv('MEDIA_ROOT', os.path.join(BASE_DIR, ''))
 MEDIA_URL = os.getenv('MEDIA_URL', '/media/')
 
+# Test runner
+TEST_RUNNER = 'modulector.tests.runner.DjangoTestSuiteRunner'
+
 # Email Server
 EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
 # This email configuration is what postfix uses, for production, use your own