robot_view_be

robot attachment viewer backend

Run in development env

Install

pip3 install -r ./requirement.txt

Run

python3 manage.py runserver --settings=robot_view.dev_settings

or

./dev_robot.sh

Run in Production env

Please install the docker-compose before you use it.

docker-compose build && docker-compose up

then visit the localhost:8000

Resource

1. deployment via docker, nginx and uwsgi
2. 使用 CircleCI 实现持续集成和持续部署

Note

add a [README.md] to typing your note and problem

use docker in production env

use docker can make deploy more easy than before, and use the same system env can solve the cross-platform problems

1. create the Dockerfile file
2. Install the docker-compose command tools and create the docker-compose.yml file
3. run docker-compose build to build the web image and database image, use docker-compose run to run these containers

use travis-ci to deploy the application automatic

1. login travis-ci and add the current repos on github
2. create the .travis.yml in project
3. add the ssh key by this command, in order to deploy the code to server

travis encrypt-file ~/.ssh/id_rsa --add

split the settings file for dev env and production env

because I prefer using mysql database in the dev env, and use postgres in production env in docker container, so split the settings is very important. there are many blog can search form google, just use the simplest method.

1. create the dev_settings.py
2. import all item in settings.py
3. overwrite the item what you want to change.
4. run commands in manage.py with --settings=robot_view.dev_settings or other settings file, default setting file is settings.py in robot_view

move app to apps path

if app has very huge number, leave them in the project root path is not a smart choice, so create the apps path to store them

1. modify the settings.py, to configure the apps as resource path

import os
import sys

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(BASE_DIR, 'apps'))

2. if use pycharm, also can mark the apps path as Source Root, trust me, It's a smart action.

change pip registry to douban

pip3 install -r /code/robot/requirements.txt -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

if use docker, modify the Dockerfile

- RUN pip3 install -r /code/robot/requirements.txt
+ RUN pip3 install -r /code/robot/requirements.txt -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

resolve the staticfiles 404:

because Django will handle the request for the static file only when the DEBUG option in settings is True, if run in production env, developer must handle it by himself.

1. declare the STATIC_ROOT in settings.py, or your custom setting files
2. install whitenoise via pip and edit the wsgi.py in robot_view

pip install whitenoise

# export the dependencies in requirements.txt
pip freeze > ./requirements.txt

from whitenoise.django import DjangoWhiteNoise
from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()
application = DjangoWhiteNoise(application)

3. then run this command:

# collect the static file from the package like rest-framework
# to the STATIC_ROOT path where declare in settings file
python manage.py collectstatic

4. then rebuild the docker image and run it

# -d option can make the process in daemon mode.
docker-compose build && docker-compose up -d

and schema and docs

django rest framework already support the docs and schema itself, just include it and add a urlpatterns is enough:

from rest_framework.schemas import get_schema_view
from django.urls import path, include
from rest_framework.documentation import include_docs_urls
from rest_framework.routers import DefaultRouter

router = DefaultRouter()
schema_view = get_schema_view(title="Server Monitoring API")

urlpatterns = [
    path('', include(router.urls)),
    path('schema/', schema_view),
    path('docs/', include_docs_urls(title='doc', description='desc'))
  ]

Fix list is not callable

After configure the router for user app, in development env, app can work very will, when build docker container, app throw a error: list object is not callable

Solution is very easy: use the tuple, don't use list.

# robot_view/setting.py
REST_FRAMEWORK = {
     # Use Django's standard `django.contrib.auth` permissions,
     # or allow read-only access for unauthenticated users.
     'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        # 'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.BasicAuthentication',
    ),
     'PAGE_SIZE': 10,
    'DEFAULT_PAGINATION_CLASS': (
        'rest_framework.pagination.PageNumberPagination'
    )
}

Change authorization method to JWT

• edit setting.py, and the AUTHENTICATION_BACKENDS

  AUTHENTICATION_BACKENDS = (
      'users.views.CustomBackend',
      'django.contrib.auth.backends.ModelBackend'
  )

• post the username and password to http://127.0.0.1:8001/login/ to exchange the jwt

• and Authorization request header and Bearer prefix for jwt string

Create thrift server in Django app

for rpc, I choose to use apache thrift framework

• install django-thrift:

  pip install django-thrift

• configure django-thrift in setting.py

  • add 'django_thrift' in INSTALLED_APPS
  • add THRIFT configure option in setting.py
  • add FILE option in THRIFT point to *.thrift file
  • add SERVICE option named is the same to the thrift server name

• write the thrift handler in django app view:

  # import the create_handler
  from django_thrift.handler import create_handler
  
  # get a handler instantiation
  handler = create_handler()
  
  # defined the thrift method
  @handler.map_function("saveRobotData")
  def save_robot_data_handler():
      return {"a": "bb"}
  
  # more thrift methods can be defined

• management thrift server on localhost 9090

  # start rpc server
  python manage.py runrpcserver

Create extra_app folder to store the library which have to modify the source code

Because I need change the thrift server listen host and port, but django-thrift library can't support change these in setting.py, so I have to modify the source code of this library.

• create extra_app folder

  mkdir extra_app

• move the django-thrift library from site-package to extra-app:

  mv to_your_site_package_path/django-thrift extra_app

• add extra-app in PYTHONPATH via modify the setting.py

  import sys, os
  BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
  sys.path.insert(0, os.path.join(BASE_DIR, 'extra_apps'))

• modify the django-thrift source code what you want to edit.

save password as django command 'createsuperuser'

before I do these, only the user which create via django manage.py command createsuperuser can generate current JSON WEB Token. So I want to know why.

the user Profile data store in users_userporfile table, the password field which user is created by command is encrypted, so I need use the same methods to encrypt the password before save it in database.

search in django source code, I find the make_password function, and when use create superuser, the manage.py don't provide the salt, so just use the like base_user.py:

from django.contrib.auth.hashers import make_password
def validate(self, attrs):
    # because the limit from serializer, add 'raw_password' prop is forbidden.
    # attrs["raw_password"] = attrs["password"]
    attrs["password"] = make_password(attrs["password"])
    return attrs

use Q and rewrite retrieve method add prop on response

the minimum code implementation :

from django.db.models import Q
from django.contrib.auth import get_user_model
from rest_framework import viewsets, status, response
from rest_framework.mixins import  RetrieveModelMixin
User = get_user_model()

class ExampleViewSet(RetrieveModelMixin, viewsets.GenericViewSet):
    def retrieve(self, request, *args, **kwargs):
        queryset = self.get_object()
        serializer = self.get_serializer(queryset)
        re_dict = serializer.data
        re_dict["username"] = User.objects.get(Q(id=re_dict["user"])).username
        del re_dict["user"]
        headers = self.get_success_headers(serializer.data)
        return response.Response(re_dict, status=status.HTTP_200_OK, headers=headers)

use thrift application interface send data to sqlite database

List Length	time
100	2s
100,000	206s

nginx configuration

server {
  listen 80;
  server_name robot.raoul1996.cn;
  root /usr/share/nginx/html/robot;
  location / {
  } 
  location ~* ^/api|static/ {
    rewrite ^/api/(.*) /$1 break;
    proxy_pass http://your_ip:your_port;
  } 
}