-
Notifications
You must be signed in to change notification settings - Fork 32
Virtualenv Development Env
Under Construction
Starting the wiki with the raw text from the previous Getting Started files from the docs folder just to archive them before making any updates. I plan to restructure the flow so that the Windows and Mac/Linux instructions are aligned inline.
This is the implementation of the Grand Comics Database in Python using the Django framework.
For basic information, see the README file in the project's root directory.
The specifics of setting up a development environment on MacOS X or Linux is documented in Getting Started on MacOS or Linux The specifics of setting up a development environment on Windows is documented in Getting Started on Windows.
After you went through the steps return to here and follow the instruction to use a current dump of our database in the dev-environment.
In the top level directory of your repository clone, open up settings.py and
take a look at it. You should create a settings_local.py file in the same
directory, and override any values from settings.py that need overriding in
that settings_local.py. Do not modify settings.py itself, unless you need
to push a change out to all other development and production environments.
Our .gitignore settings will prevent git from noticing settings_local.py.
The settings.py file has numerous comments indicating what needs to be overridden.
The most obvious are the DATABASES and CACHES group of settings.
By default, we assume that the database is called gcdonline. It can be
accessed by a user called gcdonline with no password. If you set your
database up differently, look for settings that start with DATABASE. Override
them appropriately.
Create your database and set up whatever permissions you want. Django will create the tables for you, but you need to create the database first.
E.g. from MySQL command line client do (this is without a password):
create schema gcdonline;
create user gcdonline;
grant all on gcdonline.* to gcdonline;
To create the various tables run the migrations with
python manage.py migrate
This will create the tables, populate them with some of the required initial data, and update your schema if any updates beyond the initial state are required.
If you get system check errors for models.E025, add models.E025 to the
SILENCED_SYSTEM_CHECKS array in your settings_local.py to suppress the check:
SILENCED_SYSTEM_CHECKS = ['models.E025']
Currently it is needed to manually perform python manage.py loaddata on all
files in the fixtures folders of the django apps,
e.g. apps/indexer/fixtures/, apps/gcd/fixtures, etc.
python manage.py loaddata users
this will add an admin user, an approver, and a basic indexer. The usernames,
email addresses and passwords can be seen in the fixture file, which is
located at apps/indexer/fixtures/users.yaml. Load all the relevant fixture
data for the apps you expect to be using.
If you want data in your database, and know that the current development master matches the production schema, you can load a data dump from
http://www.comics.org/download/
Contact the GCD tech team to find out if the development and production schemas currently match, and what to do if they do not.
With the default user and database-name you need to do
mysql -ugcdonline gcdonline < dump.sql
(You might want to do -v to see something is happening)
Note that we do not distribute the images for covers and other user contributed uploads of scans.
For haystack search to work one needs to run elasticsearch search backend. The latest version can be downloaded here: http://www.elasticsearch.org/download/
Either use the provided deb/rpm or unpack it and run bin/elasticsearch -f (on Linux and Mac) or bin\elasticsearch.bat (on Windows).
If you get an error like "Unsupported major.minor version 51.0" and a stack trace, it means that you need to update your java runtime.
After that you can run python manage.py rebuild_index in gcd-django directory to populate your search indexes with data.
At this time you should be able to test your installation and data import using a read-only mode, or try out editing with read-write mode.
Read-write mode is the default.
For read-only mode, set READ_ONLY = True and NO_OI = True in settings_local.py
Once your settings are chosen, run
python manage.py runserver
and take a look at http://127.0.0.1:8000/
If you don't get any page at all, check http://www.djangoproject.com/ if it looks like Django isn't configured correctly, or http://dev.comics.org/ and/or http://groups.google.com/group/gcd-tech/ if it appears to be a code problem.
When you start making changes, go to the wiki at http://docs.comics.org/ for instructions on how to get code reviewed before checkin. You'll need to set up an account with our Review Board instance so that we can review your first pull request before deciding whether to accept it. If you develop a good record of contributions, you will be granted direct access to the project.
Submissions to the project-owned repository that do not first go through code review will be reverted without notice.
This is the implementation of the Grand Comics Database in Python using the Django framework.
For basic information, see the README file in the project's root directory.
This file explains the specifics of setting up a development environment on Windows. There are some differences between x32 and x64 installs at this time.
You'll need Python. We are currently using Python 3.8. http://www.python.org/download/ python.org distribution is preferred as ActiveState Python has some purchase 'enforcing' on x64 (see Note 1). ActiveState seems OK for x32 though.
In order to use precompiled libraries with Python, you must get Python 3.8 as the precompiled binaries must match the version of Python or they will not install.
These instructions have been tested with 64-bit Python 3.8.10 for Windows as of February 2023.
Direct link to 64-bit Python 3.8.10 for Windows
After installing Python, add these directories to the front of your PATH to ensure
that they are found first:
%LOCALAPPDATA%\Programs\Python\Python38
%LOCALAPPDATA%\Programs\Python\Python38\Scripts
-
Install git, which may be found through http://github.com/
-
Clone the repository. The GitHub web page is: https://github.com/GrandComicsDatabase/gcd-django See the GitHub help documentation for options, and contact the gcd-tech mailing list if you need assistance.
Install MySQL (version 5.5.x is currently in production) http://dev.mysql.com/downloads/
Put the following lines in your configuration (my.cnf) if not already there, these are nowadays the default on Windows installations.
default-character-set = utf8
default-storage-engine = InnoDB
These settings will save you from accidentally not using foreign keys or transactions, or from trying to stuff unicode into an 8-bit character set. None of those are fun to debug or recover from.
It can be useful to install the GUI MySQL Workbench.
CSSTidy is an executable required by django-compressor. If DEBUG=True, the default setting, django-compressor is not active, therefore CSSTidy is not needed if you are only working with DEBUG=True.
http://csstidy.sourceforge.net/
It just needs to be somewhere in the PATH. Using 32bit executable for 64bit seems fine.
Get it from http://pypi.python.org/pypi/setuptools#windows
32-bit version of Python Install setuptools using the provided .exe installer.
64-bit versions of Python: download ez_setup.py and run it; it will download the appropriate .egg file and install it for you. Currently, the provided .exe installer does not support 64-bit versions of Python for Windows, due to a distutils installer compatibility issue
We recommend using virtualenv, but it is optional. It will allow you to isolate the libraries you need for the GCD, and not contaminate your system python.
For more on installing and using virtualenv, see http://www.virtualenv.org/en/latest/index.html
The recommended way to use pip is within virtualenv, since every virtualenv
has pip installed in it automatically. So do
easy_install.exe virtualenv
If you do not want to use virtualenv install pip using
easy_install.exe pip.
Use pip to install virtualenv, and then create a virtualenv for use with the GCD. For this example, we will create one called "gcd" in a directory called "virtualenvs".
Now create a virtualenv
virtualenv.exe \path\to\env\gcd
and activate the virtualenv
\path\to\env\gcd\Scripts\activate
Note that "(gcd)" at the beginning of the prompt. Now all Python execution and installation will occur inside of the "gcd" virtual env, i.e. \path\to\env\gcd.
Inside the git-shell installed by github do
\path\to\env\gcd\Scripts\activate
Or make sure git is in your path if using a different shell and activate
the virtualenv. We assume nothing is installed in it.
Now go the the directory with the cloned git repo "gcd-django".
We use pip and our checked-in requirements.txt file to install all necessary Python
packages. You either downloaded the pre-compiled packages or compile them yourself.
Compiling them yourself is an advanced topic and not necessary; it is recommended
that you simply use the precompiled versions available.
You could also just run pip without the virtualenv to install everything into your system python's site-packages tree, but we recommend to use virtualenv.
Two of the python packages need compiling. This typically additional installs and on x64 does not work out of the box. The easy way is to use the Unofficial Windows Binaries for Python Extension Packages.
That page contains "wheel" packages for various Python libraries that require compilation.
A wheel package can be installed into your virtual environment using pip:
C:> \path\to\env\gcd\Scripts\activate
(gcd) C:\path\to\env\gcd> pip install package.whl
You must obtain a wheel package that is compiled for the specific version of Python
installed on your system. As directed above this is "C Python 3.8" so choose the wheel
packages with cp38 in their name as given below.
Download the Pillow wheel package Pillow-8.4.0-cp38-cp38-win_amd64.
Install the wheel package with pip.
Download the PyICU wheel package PyICU-2.4.3-cp38-cp38-win_amd64.
Install the wheel package with pip.
Now that you've satisifed the requirements for the two packages that require compilation, you can install the remaining packages from the requirements file:
pip install -r requirements.txt
# You can also just open up the requirements.txt and install each line separately.
# It is safe to re-run pip, since if everything installed properly, it will just
# check each library and skip it as already installed.
This should install Django and every other python package you need to run the site and post code reviews for the project.
Now return to the The GCD project and apps for information on how to setup the environment using a dump from the GCD project.
Ex: pypm install setuptools The following packages will be installed into "%APPDATA%Python" (2.7): distribute-0.6.27 error: Can't install distribute-0.6.27: requires Business Edition subscription *** If you have purchased ActivePython Business Edition, please login to *** your account at: *** https://account.activestate.com/ *** and download and run the license installer for your platform.
*** Please visit to learn more *** about the ActivePython Business Edition offering.
One needs to install scons 1.3.1 (later versions didn't work for me) http://www.scons.org/ Download zip, run setup.py install. Download CSStidy source distribution from: http://csstidy.sourceforge.net/ VC9 won't build it properly Fix is not too difficult though. Solution comes from: http://stackoverflow.com/questions/3075697/non-existent-member-function-specified-as-friend Edit umap.hpp Move class iterator definition block toward the first line of public: (of class umap) s
This is the implementation of the Grand Comics Database in Python using the Django framework.
For basic information, see the README file in the project's root directory.
This documentation is always somewhat outdated, in particular in view of version numbers. See aso the Docker setup.
After following the steps here go to Getting Started file for information on how to setup the environment using a dump from GCD project.
This file explains the specifics of setting up a development environment on MacOS X or Linux. Note that your Linux distribution's package management system may be able to handle much of the installation work for you.
This file assumes that you're cloning the "master" branch from github. If you want to run the current production branch, please inquire as to which branch is current at http://groups.google.com/group/gcd-tech/
To run you'll need Python. We are currently using Python 3.
-
Install git, which may be found through http://github.com/
-
Clone the repository. The GitHub web page is: https://github.com/GrandComicsDatabase/gcd-django See the GitHub help documentation for options, and contact the gcd-tech mailing list if you need assistance.
Install MySQL, we are currently using 5.7.
Put the following lines in your /etc/my.cnf
default-character-set = utf8
default-storage-engine = InnoDB
These settings will save you from accidentally not using foreign keys or transactions, or from trying to stuff unicode into an 8-bit character set. None of those are fun to debug or recover from.
Note that these options are not needed for mysql 8 (>=8.0.1), there these are the defaults and default-character-set is replaced character_set_server anyway.
Note that MySQL may not create an /etc/my.cnf during installation
(for instance on Mac OS X Mountain Lion). Just create one yourself with these
two lines in it.
If installing using Linux packages of some sort and also using pip/virtualenv for Python libraries as described below, you will also need the devel-packages, e.g.
- RPM:
mysqlclient-devel - Debian package:
libmysqlclient-dev
Install the ICU library, which is C/C++ and may require compilation. You will need this before you install the Python modules, as one of them depends on it. This is specifically icu4c, not the Java icu4j.
libicu builds fine on Mac OS X, and pre-built packages should exist for most linux distributions:
- RPM:
libicu - Debian package:
libicu48
If using pip/virtualenv to build Python libraries, you will also need the icu-devel RPM or the libicu48-dev Debian package (libicu-dev on newest Ubuntu).
If you do compile libicu yourself, the instructions on their readme.html are absurdly complex. Just do this (which worked on Mac OS X Snow Leopard):
bash:~$ tar xzvf icu-whatever.tgz # version 49 at the time of writing
bash:~$ cd icu/source
bash:~/icu/source$ ./configure
bash:~/icu/source$ gnumake
bash:~/icu/source$ sudo gnumake install
For the image library python-pillow some libraries can be installed. Needed are libjpeg-dev and zlib by default.
You will need some additional libraries for Pillow to be able to support some of the other common image formats. If you are not doing any image editing, this is not required.
See https://pillow.readthedocs.io/en/stable/installation.html#external-libraries for further information.
TODO: Document specifics.
Install CSSTidy, which should exist as a package for most Linux distributions (and possibly for MacPorts as well): http://csstidy.sourceforge.net/
- RPM: csstidy
- Debian package: csstidy
Install Python if it did not come with your system. Python 3 is recommended. http://www.python.org/downloads/
If installing from packages you'll also need development package for python (if it's separated, like python-dev on Ubuntu).
We recommend using virtualenv. It will you to isolate the libraries you need for the GCD, and not contaminate your system python.
Install the package python-virtualenv.
Create a virtualenv for use with the GCD. For this example, we will create one called "gcd" in a directory called "virtualenvs".
bash:~$ mkdir virtualenvs
bash:~$ cd virtualenvs
bash:~/virtualenvs$ virtualenv gcd
New python executable in gcd/bin/python
Installing setuptools............done.
Installing pip...............done.
bash:virtualenvs$ ls
gcd
bash:virtualenvs$ source gcd/bin/activate
(gcd)bash:virtualenvs$
Note that "(gcd)" at the beginning of the prompt. Now all Python execution
and installation will occur inside of the "gcd" virtual env, i.e. ~/virtualenvs/gcd.
For more on installing and using virtualenv, see http://www.virtualenv.org/en/latest/index.html
Assuming you've already activated the virtualenv (but not installed anything further in it), and cloned the git repo to ~/git/gcd-django, use pip and our checked-in requirements file to install all necessary Python libraries. Make certain that you have any C/C++ libraries already compiled and installed (such as libicu, and optionally some supplemental libraries for PIL depending on how much of the image editing you want to be able to do).
Debian packages, RPMs and MacPorts packages are available for many of the
Python libraries, and if you prefer to use those, please remember to
make a copy of requirements.txt and remove the libraries that you
installed through other means. If the OS packages were of older Python
library verions, you may need to add versions to your requirements.txt
or adjust the versions to match. See the pip documentation for the
requirements.txt format.
This file tries to install versions matching our production environment. Depending on your installation, you might need more updated version.
PLEASE NOTE: If the latest version of PyICU does not match the version of libicu that you installed, you may need to try an older version of PyICU, which should generally work fine with the GCD. For instance PyICU 1.0 was known to work with libicu 4.4 as of October 2012.
(gcd)bash:~/git/gcd-django$ pip install -r requirements.txt
# A ton of output happens here, especially for PIL compilation.
# You can also just open up the requirements. text and install each line separately.
# It is safe to re-run pip, since if everything installed properly, it will just
# check each library and skip it as already installed.
This should install Django and every other python package you need to run the site and post code reviews for the project.
Now return to The GCD project and apps for information on how to setup the environment using a dump from the GCD project.