Introduce new build system exaslct for building the Script Language Container (#39)

Author: tkilias

.dockerignore

@@ -1,4 +1,5 @@
 downloads
 *.tar.gz
 tests
-udf-script-signature-generator
+udf-script-signature-generator
+.build_output

.gitignore

@@ -7,3 +7,6 @@ odbc.ini
 .env
 Pipfile.lock
 emulator/zmqcontainer_pb2.py
+.idea
+.build_output/
+luigi.cfg

.travis.yml

@@ -0,0 +1,16 @@
+[requires]
+python_version = "3.6"
+
+[packages]
+luigi = "*"
+jinja2 = "*"
+humanfriendly = "*"
+jsonpickle = "*"
+simplejson = "*"
+docker = "==3.7.2"
+netaddr = "*"
+click = "*"
+requests = "*"
+networkx = "*"
+pydot = "*"
+"stopwatch.py" = "*"

Pipfile

@@ -6,82 +6,228 @@
 ## Table of Contents
 1. [About](#about)
 2. [Prerequisites](#prerequisites)
-3. [Quickstart](#quickstart)
+3. [How to build an existing flavor?](#how-to-build-an-existing-flavor)
+4. [How to customize an existing flavor?](#how-to-customize-an-existing-flavor)
+5. [Partial builds or rebuilds](#partial-builds-and-rebuilds)
+6. [Using your own remote cache](#using-your-own-remote-cache)
+7. [Testing an existing flavor](#testing-an-existing-flavor)
+8. [Cleaning up after your are finished](#cleaning-up-after-your-are-finished)
 
 ## About
-This project contains implementations for user defined functions (UDF's) that can be used in the EXASOL database (version 6.0.0 or later)
+This project contains script language containers for user defined functions (UDF's) 
+that can be used in the EXASOL database (version 6.0.0 or later). 
+A script language container consists of a Linux container with a complete linux distribution and all required libraries, 
+such as a script client. A script client is responsible for the communication with the database and for executing the script code.
+We provide in this repository several [flavors](flavors) of script language containers, 
+such as the current standard implementation of the [script client](src) with support for Python 2/3, R und Java. 
+We will show here how to customize and build the different flavors of the script language containers. 
+Pre-built containers can you find in the [release section](https://github.com/exasol/script-languages/releases) of this repository.
+If you are interested in the script client you find more details [here](src/README.md).
 
 ## Prerequisites
 In order to build this project, you need:
 * Linux or Mac OS X (experimental)
 * Docker
+* Python >=3.6 with pip
+* We recommend at least 50 GB free disk space on the partition 
+  where Docker stores its images, on linux Docker typically stores 
+  the images at /var/lib/docker.
+* For the partition where the output directory (default: ./.build_output) 
+  is located we recommend additionally at least 10 GB free disk space.
 
-In order to follow the quickstart guide, you additionally need
-* Write-access to a bucket in a bucketfs in an EXASOL installation
-* curl
-* An SQL client connecting to the same EXASOL installation
+Further, prerequisites might be necessary for specific tasks. These are listed under the corresponding section.
 
-For running the tests you also need
-* Python
-* GCC
-* Java
-* unixODBC
-* Docker with privileged mode
+## How to build an existing flavor?
 
-## Quickstart
-1. Choose a flavor. Currently we have several pre-defined flavors available, e.g., `mini-EXASOL-6.0.0`, and `standard-EXASOL-6.1.0`.
+Choose a flavor. Currently we have several pre-defined flavors available, e.g., `mini-EXASOL-6.0.0`, and `standard-EXASOL-6.1.0`.
 This project supports different versions of script language environments with different libraries and languages.
 We call these versions _flavors_. The pre-defined flavors can be modified and extended to create customized flavors.
-Each pre-defined flavor has its own set of Docker build-files in a corresponding subfolder of [flavors](flavors).
-2. Create the language container. We choose to use the `mini` flavor which is the smallest of the currently available flavors and which only support the Python language.
+Each pre-defined flavor has its own set of Dockerfiles in a corresponding subfolder of [flavors](flavors).
+
+Create the language container and export it to the local file system
+
 ```bash
-$ ./build --flavor=mini-EXASOL-6.0.0
+$ ./exaslct export --flavor-path=flavors/<flavor-name> --export-path <export-path>
 ```
-(on Mac OS X, use `./build -f mini-EXASOL-6.0.0`)
 
-3. Export it into a standalone archive
+or upload it directly into your BuckerFS (currently http only, https follows soon)
+
 ```bash
-$ ./export --flavor=mini-EXASOL-6.0.0
+$ ./exaslct upload --flavor-path=flavors/<flavor-name> --database-host <hostname-or-ip> --bucketfs-port <port> \ 
+                   --bucketfs-username w --bucketfs-password <password>  --bucketfs-name <bucketfs-name> \
+                   --bucket-name <bucket-name> --path-in-bucket <path/in/bucket>
 ```
-(on Mac OS X, use `./export -f mini-EXASOL-6.0.0`)
-This creates the file `mini-EXASOL-6.0.0.tar.gz`.
 
-Optionally, you can run some automated tests for your flavor by using
-```bash
-$ ./test_complete --flavor=mini-EXASOL-6.0.0
+Once it is successfully uploaded, it will print the ALTER SESSION statement
+that can be used to activate the script language container in the database.
+
+## How to customize an existing flavor?
+
+To customize an existing flavor you can add your specific needs to the Dockerfile in the flavor-customization directory. 
+You can run commands with:
+
+```Dockerfile
+RUN <command>
 ```
-(on Mac OS X you need to have pip installed for this to work)
-If the test fails with the message
+
+For example, to install new software you can use:
+
+```Dockerfile
+RUN apt-get -y update && \
+    apt-get install \<packages> && \
+    apt-get -y clean && \
+    apt-get -y autoremove
+```
+    
+You need to run apt-get update, because any previous step clears the cache of apt to keep the docker images small. 
+The commands 
+
+```Dockerfile
+apt-get -y clean and apt-get -y autoremove 
 ```
-cp: cannot create regular file ‘/tmp/udftestdb/exa/etc/EXAConf’: Permission denied
+
+clear the cache.
+
+You can add to the flavor-customization directory additional files which you can use in the Dockerfile via:
+
+```Dockerfile
+COPY flavor-customization/<your-file-or-directory> <destination>
 ```
-then run `stop_dockerdb` and restart the test.
 
-4. Upload the file into bucketfs. For the following example we assume the password `pwd` and the bucketname `funwithudfs` in a bucketfs that is running on port `2580` on machine `192.168.122.158`
+or 
+
+```Dockerfile
+ADD flavor-customization/<your-file-or-directory> <destination>
+```
+
+Your changes on the file system will then be merged with the file system of the script client
+which contains all necessary libraries that are required to run the script language runtime.
+
+Be aware that the merge will override all changes which may prevent the execution of the script client.
+In details, this means if you change or remove packages or files in flavor-customization
+which are necessary for the script client they will be restored in the final container.
+
+After you finished your changes, rebuild with 
+
 ```bash
-curl -v -X PUT -T mini-EXASOL-6.0.0.tar.gz w:pwd@192.168.122.158:2580/funwithudfs/mini-EXASOL-6.0.0.tar.gz
+$ ./exaslct export --flavor-path=flavors/<flavor-name>
 ```
-5. In SQL you activate the Python implementation of the flavor `mini-EXASOL-6.0.0` by using a statement like this
-```sql
-ALTER SESSION SET SCRIPT_LANGUAGES='MYPYTHON=localzmq+protobuf:///bucketfsname/funwithudfs/mini-EXASOL-6.0.0?lang=python#buckets/bucketfsname/funwithudfs/mini-EXASOL-6.0.0/exaudf/exaudfclient';
+
+or upload it directly into your BuckerFS (currently http only, https follows soon)
+
+```bash
+$ ./exaslct upload --flavor-path=flavors/<flavor-name> --database-host <hostname-or-ip> --bucketfs-port <port> \ 
+                   --bucketfs-username w --bucketfs-password <password>  --bucketfs-name <bucketfs-name> \
+                   --bucket-name <bucket-name> --path-in-bucket <path/in/bucket>
 ```
-Now the script language `MYPYTHON` can be used to define a script, e.g., 
+
+Note: The tool `exaslct` tries to reuse as much as possible of the previous build or tries to pull already exising images from Docker Hub.
+
+## Force a rebuild
+
+Sometimes it is necessary to force a rebuild of a flavor. 
+A typical reason is to update the dependencies in order to
+fix bugs and security vulnerabilities in the installed dependencies.
+To force a rebuild the command line option --force-rebuild can be used 
+with basically all commands of ./exaslct, except the clean commands.
+
+## Partial builds and rebuilds
+
+In some circumstances you want to build or rebuild 
+only some parts of the flavor. Most likely during development or during CI. 
+You can specify for a build upper bounds (also called goals) 
+until which the flavor should be build and for rebuilds 
+you can define lower bounds from where the rebuild get forced.
+
+You can define upper bounds with the commandline option --goal 
+for the ./exaslct commands build and push. 
+The build command only rebuilds the docker images, 
+but does not export a new container.
+All other commands don't support the --goal option, 
+because they require specific images to be built,
+otherwise they would not proceed.
+
+```bash
+./exaslct build --flavor-path=<path-to-flavor> --goal <build-stage>
 ```
-CREATE SCHEMA S;
-CREATE mypython SCALAR SCRIPT small_test() RETURNS DOUBLE AS
-def run(ctx):
-   return 1.2
-/
+
+If you want to build several different build-stages at once, you can repeat the --goal option.
+
+The following build-stage are currently available:
+* udfclient-deps
+* language-deps
+* build-deps
+* build-run
+* base-test-deps
+* base-test-build-run
+* flavor-test-build-run
+* flavor-base-deps
+* flavor-customization
+* release
+
+
+With the option --force-rebuild-from, you can specify from where the rebuild should be forced.
+All previous build-stages before this will use cached versions where possible.
+However, if a single stage is built, it will trigger a build for all following build-stages.
+The option --force-rebuild-from only has an effect together with the option --force-rebuild, 
+without it is ignored.
+
+```bash
+./exaslct build --flavor-path=<path-to-flavor> --force-rebuild --force-rebuild-from <build-stage>
 ```
-The script can be executed as follows:
+
+Similar, as for the --goal option, you can specify multiple lower bounds 
+by repeating the --force-rebuild-from with different build-stages.
+
+## Using your own remote cache
+
+Exaslct caches images locally and remotely. 
+For remote caching exaslct can use a docker registry. 
+The default registry is configured to Docker Hub. 
+With the command line options --repository-name 
+you can configure your own docker registry as cache. 
+The --repository-name option can be used with all 
+./exaslct commands that could trigger a build, 
+which include build, export, upload and run-db-test commands.
+Furthermore, it can be used with the push command which
+uploads the build images to the docker registry.
+In this case the --repository-name option specifies 
+not only from where to pull cached images during the build,
+but also to which cache the built images should be pushed.
+
+You can specify the repository name, as below:
+
+```bash
+./exaslct export --flavor-path=<path-to-flavor> --repository-name <hostname>[:port]/<user>/<repository-name>
 ```
-select small_test();
+
+## Testing an existing flavor
+
+To test the script language container you can execute the following command:
+
+```bash
+$ ./exaslct run-db-test --flavor-path=flavors/<flavor-name>
 ```
-6. Afterwards you may choose to remove the Docker images for this flavor. This can be done as follows:
+
+**Note: you need docker in privileged mode to execute the tests**
+
+## Cleaning up after your are finished
+
+The creation of scripting language container creates or downloads several docker images
+which can consume a lot of disk space. Therefore, we recommand to remove the Docker images
+of a flavor after working with them.
+
+This can be done as follows:
+
 ```bash
-./clean --flavor=mini-EXASOL-6.0.0
+./exaslct clean-flavor-images --flavor-path=flavors/<flavor-name>
 ```
-Please note that this script does not delete the Linux image that is used as basis for the images that were build in the previous steps.
 
+To remove all images of all flavors you can use:
 
+```bash
+./exaslct clean-all-images
+```
 
+**Please note that this script does not delete the Linux image that is used as basis for the images that were build in the previous steps. 
+Furthermore, this command doesn't delete cached files in the output directory. The default path for the output directory is .build-output.**