Merge pull request #50 from developmentseed/develop

updates to python geolambda

Author: matthewhanson

.circleci/config.yml

@@ -41,6 +41,10 @@ jobs:
             docker tag developmentseed/geolambda:${VERSION} developmentseed/geolambda:latest
             docker push developmentseed/geolambda:latest
             docker push developmentseed/geolambda:${VERSION}
+            cd python
+            docker build --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.6.1 . -t developmentseed/geolambda:${VERSION}-python36
+            docker push developmentseed/geolambda:${VERSION}-python36
+            cd ..
             # deploy public Lambda layers
             pip install awscli
             docker run --rm -v $PWD:/home/geolambda --entrypoint package.sh -it developmentseed/geolambda:${VERSION}

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Improve documentation
 - More libraries (CURL with http2, webp, zstd)
 - GeoTIFF now compiled from scratch rather than GGDAL builtin
+- Published public lambda layers - see README for ARNs
 
 ### Changed
 - Major refactor, GeoLambda base now runtime agnostic contains system libraries only

Dockerfile

@@ -175,7 +175,7 @@ RUN \
         --with-hdf4=${PREFIX} \
         --with-hdf5=${PREFIX} \
         --with-netcdf=${PREFIX} \
-        --with-web=${PREFIX} \
+        --with-webp=${PREFIX} \
         --with-zstd=${PREFIX} \
         --with-threads=yes \
 		--with-curl=${PREFIX}/bin/curl-config \

python/Dockerfile

@@ -3,21 +3,23 @@ FROM developmentseed/geolambda:${VERSION}
 
 LABEL maintainer="Matthew Hanson <matt.a.hanson@gmail.com>"
 
-ARG PYVER=3.6.1
+ARG PYVERSION=3.6.1
 
 # install Python
 ENV \
     PYENV_ROOT=/root/.pyenv \
     PATH=/root/.pyenv/shims:/root/.pyenv/bin:$PATH
 
 RUN \
-    echo PYVER ${PYVER}; \
     curl https://pyenv.run | bash; \
-    pyenv install ${PYVER}; \
-    pyenv global ${PYVER}; \
+    pyenv install ${PYVERSION}; \
+    pyenv global ${PYVERSION}; \
     pip install --upgrade pip
 
-COPY requirements.txt ./
+COPY requirements*.txt ./
 
 RUN \
-    pip install -r requirements.txt
+    pip install -r requirements-pre.txt; \
+    pip install -r requirements.txt
+
+COPY bin/* /usr/local/bin/

python/README.md

@@ -6,30 +6,31 @@ The Python GeoLambda image is meant to be used as a template for your own Python
 
 These instructions show how to use the files in this directory to create your own Python based geospatial Lambda function and get it deployed.
 
-### Create docker image
+1. Update requirements
 
-First a Docker image will need to be created based on GeoLambda that includes the dependencies needed. Use the [Dockerfile](Dockerfile) can be used to create a new Docker image based on any version of GeoLambda with any version of Python by providing the versions as build arguments to `docker run`. This will install the specified version of Python along with any Python packages provided in [requirements.txt](requirements.txt).
+First a Docker image will need to be created based on GeoLambda that includes the dependencies needed. The first thing is to is update the `requirements-pre.txt` and `requirement.txt` files with the dependencies needed. The `requirements-pre.txt` is a way to specify build dependencies to packages in `requirements.txt`. For instance, rasterio requires numpy is installed before rasterio. This is a well known pip problem that is fixed in [PEP 518](https://www.python.org/dev/peps/pep-0518/) but is not used by all packages so this work-around is needed.
 
-    $ docker build . --build-arg VERSION=1.1.0 --build-arg PYVER=3.6.1 -t <myimage>:latest
+2. Create handler
 
-If not provided, `VERSION` (the version of GeoLambda to use) will default to `latest` and `PYVER` (Python version) will default to `3.6.1`.
+An example Lambda handler is located at [lambda/lambda_function.py](lambda/lambda_function.py). Ideally most of the logic in your Lambda should be in packages and the handler should only be responsible for taking in an event, calling relevant functions, and returning some output. Keep the handler code as small as possible. Move more complex code into packages or make them other functions in the [lambda/lambda_function.py](lambda/lambda_function.py) file.
 
-### Create handler
+3. Build image
 
-An example Lambda handler is located at [lambda/lambda_function.py](lambda/lambda_function.py). Ideally most of the logic in your Lambda should be in packages and the handler should only be responsible for taking in an event, calling relevant functions, and returning some output. Keep the handler code as small as possible. Move more complex code into packages or make them other functions in the [lambda/lambda_function.py](lambda/lambda_function.py) file.
+Now, use the [Dockerfile](Dockerfile) can be used to create a new Docker image based on any version of GeoLambda with any version of Python by providing the versions as build arguments to `docker run`. This will install the specified version of Python along with any Python packages provided in [requirements.txt](requirements.txt).
 
-### Create deployment package
+    $ docker build . --build-arg VERSION=1.1.0 --build-arg PYVERSION=3.6.1 -t <myimage>:latest
 
-All that's needed to create a deployment package is to install the Python packages into the lambda/ directory.
+If not provided, `VERSION` (the version of GeoLambda to use) will default to `latest` and `PYVERSION` (Python version) will default to `3.6.1`.
 
-    $ docker run -v ${PWD}:/home/geolambda -t <myimage>:latest pip install -r requirements -t lambda/
+4. Create deployment package
 
-The lambda/ directory can now be zipped up and deployed as a Lambda function:
+All that's needed to create a deployment package is to install the Python packages into the lambda/ directory using the package-python.sh script. This copies the installed site-packages over the lambda directory, but excluding some libraries since they won't be needed on Lambda. This includes packages that are already pre-installed in Lambda like boto3, as well as files and libraries that wouldn't be used operationally (e.g., testing files).
 
-    $ cd lambda
-    $ zip -ru ../lambda-deploy.zip ./
+    $ docker run -v ${PWD}:/home/geolambda -t <myimage>:latest package-python.sh
 
-### Testing deployment package
+This will also create a lambda-deployment.zip file in the current directory.
+
+5. Testing deployment package
 
 You can use the [LambCI Docker images](https://github.com/lambci/docker-lambda) to test out your handler in the Lambda environment by mounting the base GeoLambda Lambda layer (see the [GeoLambda README](../README.md)) and the lambda directory created above.
 
@@ -40,11 +41,19 @@ $ docker run --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt \
 
 The last argument is a JSON string that will be passed as the event payload to the handler function.
 
+6. Deploy
+
+Deploy the Lambda function now by zipping up the lambda directory and using the AWS CLI.
+
+```
+$ aws lambda update-function-code --function-name <mylambda> --zip-file fileb://lambda-deploy.zip
+```
+
 ### Pre-built images
 
 Builds of this default Python image are also available on Docker Hub as tags in the `developmentseed/geolambda` repository.
 
-    $ docker pull developmentseed/geolambda:1.1.0-python36
+    $ docker pull developmentseed/geolambda:${VERSION}-python36
 
 To run the image interactively:
 
@@ -64,7 +73,7 @@ $ VERSION=${cat ../VERSION}
 $ docker build . --build-arg VERSION=${VERSION} -t developmentseed/geolambda:${VERSION}-python36
 ```
 
-Provide `--build-arg PYVER=X.Y.Z` to build other Python versions and change the tag to be `${VERSION}-pythonX.Y`
+Provide `--build-arg PYVERSION=X.Y.Z` to build other Python versions and change the tag to be `${VERSION}-pythonX.Y`
 
 
 

python/bin/package-python.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# directory used for deployment
+export DEPLOY_DIR=lambda
+
+# Get Python version
+PYVERSION=$(cat /root/.pyenv/version)
+MAJOR=${PYVERSION%%.*}
+MINOR=${PYVERSION#*.}
+PYVER=${PYVERSION%%.*}${MINOR%%.*}
+PYPATH=/root/.pyenv/versions/$PYVERSION/lib/python3.6/site-packages/
+
+echo Creating deploy package for Python $PYVERSION
+
+EXCLUDE="boto3* botocore* pip* docutils* *.pyc setuptools* wheel* coverage* testfixtures* mock* *.egg-info *.dist-info __pycache__ easy_install.py"
+
+EXCLUDES=()
+for E in ${EXCLUDE}
+do
+    EXCLUDES+=("--exclude ${E} ")
+done
+
+rsync -ax $PYPATH/ $DEPLOY_DIR/ ${EXCLUDES[@]}
+
+# zip up deploy package
+cd $DEPLOY_DIR
+zip -ruq ../lambda-deploy.zip ./

python/requirements-pre.txt

@@ -0,0 +1 @@
+numpy

python/requirements.txt

@@ -1,2 +1,2 @@
 GDAL==2.4.0
-rasterio==1.0.21
+rasterio==1.0.21 --no-binary rasterio