Merge branch 'release/v0.2'

Author: Heindl Christoph

.gitignore

@@ -104,5 +104,7 @@ venv.bak/
 .mypy_cache/
 
 tmp/*
-scene.blend1
-indin2019/pytorch-blender.pdf
+examples/datagen/tmp/*
+*.blend1
+
+!__keep__

.travis.yml

@@ -0,0 +1,29 @@
+
+language: python
+python:
+  - 3.7
+  - 3.8
+
+cache:
+  pip: true
+  apt: true
+  directories:
+    - $HOME/.pip-cache
+    - $HOME/.blender-cache
+
+install:
+  - scripts/install_blender.sh # requires https://stackoverflow.com/questions/42154912/permission-denied-for-build-sh-file
+  - source .envs
+  - pip install -e pkg_pytorch
+  - pip install -r requirements_dev.txt
+  - python -c "import blendtorch.btt as btt; print(btt.__version__)"
+  - blender --background --python scripts/install_btb.py
+  - blender --background --python-use-system-env --python-expr "import blendtorch.btb as btb; print(btb.__version__)"
+
+script:
+  - pytest tests -m background
+
+notifications:
+  email:
+      on_success: never # change
+      on_failure: never # always

Changelog.md

@@ -0,0 +1,16 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.2.0] - 2020-08-04
+- Support for training RL agents in OpenAI environments defined in Blender. See `examples/control` for details.
+- Support for dataset transformations when `num_workers > 0`.
+- Support for message recording when `num_workers > 0`.
+- Made `blendtorch.btb` and `blendtorch.btt` installable packages. See `Readme.md` for details.
+- Remote data streaming now uses PyTorch IterableDataset and 
+hence simplifies the interface. See `examples/datagen` for details.
+- Added unit tests and CI.
+
+## [0.1.0] - 2020-07-10
+- added Blender Eevee 2.8 support

Readme.md

@@ -1,100 +1,115 @@
-# pytorch-blender
+# blendtorch v0.2
+![](https://travis-ci.org/cheind/pytorch-blender.svg?branch=develop)
 
-Seamless integration of Blender renderings into [PyTorch](http://pytorch.org) datasets for deep learning from artificial visual data. This repository contains a minimal demonstration that harvests images and meta data from ever changing Blender renderings.
+**blendtorch** is a Python framework to seamlessly integrate [Blender](http://blender.orf) renderings into [PyTorch](http://pytorch.org) datasets for deep learning from artificial visual data. We utilize Eevee, a new physically based real-time renderer, to synthesize images and annotations in real-time and thus avoid stalling model training in many cases.
 
-```
-python pytorch_sample.py
-```
-renders a set of images of random rotated cubes to `./tmp/output_##.png`, such as the following
-
-![](etc/result.png)
-
-This image is generated by 4 Blender instances, randomly perturbating a minimal scene. The results are collated in a `BlenderDataset`. A PyTorch `DataLoader` with batch size 4 is used to grab from the dataset and produce the figure.
-
-The code accompanies our [academic work](https://arxiv.org/abs/1907.01879) in the field of machine learning from artificial images. When using please cite the following work
-```
-@misc{cheindkpts2019,
-Author = {Christoph Heindl and Sebastian Zambal and Josef Scharinger},
-Title = {Learning to Predict Robot Keypoints Using Artificially Generated Images},
-Year = {2019},
-Eprint = {arXiv:1907.01879},
-Note = {To be published at ETFA 2019},
-}
-```
+Feature summary
+ - ***Data Streaming***: Stream distributed Blender renderings directly into PyTorch data pipelines in real-time for supervised learning and domain randomization applications. Supports arbitrary pickle-able objects to be send alongside images/videos. Built-in recording capability to replay data without Blender.</br>More info [\[examples/datagen\]](examples/datagen)
+ - ***OpenAI Gym Support***: Create and run remotely controlled Blender gyms to train reinforcement agents. Blender serves as simulation, visualization, and interactive live manipulation environment.
+ </br>More info [\[examples/control\]](examples/control)
 
-## Code outline
-```Python
-import torch.utils.data as data
+The figure below visualizes a single image/label batch received by PyTorch from four parallel Blender instances. Each Blender process repeatedly performs motion simulations of randomized cubes.
 
-import blendtorch as bt
+<p align="center">
+<img src="etc/result_physics.png" width="500">
+</p>
 
-# Standard PyTorch Dataset convention
-class MyDataset:
+## Getting started
+ 1. Read the installation instructions below
+ 1. To get started with **blendtorch** for training data training read [\[examples/datagen\]](examples/datagen). 
+ 1. To learn about using **blendtorch** for creating reinforcement training environments read [\[examples/control\]](examples/control).
 
-    def __init__(self, blender_launcher, transforms=None):
-        self.recv = bt.Receiver(blender_launcher)
-        self.transforms = transforms
+## Installation
 
-    def __len__(self):
-        # Virtually anything you'd like to end episodes.
-        return 100 
+**blendtorch** is composed of two distinct sub-packages: `bendtorch.btt` (in [pkg_pytorch](./pkg_pytorch)) and `blendtorch.btb` (in [pkg_blender](./pkg_blender)), providing the PyTorch and Blender views on **blendtorch**. 
 
-    def __getitem__(self, idx):        
-        # Data is a dictionary of {image, xy, id}, 
-        # see publisher script
-        d = self.recv(timeoutms=5000)   
-        return d['image'], d['xy'], d['id']
+### Prerequisites
+This package has been tested with
+ - [Blender](https://www.blender.org/) >= 2.83 (Python 3.7)
+ - [PyTorch](http://pytorch.org) >= 1.50 (Python 3.7/3.8)
+running Windows 10 and Linux.
 
-kwargs = {
-    'num_instances': 2,
-    'script': 'blender.py',
-    'scene': 'scene.blend',
-}
-
-with bt.BlenderLauncher(**kwargs) as bl:        
-    ds = MyDataset(bl)        
-    dl = data.DataLoader(ds, batch_size=4, num_workers=0)
+Other versions might work as well, but have not been tested. 
 
-    for idx in range(10):
-        x, coords, ids = next(iter(dl))
-        print(f'Received from {ids}')
+### Clone this repository
+```
+git clone https://github.com/cheind/pytorch-blender.git <DST>
 ```
 
-## Runtimes
-
-The runtimes for the demo scene (really quick to render) is shown below.
+### Extend `PATH`
+Ensure Blender executable is in your environments lookup `PATH`. On Windows this can be accomplished by
+```
+set PATH=c:\Program Files\Blender Foundation\Blender 2.83;%PATH%
+```
 
-| Blender Instances  | Runtime ms/batch  |
-|:-:|:-:|
-| 1  | `103 ms ± 5.17 ms`  |
-| 2  | `43.7 ms ± 10.3 ms` |
+### Install **blendtorch** Blender part
+```
+blender --background --python <DST>/scripts/install_btb.py
+```
+installs `blendtorch-btb` into the Python environment bundled with Blender. 
 
-The above timings include rendering, transfer and encoding/decoding. Depending on the complexity of renderings you might want to tune the number of instances.
+### Install **blendtorch** PyTorch part
+```
+pip install -e <DST>/pkg_pytorch
+```
+installs `blendtorch-btt` into the Python environment that you intend to run PyTorch from. While not required, it is advised to install OpenAI gym if you intend to use **blendtorch** for reinforcement learning
+```
+pip install gym
+```
+### Developer instructions
+This step is optional. If you plan to run the unit tests
+```
+pip install -r requirements_dev.txt
+pytest tests/
+```
 
-## Prerequisites
-The following packages need to be available in your PyTorch environment and Blender environment:
- - Python >= 3.7
- - [Blender](https://www.blender.org/) >= 2.79
- - [PyTorch](http://pytorch.org) >= 0.4
- - [PyZMQ](https://pyzmq.readthedocs.io/en/latest/)
- - [Pillow/PIL](https://pillow.readthedocs.io/en/stable/installation.html)
+### Troubleshooting
+Run
+```
+blender --version
+```
+and check if the correct Blender version (>=2.83) is written to console. Next, ensure that `blendtorch-btb` installed correctly
+```
+blender --background --python-use-system-env --python-expr "import blendtorch.btb as btb; print(btb.__version__)"
+```
+which should print **blendtorch** version number on success. Next, ensure that `blendtorch-btt` installed correctly
+```
+python -c "import blendtorch.btt as btt; print(btt.__version__)"
+```
+which should print **blendtorch** version number on success.
 
-Both packages are installable via `pip`. In order add packages to your Blender packaged Python distribution, execute the following commands (usually administrator privileges are required on Windows)
+## Architecture
+Please see [\[examples/datagen\]](examples/datagen) and [examples/control\]](examples/control) for an in-depth architectural discussion.
 
+## Cite
+The code accompanies our [academic work](https://arxiv.org/abs/1907.01879) in the field of machine learning from artificial images. When using please cite the following work
 ```
-"<BLENDERPATH>2.79\python\bin\python.exe" -m ensurepip
-"<BLENDERPATH>2.79\python\bin\python.exe" -m pip install pyzmq
-"<BLENDERPATH>2.79\python\bin\python.exe" -m pip install pillow
+@inproceedings{robotpose_etfa2019_cheind,
+    author={Christoph Heindl and Sebastian Zambal and Josef Scharinger},
+    title={Learning to Predict Robot Keypoints Using Artificially Generated Images},
+    booktitle={
+        24th IEEE International Conference on 
+        Emerging Technologies and Factory Automation (ETFA)
+    },    
+    year={2019},
+    pages={1536-1539},
+    doi={10.1109/ETFA.2019.8868243},
+    isbn={978-1-7281-0303-7},
+}
 ```
-where `<BLENDERPATH>` is the file path to the directory containing the Blender executable.
 
-**Note** The Blender executable needs to be in your PATH. On Windows it does not suffice to temporarily modify the PATH variable, as no derived shell is spawned and temporary environment variables are not passed on.
+## Runtimes
+The following tables show the mean runtimes per batch (8) and per image for a simple Cube scene (640x480xRGBA). See [benchmarks/benchmark.py](./benchmarks/benchmark.py) for details. The timings include rendering, transfer, decoding and batch collating.
+
+| Blender Instances  | Runtime sec/batch | Runtime sec/image | Arguments|
+|:-:|:-:|:-:|:-:|
+| 1  | 0.236 | 0.030| UI refresh|
+| 2  | 0.14 | 0.018| UI refresh|
+| 4  | 0.099 | 0.012| UI refresh|
+| 5  | 0.085 | 0.011| no UI refresh|
 
-## How it works
-An instance of [BlenderLaunch](blendtorch/launcher.py) is responsible for starting and stopping background Blender instances. The script `blender.py` and additional arguments are passed to the starting Blender instance. `blender.py` creates a publisher socket for communication and starts producing random renderings. Meanwhile, a PyTorch dataset uses a [Receiver](blendtorch/receiver.py) instance to read data from publishers.
+Note: If no image transfer is needed, i.e in reinforcement learning of physical simulations, 2000Hz are easily achieved.
 
 ## Caveats
-- In background mode, Blender `ViewerNodes` are not updated, so rendering have to be written to files. Currently, the Blender script re-imports the written image and sends it as message to any subscriber. This way, we do not need to keep track of which files have already been read and can be deleted, which simplifies communication.
-- In this sample. only the main composite rendering is transmitted. You might want to use `FileOutputNode` instead, to save multiple images per frame.
-- Currently you need to use `num_workers=0` when creating a PyTorch `DataLoader` as the `Receiver` object is capable of multi-process pickling.
-
+- Despite offscreen rendering is supported in Blender 2.8x it requires a UI frontend and thus cannot run in `--background` mode.
+- The renderings produced by Blender are by default in linear color space and thus will appear darker than expected when displayed.

benchmarks/benchmark.py

@@ -0,0 +1,50 @@
+import time
+import argparse
+from pathlib import Path
+import torch.utils.data as data
+from blendtorch import btt
+
+BATCH = 8
+INSTANCES = 4
+WORKER_INSTANCES = 2
+NUM_ITEMS = 512
+EXAMPLES_DIR = Path(__file__).parent/'..'/'examples'/'datagen'
+        
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--scene', help='Blender scene name to run', default='cube')
+    args = parser.parse_args()
+
+    launch_args = dict(
+        scene=EXAMPLES_DIR/f'{args.scene}.blend',
+        script=EXAMPLES_DIR/f'{args.scene}.blend.py', 
+        num_instances=INSTANCES,
+        named_sockets=['DATA']
+    )
+
+    with btt.BlenderLauncher(**launch_args) as bl:                        
+        ds = btt.RemoteIterableDataset(bl.launch_info.addresses['DATA'])
+        ds.stream_length(NUM_ITEMS)
+        dl = data.DataLoader(ds, batch_size=BATCH, num_workers=WORKER_INSTANCES, shuffle=False)
+
+        # Wait to avoid timing startup times of Blender
+        time.sleep(5)
+        
+        t0 = None
+        imgshape = None
+        
+        n = 0
+        for item in dl:
+            if t0 is None: # 1st is warmup
+                t0 = time.time()
+                imgshape = item['image'].shape
+            n += len(item['image'])
+        assert n == NUM_ITEMS
+
+        t1 = time.time()    
+        N = NUM_ITEMS - BATCH
+        B = NUM_ITEMS//BATCH - 1
+        print(f'Time {(t1-t0)/N:.3f}sec/image, {(t1-t0)/B:.3f}sec/batch, shape {imgshape}')
+
+if __name__ == '__main__':
+    main()