Merge branch 'release/v1.0'

Author: cheind

Readme.md

@@ -1,100 +1,92 @@
-# pytorch-blender
+# blendtorch
 
-Seamless integration of Blender renderings into [PyTorch](http://pytorch.org) datasets for deep learning from artificial visual data. This repository contains a minimal demonstration that harvests images and meta data from ever changing Blender renderings.
+**blendtorch** is a Python framework to seamlessly integrate [Blender](http://blender.orf) renderings into [PyTorch](http://pytorch.org) datasets for deep learning from artificial visual data. We utilize Eevee, a new physically based real-time renderer, to synthesize images and annotations at 60FPS and thus avoid stalling model training in many cases.
 
+Feature summary
+- Blender Eevee support for real-time rendering.
+- Seamless streaming into PyTorch data pipelines.
+- Supports arbitrary pickle-able objects to be send alongside images/videos.
+- Builtin recording capability to replay data without Blender.
+
+## Minimal sample
+Running [demo.py](./demo.py) using the [cube](./scenes/) scene
 ```
-python pytorch_sample.py
+python demo.py cube
 ```
-renders a set of images of random rotated cubes to `./tmp/output_##.png`, such as the following
+will generate batch visualizations in `./tmp/output_##.png` like the following
 
 ![](etc/result.png)
 
-This image is generated by 4 Blender instances, randomly perturbating a minimal scene. The results are collated in a `BlenderDataset`. A PyTorch `DataLoader` with batch size 4 is used to grab from the dataset and produce the figure.
+This image is generated by reading from 2 Blender instances that randomly perturbate a minimal scene. Individual results (images + corner annotations) are received through a standard PyTorch `Dataset`. We configure a `DataLoader` to form batches of size 4. We iterate the `DataLoader` and create an output image for each batch using `matplotlib`.
+
+Shown below is a batch visualization from 4 Blender instances running physics enabled falling cubes scene.
+
+![](etc/result_physics.png)
+
+To reproduce, run
+```
+python demo.py cube_physics
+```
 
+## Cite
 The code accompanies our [academic work](https://arxiv.org/abs/1907.01879) in the field of machine learning from artificial images. When using please cite the following work
 ```
-@misc{cheindkpts2019,
-Author = {Christoph Heindl and Sebastian Zambal and Josef Scharinger},
-Title = {Learning to Predict Robot Keypoints Using Artificially Generated Images},
-Year = {2019},
-Eprint = {arXiv:1907.01879},
-Note = {To be published at ETFA 2019},
+@inproceedings{robotpose_etfa2019_cheind,
+    author={Christoph {Heindl} and Sebastian Zambal and Josef {Scharinger}},
+    title={Learning to Predict Robot Keypoints Using Artificially Generated Images},
+    booktitle={24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)},    
+    year={2019},
+    publisher={IEEE},   
+    pages={1536-1539},
+    doi={10.1109/ETFA.2019.8868243},
+    isbn={978-1-7281-0303-7},
 }
 ```
 
-## Code outline
-```Python
-import torch.utils.data as data
-
-import blendtorch as bt
-
-# Standard PyTorch Dataset convention
-class MyDataset:
-
-    def __init__(self, blender_launcher, transforms=None):
-        self.recv = bt.Receiver(blender_launcher)
-        self.transforms = transforms
-
-    def __len__(self):
-        # Virtually anything you'd like to end episodes.
-        return 100 
-
-    def __getitem__(self, idx):        
-        # Data is a dictionary of {image, xy, id}, 
-        # see publisher script
-        d = self.recv(timeoutms=5000)   
-        return d['image'], d['xy'], d['id']
-
-kwargs = {
-    'num_instances': 2,
-    'script': 'blender.py',
-    'scene': 'scene.blend',
-}
+## Prerequisites
+This package has been tested using the following packages
+ - [Blender](https://www.blender.org/) >= 2.83 (Python 3.7)
+ - [PyTorch](http://pytorch.org) >= 1.50 (Python 3.7)
 
-with bt.BlenderLauncher(**kwargs) as bl:        
-    ds = MyDataset(bl)        
-    dl = data.DataLoader(ds, batch_size=4, num_workers=0)
+Other versions might work as well, but have not been tested.
 
-    for idx in range(10):
-        x, coords, ids = next(iter(dl))
-        print(f'Received from {ids}')
+## Installation
+First install the prerequisites and clone **blendtorch** to `<SRC>`
+```
+git clone https://github.com/cheind/pytorch-blender.git <SRC>
+```
+Next, ensure Blender executable can be found via `PATH` environment variable and install Python dependencies into Blender's packaged Python distribution
+```
+blender --background --python <SRC>/pkg_blender/install_dependencies.py
+```
+To access **blendtorch** from PyTorch and Blender, we currently recommend updating your `PYTHONPATH` as follows (Windows)
+```
+set PYTHONPATH=%PYTHONPATH%;<SRC>/pkg_pytorch;<SRC>/pkg_blender
+```
+or (Mac or GNU/Linux) 
+```
+export PYTHONPATH="${PYTHONPATH}:<SRC>/pkg_pytorch:<SRC>/pkg_blender"
 ```
 
 ## Runtimes
+The following tables show the mean runtimes per batch (8) and per image for a simple Cube scene (640x480xRGBA). See [benchmark.py](./benchmark.py) for details. The timings include rendering, transfer, decoding and batch collating.
 
-The runtimes for the demo scene (really quick to render) is shown below.
-
-| Blender Instances  | Runtime ms/batch  |
-|:-:|:-:|
-| 1  | `103 ms ± 5.17 ms`  |
-| 2  | `43.7 ms ± 10.3 ms` |
+| Blender Instances  | Runtime sec/batch | Runtime sec/image
+|:-:|:-:|:-:|
+| 1  | 0.236 | 0.030|
+| 2  | 0.14 | 0.018|
+| 4  | 0.099 | 0.012|
 
-The above timings include rendering, transfer and encoding/decoding. Depending on the complexity of renderings you might want to tune the number of instances.
+## Architecture
+**blendtorch** is composed of two distinct sub-packages: `bendtorch.btt`, in folder [pkg_pytorch](./pkg_pytorch]) and `blendtorch.btb`,in folder [pkg_blender](./pkg_blender]), providing the PyTorch and Blender views on **blendtorch**.
 
-## Prerequisites
-The following packages need to be available in your PyTorch environment and Blender environment:
- - Python >= 3.7
- - [Blender](https://www.blender.org/) >= 2.79
- - [PyTorch](http://pytorch.org) >= 0.4
- - [PyZMQ](https://pyzmq.readthedocs.io/en/latest/)
- - [Pillow/PIL](https://pillow.readthedocs.io/en/stable/installation.html)
-
-Both packages are installable via `pip`. In order add packages to your Blender packaged Python distribution, execute the following commands (usually administrator privileges are required on Windows)
+### PyTorch
+At a top level `blendtorch.btt` provides `BlenderLauncher` to launch and close Blender instances, and communication a channel `BlenderInputChannel` to receive from those instances. Communication is based on [ZMQ](https://zeromq.org/) utilizing a `PUSH/PULL` pattern to support various kinds of parallelism. Besides, `blendtorch.btb` provides a raw `Recorder` that saves pickled Blender messages which can later be replayed using `FileInputChannel`.
 
-```
-"<BLENDERPATH>2.79\python\bin\python.exe" -m ensurepip
-"<BLENDERPATH>2.79\python\bin\python.exe" -m pip install pyzmq
-"<BLENDERPATH>2.79\python\bin\python.exe" -m pip install pillow
-```
-where `<BLENDERPATH>` is the file path to the directory containing the Blender executable.
-
-**Note** The Blender executable needs to be in your PATH. On Windows it does not suffice to temporarily modify the PATH variable, as no derived shell is spawned and temporary environment variables are not passed on.
-
-## How it works
-An instance of [BlenderLaunch](blendtorch/launcher.py) is responsible for starting and stopping background Blender instances. The script `blender.py` and additional arguments are passed to the starting Blender instance. `blender.py` creates a publisher socket for communication and starts producing random renderings. Meanwhile, a PyTorch dataset uses a [Receiver](blendtorch/receiver.py) instance to read data from publishers.
+### Blender
+The package `blendtorch.btb` provides offscreen rendering capabilities `OffScreenRenderer`, animation control `Controller` and `BlenderOutputChannel` to publish any pickle-able message. When Blender instances are launched by `blendtorch.btt.BlenderLauncher`, each instance receives specific arguments to determine binding addresses and **blendtorch** instance ids that can later be used determine which instance sent specific messages.
 
 ## Caveats
-- In background mode, Blender `ViewerNodes` are not updated, so rendering have to be written to files. Currently, the Blender script re-imports the written image and sends it as message to any subscriber. This way, we do not need to keep track of which files have already been read and can be deleted, which simplifies communication.
-- In this sample. only the main composite rendering is transmitted. You might want to use `FileOutputNode` instead, to save multiple images per frame.
-- Currently you need to use `num_workers=0` when creating a PyTorch `DataLoader` as the `Receiver` object is capable of multi-process pickling.
-
+- Despite offscreen rendering is supported in Blender 2.8x it requires a UI frontend and thus cannot run in `--background` mode.
+- The renderings produced by Blender are in linear color space and thus will appear darker than expected when displayed. See `gamma_correct` transform [demo.py](./demo.py) to fix this.
+- Currently we do not have support for a feedback channel from PyTorch to Blender.

benchmark.py

@@ -0,0 +1,45 @@
+import torch.utils.data as data
+import argparse
+import time
+from contextlib import ExitStack
+
+from blendtorch import btt
+
+from demo import MyDataset, gamma_correct
+
+BATCH = 8
+INSTANCES = 4
+WORKER_INSTANCES = 2
+        
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('scene', help='Blender scene name to run')
+    args = parser.parse_args()
+
+    with ExitStack() as es:
+        bl = es.enter_context(
+            btt.BlenderLauncher(
+                num_instances=INSTANCES, 
+                script=f'scenes/{args.scene}.py', 
+                scene=f'scenes/{args.scene}.blend'
+            )
+        )
+        channel = btt.BlenderInputChannel(addresses=bl.launch_info.addresses)        
+        ds = MyDataset(channel, stream_length=256)
+        dl = data.DataLoader(ds, batch_size=BATCH, num_workers=WORKER_INSTANCES, shuffle=False)
+        
+        t0 = None
+        imgshape = None
+
+        for item in dl:
+            if t0 is None: # 1st is warmup
+                t0 = time.time()
+                imgshape = item[0].shape
+
+        t1 = time.time()    
+        N = len(ds) - BATCH
+        B = len(ds)//BATCH - 1
+        print(f'Time {(t1-t0)/N:.3f}sec/image, {(t1-t0)/B:.3f}sec/batch, shape {imgshape}')
+
+if __name__ == '__main__':
+    main()