Plan "Back on Track" - Proposition

[howetuft]

Proposed Plan to Get LuxCore "Back on Track"

Dear all,

Following our discussions on Discord, I would like to present a proposed plan to get LuxCore fully functional again now and in the future.

Diagnosis (as of early 2025)

LuxCore development has been halted for several months, and most of the core developers have left the project. The consequences are as follows:

• Dependencies have become obsolete.
• The core base is now partially unbuildable. Builds for Mac, Linux, etc., have been deactivated.
• LuxCore is slowly fading from the CG landscape (removed from main Linux distro repositories, no longer compatible with the latest Blender versions, etc.).
• Feature requests are not being addressed.
• Governance is no longer in place.

What we still have

• The community is still active.
• A few historical developers are still available. Some fixes have been made on BlendLuxCore, for instance.
• Invaluable work is preserved in the codebase, awaiting minor adaptations to current dependency versions.
• A new experimental build chain, the "wheel maker," has been released as a side initiative. This chain builds pyluxcore.so for four platforms, against the latest versions of dependencies, and packages it into Python wheel format ("wheels"). These wheels have been published on PyPI.
• An experimental application of the wheels to BlendLuxCore has been created, compatible with Blender 4.2 LTS ("BlendLuxCore Wheels"). It appears promising and generally usable, though it still contains bugs that need fixing.

About the Wheel Maker

Strengths:

• Based on the Conan package manager (Conan.io), which simplifies dependency management and offers a vast, active recipe database with rich features.
• Targets the latest versions of dependencies.
• Builds all dependencies from source to ensure access to the latest versions and binary compatibility.
• Utilizes GitHub Actions (GitHub's continuous integration system).
• Built on the for_blender4.1 branch of LuxCore (no dedicated branch yet, though this has resulted in some patches).

Pain Points:

• Dependencies are rebuilt each time, which is not ideal for development.
• Several "ugly" patches have been applied to the build tree (e.g., using sed and similar tools) to adapt it to new dependency versions
• Some hacks are used, especially for OIDN.
• Not fully optimized in some areas (e.g., Boost usage, OIDN limited to CPU only).
• Lacking CUDA support
• Is a separate project at the moment

Those limitations are not definitive and mainly due to lack of time. However, an effort is required to complete this work in full.

Proposed Strategy: "Back on Track"

The task ahead is significant, and we need to set clear priorities to avoid burnout. I propose the following three priorities:

1. Make LuxCore Buildable Again: Replace the old build system with a new one derived from the wheel maker, once the existing pain points are resolved.
2. Alleviate LuxCore Maintenance: Focus on pyluxcore.so, deprecate C++ samples (e.g., pyluxcoreconsole, pyluxcoreui), and replace them with Python equivalents. Use a package manager to control dependency versions.
3. Make BlendLuxCore Available Everywhere: Capitalize on BlendLuxCore Wheels, stabilize the version for Blender 4.2, and make it widely available as a Blender extension via Blender web.

To the contrary, for now, while the build system is being reconstructed, I would recommend postponing any major functional changes to the core codebase. I would also propose to focus on Blender integration only (BlendLuxCore),
We should also be conservative about adding new features to BlendLuxCore, focusing primarily on bug fixes at this stage and features that don't need any evolution on the C++ part.

Prerequisite: Staffing and Governance

It's a pretty big job, and requires a team and a governance. In my opinion, we would need at least 2 lead devs (vision, reviews, git plumbing and so on), one for C++ and one for BlendLuxCore, and a good staffing behind them.

Please note that, personally, I can only reasonably free up bandwidth on one subject, specifically the build system.

We should then ask for contributors...

If we agree on these guidelines, I can prepare an action plan with more details.

Your feedback is welcome.

[CodeFHD]

+1 for the strategy steps. No core feature updates at the moment, first get the build pipeline, platforms and dependencies updates, then resume work on the core.

From my past expierence with development, the first pain point mentioned above is something I am also thinking about. Not just because of the BlendLuxCore + wheels system, but also because of Blenders new extension system. Maybe I just still need to understand it better, but still I'm wondering at the moment:

• Since around the time LuxCore became "incactive", Blender has introduced LTS versions. This is something we should consider supporting as well, incl. the codebase/branch/build pipeline management.
• Similarly, there is now the "VFX reference platform" which might need to be followed as well. (Note, I don't know if that is really needed for LuxCore)
• With Addons, it was simple to have multiple versions of an addon and quickly switch between them inside Blender by (de-)activating. One could have stable release, nightly build, and local development version side by side.
• Between multiple Blender versions (e.g. 3.6 LTS, 4.2 LTS, ...) you need to ensure a code update is installed for each version. With my own addon (OptiCore) I currently solve this by developing the addon in a separate location, and installing only a wrapper addon in Blender (containing only the bl_info dict, a sys.path.insert() to my development folder with a "from x import *" and finally "if name == 'main': register()'" ). A similar workflow with extensions would be nice.
• Regarding the recompilation of dependencies it sounds like you have that on the list, so not too much comment. I imagine the following in my head, let me know if this matches your mid-term plan:
  a) a dependenc manager takes care to collect and compile the dependencies into a repository like the current LinuxCompileDepes etc.
  b) A core-compilation script build the core code against these dependencies (like the current LinuxCompile repository)
  c) The wheel-build collects the python addon, LuxCore binaries and needed dependency-libraries into a wheel package and publihes it.

Before working on new core features, it would also be helpful to clear out the GitHub issues. Either by fixing what is possible, closing what became obsolete, or otherwise transfer it to. This is also a bit of a question of personal taste, i.e. are issues just for bugs, or also feature requests, or other ToDo-lists? At the moment, it even contains rather trivial things such as "improve documentation", which is a neverending topic and within issues quickly drops to the very bottom.

• As I wrote on Discord the other week, I have already started looking through all open Issues on BlendLuxCore. This is also a good exercise for myself to get a deeper insight on what kind of topics are outside my current knowledge, what other users are interested in etc. Hopefully it will help to reduce the list a bit.
• We should consider how to use tools like discussion and projects as well. If we want to use them. Same for Discord and the forum: Which platform for which purpose?
• Particularly for Discord, it would be good to assing responsibility to transfer bug reports to GitHub. Discord as a chat can quickly lead to things being forgotten and hard to find.

[howetuft]

Totally agree with LTS considerations. VFX however may conflict with wheel and Github ecosystem (especially compilers versions): to be kept as a guidance, maybe.

I currently solve this by developing the addon in a separate location, and installing only a wrapper addon in Blender (containing only the bl_info dict, a sys.path.insert() to my development folder with a "from x import *" and finally "if name == 'main': register()'" ). A similar workflow with extensions would be nice.

Very good idea! We could package the core of BlendLuxCore as a Python package and just distribute a "boot" module, that pip install the core (and the wheel with pyluxcore). And, for development, make the "boot" point to an editable version of BLC.

Regarding the recompilation of dependencies it sounds like you have that on the list, so not too much comment. I imagine the following in my head, let me know if this matches your mid-term plan:
a) a dependenc manager takes care to collect and compile the dependencies into a repository like the current LinuxCompileDepes etc.
b) A core-compilation script build the core code against these dependencies (like the current LinuxCompile repository)
c) The wheel-build collects the python addon, LuxCore binaries and needed dependency-libraries into a wheel package and publihes it.

a is right. b and c would be slightly different from what you're describing, though. In fact, building the core for development and building the wheels cannot be sequential. Building wheels amounts to building the core for a complete matrix of <platform / Python version> configurations and, what's more, under very standardized conditions (see pypa manylinux on this subject).
b and c would therefore be two disjoint processes. On the other hand, the idea would be for them to use the same build system, which would be able to perform the build for all matrix configurations. The core build for development would just be the instantiation of this system for the developer's < platform / Python version > configuration. This would be fairly transparent for the developer.
Compiling the wheels would be done in continuous integration, classically walking through the whole matrix.

Before working on new core features, it would also be helpful to clear out the GitHub issues.

I totally agree. It's not a task I particularly enjoy, but it's really necessary!

And same opinion as yours about Discord...

[odil24]

Great attention to detail! I can confirm with you guys and especially with bug (issues) fixes...

[howetuft]

@ALL,

This post outlines an action plan to overhaul the LuxCore build system, starting with the wheel maker (https://github.com/howetuft/LuxCoreWheel).

Guidelines for the New Build System

• Keep the legacy build system accessible, but isolate disruptions in a separate branch.
• Focus on the build target pyluxcore.so.
• Ensure compatibility with the following platforms:
  • Linux
  • Windows
  • macOS (Intel)
  • macOS (ARM)
• Support Python versions 3.8 to 3.13 (and newer) — or ensure compatibility with the stable ABI (version 3).
• Build wheels in compliance with the following conditions:
  • Compatible with manylinux_2_28
  • Buildable with specified versions of MSVC and Apple Clang.
• Deprecate C++ samples (e.g., luxcoreconsole, luxcoreui) in favor of Python equivalents: pyluxcoreconsole (already written), pyluxcoreui (to be written), etc.
• Rely on Conan for dependency management.
• Split the build system into three modules:
  1. Dependency Build: Responsible for building dependencies across various platform/Python version matrices.
  2. Unitary Build: Builds pyluxcore for a specific platform and Python version, useful for development.
  3. Wheels Build: Focuses on building wheels via continuous integration (CI).

Build System Action Plan

1. Clean the Wheel Maker Code

   1. Create a new branch in LuxCore and apply all patches (refer to scripts/patch_source_tree.sh) directly as commits.
   2. Remove the hack involving "-reserved.so" in OIDN for Linux (see deps/conan/oidn/conanfile.py).

2. Create the Dependency Module

   1. Investigate the possibility of relying on stable ABI 3 instead of building for each Python version.
   2. Separate the dependency build module from the global build module. Key notes:
      • The dependency module should output conan save results (including profiles, if applicable).
      • Using stable ABI will simplify the output (one per platform).
   3. Document the process.

3. Create the Unitary Build Module

   1. Write installation scripts for dependencies.
   2. Adapt CMake files within the codebase.
   3. Document the process.

4. Create the Wheel Builder (CI)

   1. Adapt Github Workflow to new build system

5. Develop pyluxcoreui

6. Remove Remaining Hard-Coded Dependencies

   • Example: JSON, Optix, etc.

Additional Proposals:

• Bump LuxCore version to 3.0 after this new feature

[CodeFHD]

Only question I have here is for some details on action plan step 6:
Your examples of JSON and Optix sound like something that is needed (e.g. LoL parses json files, OptiX is important for GPU rendering in the Core and available as a viewport denoiser in BlendLuxCore).
How, or for which part of the code, do you plan to remove anything? Is it really a removal or should I understand this as a replacement by the new Conan management?

[howetuft]

The code I'm aiming for is here: https://github.com/LuxCoreRender/LuxCore/tree/master/deps
Yes, I may have misspoken, it's a replacement by a dependency managed by Conan (and whose version you can control...), not an outright deletion.

[CodeFHD]

ok, that link was helpful, I think I understand better

[CodeFHD]

Picking up on some comments regarding automatic updates in this issue: #948

There it is mentioned that latest versions of the wheel will be fetched automatically, i.e. an auto-update mechanism. That is, assuming that I understood it correctly.

Such auto-updates should be carefully considered. Users working on large production projects would not like it if there are changes that affect the rendering output. Doesn't matter if this is due to bugfixes, feature deprecation or new implementations (that could not be done with backward-compatibility with reasonable effort). Of course, we always want to avoid breaking changes, but its bound to happen sometimes.

Same point when it comes to compatibility with multiple Blender LTS versions. This will be covered by separate wheel distributions.

Also for development/bugfixing we need control which subversion to use when analyzing bug reports.

[howetuft]

Yes, you're absolutely right. The mechanism now is an automatic update, but it's not satisfying. Actually, it was not a voluntary design choice, but just the fastest way to code it for a proof-of-concept.
It should be changed for a solution where the user can control the underlying pyluxcore version. You can take control of this topic if you have a clear vision of it, no problem for me.

[CodeFHD]

I have no vision beyond what you described. So no taking control needed at the moment, this is best left to you and I continue with the open issues.
All I wanted is to have this item documented here with the main plan. Given that we agree, all is perfect for me.
(And I see no issue to keep this auto-update during the current development phase. So no rush from my side.)

[odil24]

Picking up on some comments regarding automatic updates in this issue: #948

There it is mentioned that latest versions of the wheel will be fetched automatically, i.e. an auto-update mechanism. That is, assuming that I understood it correctly.

Such auto-updates should be carefully considered. Users working on large production projects would not like it if there are changes that affect the rendering output. Doesn't matter if this is due to bugfixes, feature deprecation or new implementations (that could not be done with backward-compatibility with reasonable effort). Of course, we always want to avoid breaking changes, but its bound to happen sometimes.

Same point when it comes to compatibility with multiple Blender LTS versions. This will be covered by separate wheel distributions.

Also for development/bugfixing we need control which subversion to use when analyzing bug reports.

Plus to it we also need to add release notes features when will be a new releases: what is fixed, whas is added or like so.

P.s.: Blender Projects platform like Gitea has this kind of issues:
https://projects.blender.org/blender/blender/commits/branch/main

[howetuft]

Totally agree with you

[howetuft]

Hello,

Just to keep you updated:

I've managed to remove the hack I had to introduce in OIDN compilation to circumvent a SIGSEGV issue (see above in "Build System Action Plan", point 1.ii). The root cause of the SIGSEGV was the presence of two compiler versions in the build environment (docker container): it happened that memory was allocated by a code compiled by version A and freed by a code compiled by version B, leading to segmentation fault. Once identified, it was easy to fix, by targeting only one version of gcc.
So, point 1.ii is clear, now.

Thus I can move on and start to tackle another point: 1.i "create a new branch and apply all needed patches to build against latest versions of dependencies". I'll call this new branch "3.0", if you're ok.

[howetuft]

Update: After giving the subject some thought, I'm going to open a separate discussion on the next version (number, content, out-of-scope) instead.

[odil24]

@howetuft I accepted and confirmed! 👍

[howetuft]

Update:

I've achieved "1.i Create a new branch in LuxCore and apply all patches" of Build System Action Plan, so 1. is complete. Work has been pushed to for_2.10 branch.

Now heading to 2., starting with Stable ABI topic.

[howetuft]

Update:

The stable API seems unfortunately out of reach at the moment. I've replaced Boost.Python with Pybind11, which I discuss here, and which simplifies dependency management while optimizing the build, both in speed and size. Resulting code has been pushed to for_v2.10.

However, Pybind11 doesn't provide strong support for the stable API. We'd probably have to use Cython, which would mean completely rewriting pylucore.cpp, pyluxcoreforblender.cpp and probably luxcoreimpl.cpp. This seems too time-consuming and risky at this stage. Maybe in the future?

In any case, I'm abandoning this target: at the end, it means keeping multiple wheels (24), but that's not blocking.

Now heading to separation between dependency build and core build in the Wheel maker, but beforehand I have to do some cleanup and simplification in the wheel process and deps.

[howetuft]

Update: A Significant Milestone Reached - Dependency Provider

I am pleased to announce that a significant milestone has been reached in the plan, with the implementation of a dependency provider.

Dependency Provider - LuxCoreDeps

I successfully installed a dependency provider into the project, under the name of LuxCoreDeps. You can find it here:

• LuxCoreDeps GitHub Repository

The dependency provider precompiles the binary dependencies needed to build LuxCore from the source code of each dependency, using Conan's build recipes, and makes them available for core build.

Binaries are compiled via continuous integration (Github Actions), for 4 platforms: Linux, Windows, MacOS x86_64, MacOS arm 64.

Please note that compiling dependencies from source offers multiple advantages, compared to plain binary imports:

• easy version control and upgrade: version upgrades are performed by reparameterization and rebuild
• control over the compilation toolchain (required for Python wheels). Same toolchain is used for dependency and core compilations.
• control of compilation options for each dependency (choice of functionalities, compilation flags, etc.)
• 4 platforms addressed in the same process, thanks to Github Actions. Potential for extension to other platforms in the future.

Once compiled, the approach is to publish the dependencies in Conan format within the repository's releases. To get started, I have created the first release:

• Release v0.0.2

Retrofitted Build System - LuxCore for_v2.10

On the LuxCore side, the build system has been modified to allow the downloading and installation of the dependencies mentioned above. This change is available on the for_v2.10 branch:

• for_v2.10 branch

The new build statements are detailed in the project’s README but, in short, after required build tools have been installed, it consists in:

make deps
make

ATTENTION: this is still an ongoing development

• The build system (LuxCore side) is currently only completed for Linux and MacOS (Windows is under development) and MacOS has not been throughly tested yet.
• There is still much work to be done, including adapting the Wheel Builder to the new system. This will also allow us to test it under real-life conditions.

Ongoing developments will gradually address these points.

Thank you for your patience and support as I continue to improve the project.

Your feedback is welcome!

[CodeFHD]

Great news!
I should find some time tomorrow or on Monday to test it out on Linux.

[howetuft]

Update

Today, I’ve successfully built luxcore, pyluxcore, and luxcoreconsole on Windows, although the luxcoreui build is still in progress.

The next steps involve extending the build process to include luxcoreui, and I also plan to create a GitHub Actions workflow to validate the build process in a clean environment.

Anyway, just wanted to share this progress!

[howetuft]

Update - Luxcoreui@Windows

I've succeeded in building luxcoreui on Windows!
An install statement has also been added (make install).

Next step will be to create a Github Actions workflow to test/validate the build in a clean environment.

[odil24]

Update - Luxcoreui@Windows

I've succeeded in building luxcoreui on Windows!
An install statement has also been added (make install).

Next step will be to create a Github Actions workflow to test/validate the build in a clean environment.

Great! 😃👍

[CodeFHD]

@howetuft perhaps before we close this one (and lose sight of discussions), we should review and incorporate into any new v2.11 roadmap?

[howetuft]

Hello,

Among things we should incorporate in 2.11:

• Replace luxcoreui/luxcoreconsole by pyluxcoreui/pyluxcoreconsole
• Remove Blender from pyluxcore dependencies
  And in long-term plan:
• Base pyluxcore on stable ABI

For the rest, I think the plan above is completed and this discussion can be closed.