Conan package

[helmesjo]

I'm writing a conan recipe for Magnum (already got one for Corrade successfully building). I've wrestled a few days now with these issues (each try takes a lot of time) so I'd thought I'd look for some help/ideas here!

Issue

I'm building conan packages for GCC, Clang (etc.) inside docker containers.
In the conan recipe, prior to building magnum, it installs libgl1-mesa-dev (or libgl1-mesa-dev:i386 respectively) on the host. But when later building, it either:

1. Fails to find OpenGL libs when targeting x86 (for all builds on Ubuntu) if only installing libgl1-mesa-dev:i386:

   CMake Error at /usr/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:137 
   (message):
       Could NOT find OpenGL (missing: OPENGL_opengl_LIBRARY OPENGL_glx_LIBRARY)
   

   Above can be fixed* (for all but Ubuntu Trusty) by installing both libgl1-mesa-dev & libgl1-mesa-dev:i386 (in that order, else 1. happens again), but then it:
2. fails to link OpenGL (error adding symbols: File in wrong format) on Ubuntu Bionic & Cosmic when targeting x86 (it appears link the 64bit lib).

*There are a lot of issues with multiple mesa-dev packages on the same host.

Details:

• CMake 3.12
• Ubuntu x64: Trusty (GCC 4.9), Xenial (GCC 5), Artful (GCC 6 & 7, Clang 3.9. 4, 5), Bionic (GCC 8, Clang 6), Cosmic (Clang 7) (using the conan docker images)

So, any ideas? Anyone seen similar issues? Am I missing something obvious?

[mosra]

Yay, Conan! 😍 There's an in-progress Hunter package as well (#298), so with this it's complete :)

If I see correctly, this happens only when cross-compiling from 64b to 32b, right? In my experience, building natively on either 32b or 64b never had any issues (in particular, I have a Travis CI build and it also needed just libgl1-mesa-dev).

I wanted to suggest looking at how other CMake-based packages handle this, but seeing your other open issues (bincrafters/community#602 / bincrafters/community#603), it seems you already tried that. Other ideas:

• (I have no idea how Conan works, but) is it possible to supply location of these libraries explicitly? That's the final solution when everything else fails and always did the job for me. Magnum doesn't need the headers, just the libs.
• FindOpenGL in CMake 3.10 switched to preferring GLVND, if found, and I'm opting-in for that. The error message looks like it tried to find GLVND (so libOpenGL.so + libGLX.so instead of libGL.so) and failed. I have no idea what Ubuntu ships, it's possible that the older versions don't supply these libraries at all. However, in that case FindOpenGL should fall back to the old thing if I understood correctly. Maybe something related to the cross-compilation where FindOpenGL assumes something it shouldn't? Maybe the compat 32b libraries don't ship the GLVND things?

[helmesjo]

Yeah, conan is a dream come true (once the packages are in place)! 😇

From what I can tell (I've been staring at this too hard recently, so my brain is pretty cooked), yes the issue is only related to cross building from a x64 host to x86. Building natively is probably a sane path to go considering all issues around OpenGL libs in general (been seeing a lot in my google quest). I'll probably try this first.

1. Yes, conan recipes are just the glue for downloading, building & packaging software, and written in python so patching/fixing up of any sort is rather "trivial". For example, find_package allows "hints" of where it should look first. I could replace all find_package related to OpenGL with an equivalent that also specifies the path. Then it's just a matter of reliably obtaining the correct path (shouldn't be too hard in linux I assume).

2. Yeah, something is probably off with the FindOpenGL-script regardless. It should be "target arch"-aware and select the correct one... But those scripts are all just "naive brute force locators", so I guess one can't expect too much magic... 🎆

Thanks for the feedback! I'll keep digging!

[mosra]

I think it should be possible to dig the problem out of the FindOpenGL script and fix it (after all, the core of the problem is just a file not being found in this particular crosscompilation setup), but then:

• if I submit a PR to CMake, it'll be a year or so until it gets integrated & released, not to mention conan will probably not jump to latest and greatest CMake right away
• I could maintain the forked/fixed version inside magnum, but this stuff is so complex right now given GLVND and all that stuff that I'd rather rely on CMake devs having the responsibility

I think the more straightforward way would be to just build natively on 32bit. Besides that, didn't Ubuntu ditch 32bit support for the latest release already (19.04)? I heard something related to that.

[helmesjo]

I agree, and did just that! Now we'll just have to wait and see if this goes from red to green! 😊

[Croydon]

Hi @mosra!
Nice to see that you are interested in Conan as well!
Would you be interested in providing official support for Conan once the recipe is stable?
We can help you to setup everything which is required if you want 😄

Otherwise, I believe @helmesjo is going to include the recipes in @bincrafters and we are maintaining it there, but obviously official support is always nicer 😄

[helmesjo]

Alright, so native builds did the trick. Closing this for now!

[mosra]

@helmesjo yay!

@Croydon I'm afraid I don't have the capacity to provide official support (as in maintaining them myself), I'm already offloading Vcpkg and Debian package support to people who are using those platforms directly -- because they can spot the issues much easier. What I can do, however, (and what I'm usually doing), is reviewing them to ensure they provide all desired options and features (and also pinging you in case a new version is about to get released).

[Croydon]

You could still ping us at any time when either CI or users are complaining about problems, so it would be more reviewing then maintaining by yourself. We (as in Bincrafters) kinda have this relationship with the maintainer of Flatbuffers, as he doesn't know much about Conan as well: https://github.com/google/flatbuffers/commits/master/conanfile.py

But if you don't feel comfortable with this is also fine and we will go ahead at @bincrafters 😄

[mosra]

Right, so what the stuff I need to do for this semi-official support? :) Looking at flatbuffers, I see just conanfile.py there + a large build matrix in .travis.yml, while https://github.com/helmesjo/conan-magnum has many more files. If it would be about just maintaining conanfile.py directly in this repository, I'm all for it. This also means if there's a problem, it doesn't depend just on PRs to @helmesjo's repository, but anybody can just pick it up from the main magnum repo and fix it.

But ... the problem I see here is that I already have quite a huge build matrix (see here: https://magnum.graphics/build-status/) and adding a dozen of other jobs just for Conan will make it unusable, since the builds would take forever (it's already bad enough with AppVeyor taking 2+ hours). Is there a way around that?

[Croydon]

The amount of files is depending on the question if we would use your CI or another one. If it is only about minimal Conan support without CI support, then this would be conanfile.py + test_package https://github.com/helmesjo/conan-magnum/tree/stable/2018.10/test_package

All of the other files are only needed to get the CI + publishing going (see https://github.com/bincrafters/conan-templates-upstream as a reference)

We could maintain conanfile.py + test_package (= 4 files in total) here and use our Bincrafters CI and repository to publish the build packages

[helmesjo]

Just the last piece of the puzzle... (semi-related to this issue so I hope it's ok to ask here 😇 ):

I fixed the link order, and also added OpenGL to the link list (all of this is applied to packages depending on Magnum), and in the process added a line in the test package that verifies all is linked correctly (this is like a mini-package depending on Magnum & build+run by conan after building Magnum, just for verification).
This worked dandy for all but MacOS, which just anonymously fails when executing the test package...

@mosra Do you know of anything else that is required on Mac? A package that needs to be installed? Some other framework that needs to be linked/added for Mac? Could this be a Travis+Mac thing?