(Linux) Packaging

[dvzrv]

I'm currently trying to package sonic-pi for Arch Linux' [community] repository.
However, this proves extremely painful, because of

• conflicting installation information and about a dozen different "package scripts" for various distributions (that all seem slightly different and/or outdated)
• four (!) different make systems (qmake, erlang, something that should probably be ruby bundler and cmake) in use to create (but not install!) this environment
• no FHS compliance (which is bad as distribution usually try to avoid dumping applications in /opt)
• no XDG desktop entry integration

This all makes it very hard to package sonic-pi without major headache.

It would be very beneficial to have a Make-like system, that takes care of all of the building and installing, as maintaining a package with a lot of customization and workaround steps, such as this, is very time-consuming and therefore off-putting to any packager.

There are a few open pull requests, that should be followed through with (e.g. #218, but especially #1291), as they improve on exactly this issue.
However, I don't think the latter approach should be chosen for e.g. building the Qt interface. If a standard GNU make is chosen, this can use rake, build the GUI and place things where they should go.

According to FHS, all things below etc should actually end up in /usr/share/sonic-pi, the executables below app in /usr/bin and so on.

I'm currently trying to find a way to make sonic-pi FHS compliant, but I'm not sure it's possible in its current form.
I'd definitely appreciate any help and pointers!

[samaaron]

@dvzrv you make some excellent points :-)

It's also important to point out that Sonic Pi has never really had a strong focus on making things friendly for developers. All my energy has gone towards making the end user experience as simple, intuitive and as fun as possible. Given that the majority of Sonic Pi users aren't developers, this felt (and continues to feel) a the best focus to have.

However, this has meant that the development side of things has a lot of room for improvement. I'm totally in favour of this - and if you're able to help out in any way, that would be amazing.

Let me address each of your points individually:

conflicting installation information

Currently we don't have the resources to support generic Linux distros. We are able to support Raspberry Pi because it is a single hardware and software platform and there are fewer moving parts. As you can see with all the conflicting installation info - Linux isn't consistent across distros which gives rise to these issues. What you see is the result of lots of people trying to get things working on their particular linux distro and the resulting info is a bit of a mess. I'd love to hear any thoughts you might have on unifying and simplifying this in a way that works for all linux users.

Multiple make systems

As mentioned previously, simplifying the creation of the system has never been a goal. The goal was always to have a single point of installation that anybody could install. On macOS this is an app bundle and on Windows it's the msi installer. How that is created can be magic from the perspective of the user. I'm happy to look into solutions for this, but it would have to be something that was cross platform and worked just as well on Windows and macOS.

No FHS compliance

Again, generic linux has never been a priority. I hadn't even heard of FHS before. I believe @hzulla did some work on this for the previous Debian release - although I'm not sure how that related to the current file structure.

No XDG desktop entry integration

Also, I have no idea what this is and due to Linux not being a target platform it was never a priority. If you see a way of integrating this without affecting current platforms then I'd love to know what might be involved.

In conclusion, I completely take your points and would love any and all help improving these matters!

[dvzrv]

@samaaron thanks for getting back on this so quickly!

Well, I guess there are many possible entry points (apart from a make-system automating the installation process later... and if you use cmake, you can do this cross platform!).

The first one to tackle is probably to have a look at util.rb and see how paths can be separated out and/or made global (as especially on Linux, the philosophy is to not bundle everything, but make software as generic and reusable as possible).
It would be great to find a way to do this, without patching!
This would mean to set global paths for the Qt GUI, sox (which is not directly mentioned as a dependency btw), osmid (now packaged for Arch at least) and a bunch more I probably forgot.
Generally, I think that the application should be more reliant on PATH (as this should be set for all sane Linux distributions).

Writing this, I realize, that one (very easy) way to get around the whole "multiple installation instructions" mess is to only provide a single location in the repository, where dependencies (and their minimum versions) are listed. Installation instructions should be moved to a wiki.
This would already minimize the noise significantly.

The XDG stuff is basically only a bunch of flat files and pictures, that have to be installed to certain locations (e.g. /usr/share/applications/sonic-pi.desktop, /usr/share/pixmaps/sonic-pi.png, etc.). So that's straight forward and can just be not installed on macOS/Win.

The Qt GUI should get an install target. I'm not a huge fan of cmake, but I think it will get the job done (for all platforms). Another option could be meson. In any case, it would be great to drop qmake!
Install targets have to be applied to all components of sonic-pi. This means, that a make system will take care of building (also help files) and placing of built files, while discarding the (unneeded) source files. Currently everything is moved (which is messy and unneeded).
This would make all install scripts obsolete (e.g. those in app/gui/qt)!

To really have a deeper look into this, I need to understand a couple of things first though:

• why is there a part written in erlang? Was this not possible in ruby? It needs compilation and then be moved somewhere, where executables reside (e.g. /usr/bin), while settings its PATH in util.rb.
• What is the usual entry point for the application? I assume it's the Qt GUI, right? To be able to place it below /usr/bin, its internal pointers to the ruby part of the application need to be made absolute (btw. in mainwindow.cpp I wonder why most of the includes from mainwindow.h are done again and then mainwindow.h is included?!) before compiling it.
• How does app/gui/html relate?

There are probably more things, but sieving through this project is a tough cooky! I need a break ;-)

In any case: Using a cross-platform make system such as cmake or meson would be of benefit and improve the current situation a lot.

[hzulla]

Hey there. I did address some of these things in the Debian packaging. You can see my packaging choices described in the Debian source package's README.

[hzulla]

After the Debian packaging, I had tried to make all compontents of Sonic Pi using a Rakefile to simplify building all dependencies. That effort ultimately failed, partly due to the rapid speed of the development compared to my ability to get the Rakefile done.

Sonic Pi is a collection of a wide range of software components (core in ruby and erlang, gems in ruby and C, synths in clojure, GUI in C++/Qt, help documents from Markdown, etc.), some of which require specific patches curated by @samaaron.

Distributions usually think differently about software packaging, they try to make everyone use standard packages wherever possible and reject those who don't follow that advice. My Debian packaging attempt took most time where I tried to get Debian to upgrade outdated software dependencies, de-couple the ruby gems required by Sonic Pi, use those delivered by Debian instead and/or to convince Debian that Sonic Pi requires a specific earlier variant of a gem.

Looking back, my best guess would be to avoid using traditional distribution packaging rules altogether and move to Snap / Flatpack instead. (That still wouldn't help on Raspbian, though.)

[hzulla]

All in all, Linux packaging is terribly frustrating for a large and interwoven software package like Sonic Pi. It requires bureaucratic processes and the willingness to adhere to each distribution's specific quirks and rules. There's a reason why it's easier to get software distributed to Win10 / Mac.

[hzulla]

As mentioned in #1291 I'd argue for using a Rakefile as a viable cross-platform build tool. It works on all platforms, you can adapt it like hell, it is written in ruby, which is a scripting language that you will install on your computer anyway if you want to build Sonic Pi, so it is a very small dependency compared to cmake / GNU Make / autotools / whatever.

[dvzrv]

@hzulla Thanks a lot for the README pointer! I'll take that into account when trying to build my package!

That being said, my suggestion with cmake/meson is my personal bias of things I've seen and worked with so far. I honestly can't evaluate, if ruby rake makes more sense in this context, as I don't really use ruby. But it sounds like a great approach! Anything that betters the modularity and lowers the complexity of the project is good! :)

I'll have to evaluate, if it makes sense for me to patch the installation into position (time wise probably not ;-) ).

[dvzrv]

I have to add:
Many other projects, that I have packaged so far have managed to achieve decent cross-platform builds (e.g. supercollider).
It is possible! It's just not very easy, when the project was not started that way (which is not to say, that it shouldn't be tried!).

[hzulla]

@dvzrv do note that my Debian packaging was made before the Erlang component became part of Sonic Pi.

[dvzrv]

@hzulla noted!
@samaaron which components below etc are in use in version 3.1.0?

[samaaron]

@dvzrv most of the stuff in etc is used at runtime:

• buffers - contains a wav file that drives the random system
• doc - only used at build time
• examples - also only used at build time
• samples - all the built-in audio samples
• snippets - hidden feature, but planned to be used in the future
• synthdefs - contains both compiled synths (required at runtime) and the designs to create them
• wavetables - not yet used (@xavriley was looking at getting these working a while back)

[dvzrv]

@samaaron thanks for the info!

I'll check, how much work it'll be to do the patches, and if they can be easily transferred to other distributions (which I think they should).

[dvzrv]

Okay, my first attempt can now be used on Arch Linux.
I have only done the patching for the GUI and the ruby part (and most likely I forgot stuff), but no real testing (in a use-case situation) yet.

As you can see in the first patch: It's not that much. The second one however is a little more extensive and it would be great, if the paths could be set in a more useful and unified location.
It would also be great, if the main ruby scripts could be moved to /usr/bin, so the PATH environment variable can be used, instead of a long hierarchy of folders to find things.

For "kind of" FHS compliance, I had to move files to other directories than intended by the project, which can be seen in the package() function. Those make some base directories obsolete and move everything to /usr/share/sonic-pi or /usr/share/doc/sonic-pi

[samaaron]

@dvzrv - this is great.

Do you have any suggestions on how we could modify the current directory hierarchy to make FHS compliance a little easier?

[SunderB]

Maybe we could have some config file to specify where files and folders are located?