Describe/clarify releasing processes

[okybaca]

I wonder, if we could describe or clarify the releasing process of YaCy, to make the steps reproducible and understandable, ideally with command examples.

I understand that there is a lot of work with the releasing process for @Orbiter, and I hope the description would help to ease that or to find help from us others.

Ideally, releasing.md, describing the process would be present in dev-docs, with this discussion serving as a inception of that.

There is also a lot of user calls for releasing in Github Issues, I tried to tag them all:
releasing related to release process, artifacts

Web

Latest version of YaCy published in "Download" section for Windows is 1.924, latest version for Mac is 1.930. That's IMHO the reason, why 28% of instances use old 1.924 and 15% the 1.930, according to the yacystats:

https://yacystats.de/d/edquz3nqqr11cc/yacystats-de-dashboard?var-Interval=$__auto&orgId=1&from=now-7d&to=now&timezone=browser&var-Peers=$__all&refresh=5m&viewPanel=panel-73

breakout by minor versions:
https://yacystats.de/d/edquz3nqqr11cc/yacystats-de-dashboard?var-Interval=$__auto&orgId=1&from=now-7d&to=now&timezone=browser&var-Peers=$__all&refresh=5m&viewPanel=panel-22

Linux

Latest Linux release linked from "Download" section is 1.930 too, but there is also 1.940 build present (and not linked) at downloads.yacy.net: https://download.yacy.net/yacy_v1.940_202405270005_70454654f.tar.gz

Is that signed? How to find that and check the signature?
Should the link for Linux lead to newer version?

Auto update feed

There are two main kinds of releases used in YaCy's in-built update feature:

• signed (by @Orbiter)
• and unsigned (auto-built by github?).

If I understand that right, the updates are platform independent since it's java and the only thing platform-specific packages is neccessary for is the installation. (Would java version change break that, for example for Windows user?)

Sources are described in defaults/yacy.network.freeworld.unit as download.yacy.net and release.yacy.net. Is the difference between them, that the earlier are signed and the latter are latest/developer/unsigned build?

At release.yacy.net, the latest build is 2024-12-02. How the releases are build, by hand, or is that a github's compilation? How they're updated? Is that so that the releasing of latest builds could be automated by a script?

In a case that someone wants to use the latest build, the only option is to build by oneself now.

I suppose that from some reason (privacy/trust?) we don't want releases from github directly, since the line:
# network.unit.update.location2 = https://github.com/yacy/yacy_search_server/releases
is commented out.

Docker

I personally don't understand Docker at all.

From what I see, Docker images are published at dockerhub:
https://hub.docker.com/r/yacy/yacy_search_server/
for architectures: linux/amd64, linux/arm64/v8 (11 months old) and linux/arm (4 years old -- and issue with patch)

How the docker images are built and published? By hand? How? What's the process?

It seems that building arm image is complicated (@Orbiter using raspi), could that be cross-compiled? Does anyone have experience with that?

• effort to do automatic multiarch builds by @MisterX2000: Automate Docker Build Process with Multiarch Support (and GitHub Release action) #698 (draft)

Windows

Latest Windows release is 1.924.
@smokingwheels did some effort in that direction, with @thkoch2001 involved. From what I understood, the problem is in using non-free java runtime and the obsolete NSIS packaging system. Does anyone know some viable alternative to do that?

Mac

Latest macos release is 1.930. How is that built, what is the process?

Not sure, but that seems that macos releasing is stuck on compilation system decision ant/gradle/maven and that macos release was dependent on gradle before? Is that right?

Debian/distro packages

There used to be an official Debian package at debian.yacy.net, but is not any more (#681) (althought referenced from the old documentation, transfered from the old wiki. May act as a linux distribution packaging inspiration).

@jason1987d offered to create a package for Linux, for that he needs to tag the releases (since packages got to refer to a tag rather than a github commit). That costs us nothing and would ease the packaging process done independently by other people, or various distributions of linux/bsd, packaging systems etc.

I identified the commits where the version changed, but since I have no rights in the repository, I cannot tag them. @Orbiter, could you tag these please, so we don't block such volunteer effort?

[Orbiter]

Hi @okybaca sorry for the late response but this is a relevant question that I also ask myself to find the best solution; what we have now is partly broken and not well maintained. During the last years I was involved in many other projects (loklak/susi) and also in my day-to-day job. So the primary focus should be here to implement a release process which does not need manual work. Below I answer to your points, but those answers are not really solutions; I will describe how we got there and I am happy for suggestions how to proceed there in a proper and elegant way.

Web

We had download links on the web page to make it easy to anyone without much technical knowledge to pick the latest release while releases had release numbers. If you want to make releases based on "here is a release directory, just pick the right one which is the latest and most appropriate for your system" then we might not get users who are used to the shortest path. However, this strategy needs a lot of manual management and I got tired of updating the home page with the right links.

Release Names

The root of a manual release process is the naming of the release. It was a common practice to name software with a release number and giving a release a new release number had impact on the release management, like things that existed before devops like "Code Freeze" and so on. We practices that and I still have that in mind when giving a new release number. I don't want to do that because it is simply silly. New release numbers are also marketing tools, to be able to write a press release and so on, we don't need that. If you require a release number for a bug report then a build number would do as well. So I believe we should move away from release numbers and use build numbers only.

I already prepared a possible transition to build numbers with this commit: 60c9986 which introduced the release date and commit hash into the release name. So to get to a build-number based release, we can move from

<property name="stdReleaseStub" value="yacy_v${releaseVersion}_${repository.revision.date}${repository.revision.time}_${repository.revision.hash}"/>

to a string like

<property name="stdReleaseStub" value="yacy_<releasename>_${repository.revision.date}${repository.revision.time}_${repository.revision.hash}"/>

where <releasename> would be a constant name that follows a to-be-defined strategy. Such a strategy could be that we use a branch name (that would imply that we have a branch strategy like "dev", "stage", "prod" which we don't have) or a main release name like it is done in debian (which I not like, that is just confusing). So I don't know yet about that releasename, but I definitely want to walk away from release numbers. Maybe we just use "search_server" as release name, then the release has the same name as the github repository.

Web Release

I see some options here, all of them must be discussed under the aspect how to implement them without manual intervention:

• some deployment process may be able to maintain the web page and update the download link to the current build name. While this is technically possible I personally don't like that. It would create a commit to the documentation for each commit in the source code, that would be silly. However, I don't see any other way if we want to enable users to directly download the latest release from the link on the home page.
• we link to a release directory instead where the user can pick the latest build. That would make it more difficult for new users to get the best one. However, we can reduce the possible risk to pick the wrong release by cleaning the release directory frequently (automatically). I don't like that either.
• we keep the direct download of the latest release visually, but produce that link within the web page using javascript that takes the right link by retrieval of some back-end service. I like this but we need to find out how to do that within a new deployment process.

Linux

That is the same question here, this is just about the download of the tarball. Some project don't bother at all and just link to the github repository where users have to find the release, for us that would be https://github.com/yacy/yacy_search_server/releases
That rises the question why we don't have that and why we don't have releases there. The answer is: I did the release build on a root server in a remote github runner because the build process was difficult before I moved the http servlets to the main class path. Maybe we should set up just an automatic build process to make releases automatically. Then link to the latest release as hosted by github.

Auto update and release signing

Yes I did release signing many years ago and it was a pain because always there was something to be done manually. At some time the automatic signing process died and did not fix it, so it is dead. The automatic build process was done by a github runner process on my root server. See above, we should replace that with a github-hosted process. I will not do any signing on my server again, we should find a way that trust is enabled through the github build process.

Docker

The Docker build process is also done with the github runner on my root server:

• the docker release was supposed to replace all linux-distribution related releases because there was no maintenance for this. I still strongly support the idea that docker is the right way to go for all systems, not only linux but also for Mac and Windows this is the best solution. So I want to keep it.
• in the past it was easy to produce docker images at dockerhub because they just did the production of the docker images, auto-build is documented here: https://docs.docker.com/docker-hub/repos/manage/builds/setup/ ..however at some point they stopped that and turned it into a paid service. https://www.reddit.com/r/docker/comments/nv54fj/docker_hub_discontinuing_autobuilds_on_the_free/ Maybe we can do auto-builds with https://github.com/marketplace/actions/build-and-push-docker-images

Windows

I don't know how to make Windows Releases and we are stuck here with the current attempts. Maybe there is a new way to package windows software for java applications. Until we have that, we should abandon the windows release and tell windows users to simply install java and extract the tarball, the click on startYACY.bat. It cannot be that difficult to do that.

Mac

Apple continuously bans unsigned applications and makes it almost impossible to start applications that does not come from the apple store. I am a MacOS (not iOS) user myself and YaCy runs on my mac from the tarball or from the development environment. We can also abandon the Mac release as for Windows and tell the user step-by-step to untar the generic release to run YaCy from to console.

Debian/distro packages

I am happy for every supporter who does such kind of packages, but we should not integrate them as official releases but as contributed releases simply because I don't have time to maintain these contributions as part of the release process. Maybe we can maintain a list of contributers and their repositories where YaCy users can take their work.

Next Steps

I don't know. Lets find out what would be the right way to set up an automatic devops process. Your suggestions?

[okybaca]

Great, Orbiter, thanks for replying and for moving on!

Now just a remark regarding the release numbers: does it make sense to keep at least some version numbers, reserved for big & incompatible changes like using another solr or java version? Now, it's quite straightforward, that each 1.X changes with the major solr version -- and IMHO it's pretty clear&understandable.

Debian/distro packages

Yes, for sure! No additional maintenance for us, but let's clear obstacles for people, who want to do that on their own. Now, the obstacle is relese tags. (#665, #348)

With Windows & Mac
The more easy installation is, the bigger adoption & size of the network.

Understand that Windows is pain.

I can imagine that installing Java and undtarring tarball and/or cli-running yacy could be deal breaker for plenty of "regular" users.

Was MacOS build dependend on graddle? Friend of mine, in need of IDE debugging, resurrected gradle and it worked, so this way is still not closed probably.

[okybaca]

the point with version numbering is also that (if i understand it well), you cannot upgrade solr 2 versions up. the users should be instructed in upgrade interface in yacy and major version numbers are good clue here. anyways, we probably need either: * stable and dev branch, * or use "stable points" in commit history, such as version change & tag -- following commits could be 'dev' to keep things simple, until next 'stable' (tagged release). ad releases on the yacy web: latest auto-build could be just latest.tgz, to keep links stable, and a stable release time to time.