Manual package compatibility result reporting

[daveverwer]

There are many reasons that our build system could fail for reasons that are hard to fix either by the package author or by us. One came up today, but I'm sure more already lurk undiscovered in the index.

The compatibility matrix is so prominent on the package page that it puts those packages at a disadvantage, so I wanted to discuss the potential for a manual compatibility statement.

If (and it's a big if) we do this, we should tie the results to package versions just as we do for the automated build results. This means that package authors will need to update their compatibility information manually with every release. However, giving them that task means it's more likely that results (if they exist) will be current.

I don't think we should allow compatibility for a branch, only the latest tagged beta and stable versions. We should also put a note close to the compatibility matrix indicating it's manually set rather than automated.

I can see a new key in the .spi.yml file that could work for this:

compatibility:
  - version: '1.0.0-beta1'
    platforms: ['ios', 'macos', 'linux']
    swift_versions: ['5.6', '5.5', '5.4']
  - version: '1.0.0'
    platforms: ['ios', 'macos', 'linux']
    swift_versions: ['5.6', '5.5', '5.4']
  - version: '0.9.5'
    platforms: ['ios', 'macos', 'linux']
    swift_versions: ['5.6', '5.5', '5.4']

Naturally, this will never be as good as automatic results, but it's better than no results. Or is it? That's the discussion 😂

[finestructure]

I was thinking the same with respect to the package info being at a disadvantage with builds that cannot be fixed.

The problem with allowing manual configuration is it's easy for maintainers to just switch it on and forget about it when compatibility changes.

I think what we should do instead is allow them to disable the matrix entirely via .spi.yml. Since we show the package readme just below, they can embed whatever badges or other compatibility info they already use to highlight their status just like they do on Github. By switching off our matrix we'd allow that information to come front and centre.

[daveverwer]

The problem with allowing manual configuration is it's easy for maintainers to just switch it on and forget about it when compatibility changes.

This was why I proposed making it a per-version key but I didn't go into enough detail. If the version in the spi.yml is not the current stable/beta version, we'd not show it. This would mean they need to update the key with every release to keep it current.

I think what we should do instead is allow them to disable the matrix entirely via .spi.yml. Since we show the package readme just below, they can embed whatever badges or other compatibility info they already use to highlight their status just like they do on Github. By switching off our matrix we'd allow that information to come front and centre.

This is a possibility, but with the current page design, the version information on the right-hand side would mean a huge gap where the matrix is. We could mitigate this by using the space for a message explaining that the package author has opted out of automatic compatibility results.

[finestructure]

Alternatively, we could translate some compatibility info from .spi.yml into something clearly marked as being maintainer provided and distinct from our matrix so there's no confusion as to whether it's actually verified compatibility. It could be like our badges currently work, just automatically shown on the package page.

[lukasz-szyszkowski]

Manual changing could make problems as @finestructure mentioned, but how about reading Package.swift file header (swift version) and platforms property array? It's not an ideal solution because we are not 100% sure that build will pass but IMO it's something between manual and automatic versions matrix.

[daveverwer]

Ironically, that key was our first attempt at getting this information, but it's really not ideal for a couple of reasons.

First, the key is intended to be used for something a little different to indicating package compatibility. According to the docs:

The version of the Swift language to use for compiling Swift sources in the package.

Then, the values for the enum are problematic as they only cover v4, v4.2, and v5 values. Yes, there's an arbitrary string value, which people could use, but it's not great.

I think if we did allow manual specification, I'd rather not abuse that key for a different purpose and do it via spi.yml instead.

[lukasz-szyszkowski]

Hi, any movement in this thread? I mean is it possible to set the compatibility matrix manually yet?

[finestructure]

Well, it's not a particularly simple thing to add and it's not even entirely clear what the best way of doing it is... so it's having a hard time jumping the queue of other things we're prioritising for the moment :)

[daveverwer]

One thing that might be worth mentioning is that this is an open-source project and if something is important to you then we would love to work with you to get it implemented.

If this is of interest, I'd suggest we start simple and then work up from there if you enjoy contributing after getting the first step implemented.

So, for step one I could see that we'd just allow package authors to replace the compatibility matrix with a short text message via a key (or two) in the .spi.yml file. My first thought was that we could let the package author specify that message (including describing the compatibility information, if they choose to), and while I'd have some concerns about letting someone insert random text into the site, we can make it a simple string (not HTML) and we also include user-generated content in the README file, so I shouldn't fret so much! 😅

I'd say that's a pretty reasonable step one for this feature and from there we can look if people are using it and if it's popular maybe make something more structured from there.

Is this of interest, Łukasz? If so, I'll write up an issue and we can get started. Oh, and if you want to discuss it then we have a Discord where we are online and happy to discuss it and give you what you need to get up and running.