Should we allow public workflow runs with non-formal software release?

[mlocardpaulet]

One of ProteoBench objectives is to allow comparison of software versions, but shall we allow public points with non-formal versions?
I see two cases where these versions could be used for generating a public ProteoBench point:

• if the developers of the tool submit an output
• if these version are publicly available for download, they can be used in any lab, and anybody can submit an output. This could be problematic if the version is available with a clear warning that this intermediate version should not be used in production since the resulting point may not reflect performances of formal releases of the software in question.

We need to decide if we should exclude non-formal releases from the public points. In such case, users could always plot these privately.

What do you think?

[julianu]

I find this is a valid point One way could be, that a developer requests the removal because a not-production-ready version was uploaded. Reviewers then would need to check the status of the version and can remove it, if it is listed somewhere, that the version was not production ready, or indicates something like "alpha", "beta" or "dev" in the version tag. Maintaining a blacklist of versions with this info could also be a (parallel) way to prevent future uploads.
BUT it should be clear at any time that this version was never meant to be used in production. Finding out that a version had problems and afterwards requesting the removal is - at least for me - not valid. There will always be errors and improvements, and ProteoBench is actually a nice way to show this.
I would rather like, if the points were only "not shown by default" and also allow the plotting of "obsolete / invalid" points. So, we could keep all info ever uploaded - but also provide a way for developers to retract "wrong" information on the landing pages of modules. But the double-check whether a version qualifies to be hidden is important IMHO to prevent that only the "best" version is shown. Writing this, one exclusion might be "dev" or "nightly" versions. Though even there the info of improvement / worsening might be interesting

[fcyu]

I vote for removing the beta version result, because

1. It is not intended for production use.
2. The beta version is likely not publicly available, meaning others won't be able to reproduce the result. Imagine if someone submitted an exceptionally good or poor result using a "beta" version that no one else can access—how would you handle that? It would distort the ProteoBench plots. In extreme cases, if most of the database points come from non-reproducible beta versions, it could render the entire ProteoBench benchmark meaningless.

Best,

Fengchao

[RobbinBouwmeester]

IMHO the status of the tool should be clearly stated in the version tag. Personally, I think there is value in allowing beta (or alfa) versions as people would like to know how the tool performs. Lets be honest, beta tools are being run in production in proteomics workflows. Especially if they provide a substantial benefit over stable versions or people are less familiar what a beta version means (let alone the ambiguity in the definition of what constitutes a beta version anyway).

For me removal of points from a beta version is fine, removing old versions is not. Personally I would also only remove points after a developer or maintainer of the tool requested it. Performance of beta (alfa) tools can still be relevant.

I do strongly agree with Fengchao that it is very desirable that the tools version is publicly available.

[mlocardpaulet]

After discussion, we won't check ProteoBench submissions to identify non formal versions. But what we can do is remove points upon request of the tool developers. For example, @fcyu , could you let us know exactly what points were made with non-accessible versions of FragPipe that were not intended for production use?