Documentation shield with modified date.

[mattesmohr]

Heya,

I have an idea and it makes totally sense in my head, but... I am not sure, if it does for others. Anyway, I think it's worth a shot.

So... the idea is about a documentation badge (shield) with the latest modified date of the documentation. Like I have tried with HTMLKit.

Maybe it is just me, but often when I am reading a documentation, I want to know, if the documentation is up to date or at least when were the latest changes. Having this information helps me to decide if it is worth to read or not.

Would it be interesting? Would it be technically possible?

[finestructure]

Hi @mattesmohr , thanks for the suggestion! That's an interesting idea and I think that should be feasible. We should have all the information available to generate that badge just like we do for the compatibility badges and it'd be a nice way to enable links to the documentation we host also on Github. 👍

[daveverwer]

Apologies for the slow response to this @mattesmohr. I also haven’t discussed this with @finestructure, so we may not be completely in alignment on it.

I agree with you both that it’s a good idea in principle, but I do have some doubts about it. I’d categorise these under two points:

1. What is the date we use? I don’t think you’re suggesting using the date of the package version release, as that would always be up to date and wouldn’t really show anything useful above it being a simple “Yes, this package has documentation” (more on this later). I think what you’re suggesting is the latest modification date of any file that’s documented? I don’t really see how we can figure that out, given that docs can be in source files that could easily be touched without any documentation being improved.
2. More importantly, is the date of documentation updates important? Clearly, having up-to-date documentation is good, but I think at this stage of DocC adoption, a simple flag is a great indicator. I don’t want people to feel like they need to touch a file just to keep this flag up to date.

My gut feeling leans towards making this badge, if we implement it, a flag. I would be much more comfortable supporting the SPI generating a “Documented” badge that displays a tick. If we had documentation coverage stats, that might make a good second step, but I think I come down on the side of not making this a date.

This is just my opinion, and I only speak for myself rather than the project as a whole, but here are some ideas based on what I just said:

1. “Documented” badge with a green tick for yes and a grey or red cross for no. No one would add the “no” badge, of course, but we should support the situation! The tick would turn green if either we have hosted documentation or external documentation for a package.
2. “Documented: [version]” badge where [version] is the latest stable release tag or branch name if no stable releases are documented.

Of the two, I’d suggest number 1 for now.

Is this something you’d be interested in helping to contribute?

[mattesmohr]

Haha :-D, sorry. I meant the green, grey or red tick, Dave had mentioned. I think, I do not understand, what you mean with hosted or external documentation. For me 'hosted' means SPI hosts the documentation. But isn't that always the case?

[mattesmohr]

Or if someone decides to put the spi-documentation badge into his readme - what I proudly will - , then I assume the spi-hosted documentation already exists.

[finestructure]

We support both hosting people's documentation as well as linking to their own hosted (external) documentation. This can be configured by authors in their .spi.yml file.

The documentation button on the package page will then resolve to either our or their own documentation. I envision any badge for the Readme to effectively behave exactly like our Documentation button on the package page, just with "badge" styling.

[daveverwer]

The tick was more about having something to display if someone puts the badge on their README when they don't have documentation (either hosted by us or external that we link to). The tick or cross was to indicate whether we are going to link to docs for the package.

I guess we could just have a badge that says "Documented" and if it links to a 404 page, that's the author's concern. That makes this feature trivial to add. It'd just be an extra box on the "Information for Package Maintainers" page.

[finestructure]

We just show "unavailable" as the badge text in that case. Could be a useful way for authors to spot if there are any issues.

[maxxfrazer]

I had some documentation badge suggestions, so figured this would be the closest open discussion. Let me know if this would be better off as a separate thread though.

A documentation shield should have some indicator of the level of coverage shown. These are the details that can currently be fetched with docbuild:

With 9 percentage values, there's too much information for a badge. As pointed out by @heckj on Discord, the values do not include code samples in Articles or Tutorials, only those inline and in Extension Files. Articles and Tutorials should be a huge metric for valuing quality of docs, so even if it's an absolute number that would be good to have on the badge.

Back to just general coverage; not all of the above percentages should be seen as equal IMO. If these were to be combined and put into a special SPI weighted value, these would be my suggested weighting order:

1. Types x Abstract
2. Members x Abstract
3. Types x Curated
4. Members x Curated
5. Types x Code Listing
6. Globals x Abstract
7. Members x Code Listing
8. Globals x Curated
9. Globals x Code Listing

The issue of low-quality, one-liners that push up coverage is a slight issue, but even a project with a bunch of one-liners is better than a project without them in most cases I'd say. The difference between a project with 20% Abstract coverage vs 80% would be clear, regardless of how long the inline docs are.

I'd love to get others input on this, especially as I have my biases when it comes to the weighting order posted above.

[daveverwer]

Thanks for continuing this discussion, Max

I’m not sure I like the idea of trying to distill this down into a single percentage value, or indeed any numeric value. A percentage indicates something absolute, and as you point out, even the 9 percentages that DocC gives do not cover how well something is documented.

Don’t get me wrong from saying that. I think that report output from DocC is useful, and I wouldn’t be against capturing those numbers and storing them against a package for display somewhere on the package page. I’m just not sure we should be messing with anything that produces a percentage. It sends all the wrong messages. A 100% rating (whatever that would mean) shouldn’t even be a goal, I don’t think.

I’d be slightly more in favour of a “1 star”, “2 star”, or “3 star” system, but my criteria for reaching each level of star should be much simpler than the calculation you outline.

Something like:

• No stars - No documentation
• 1 star - Any documentation at all.
• 2 stars - More than 20% of any category of symbol has an abstract.
• 3 stars - More than 40% of any category of symbol has an abstract and there is some level of curation.

What I don’t like about this is that there would be no way for us to explain that star system (or any other system) from a badge since it would link directly to the documentation pages.

I still think a simple “Documented” badge would be better. Just my 2 pennies, though!

[maxxfrazer]

Yeah totally get that, I might be trying to find a convoluted solution for something that no one really needs or is asking for 😅

⚪, 🔴 , 🟠, and 🟢 colours on the badges could convey the star levels pretty well. I'd side for raising the 3 stars to ~60%, but it might be more intuitive if the current package numbers were graphed (@finestructure 👀).
For the logic, I'd suggest just looking at just Types and Members, as Globals are usually very small in number in my limited experience, so could sway the values with just one or two global properties.

[finestructure]

Graphed how exactly? I guess you're referring to the graphs I made across all our docs packages but I don't quite see how that related to docs for an individual package 🤔

As to coverage stats, I feel like coverage has some value to authors to spot gaps but IMO less so as an overall quality metric. In testing it's a useful tool to see where you need to focus. However, for a package consumer who doesn't know the codebase a test coverage difference of 10% or 20% might look meaningful when it really isn't (because of what those tests cover).

I feel like the strongest signal is the presence. Once you turn it into a KPI it'll just end up being gamed. (I can make any package get > 95% coverage with my 1M LOC 100% coverage "NOP bomb" 🤣.)

To bring this back to docs, I think the bigger value in exposing these values is for authors to be guided to where they may want to invest effort. I'm not 100% convinced these numbers would do that for me in the case of docs, though. That's something that's much clearer in case of test coverage.

I think for docs, there's a lot more value in a simple "getting started" doc and some "common use case" examples. But that's very hard to capture with automated metrics I think.

You mention docbuild - how is that run exactly to get these numbers? If it's easy to integrate it might be something to run and have report back some numbers so we don't discuss so much in the abstract.

[finestructure]

Oh, I see, I guess you meaning graphing the coverage numbers across the docs package to see the coverage distribution 👍

[heckj]

example script from one of my repos (docreport in my case) that provides these numbers:

#!/bin/bash

set -e  # exit on a non-zero return code from a command

rm -rf .build .crdt-graphs
mkdir -p .crdt-graphs

$(xcrun --find swift) build --target CRDT \
    -Xswiftc -emit-symbol-graph \
    -Xswiftc -emit-symbol-graph-dir -Xswiftc .crdt-graphs

$(xcrun --find docc) convert Sources/CRDT/Documentation.docc \
    --analyze \
    --fallback-display-name CRDT \
    --fallback-bundle-identifier com.github.heckj.CRDT \
    --fallback-bundle-version 0.1.9 \
    --additional-symbol-graph-dir .crdt-graphs \
    --experimental-documentation-coverage \
    --level brief

[finestructure]

Great, thanks Joe!

As an aside, is there are reason you're using $(xcrun --find $tool)? We're using just xcrun $tool everywhere and I'm wondering if that comes with downsides.

[heckj]

It's lingering from when I had multiple Xcode beta's installed and kept forgetting to switch around in them. I'm not aware of any notable delta between them - the find returns the path to the specific tool, which was useful in some earlier "workaround" scenarios (specifically for docc-exec)

[finestructure]

What I've found to be very reliable (it's what we use in our builder machinery) is

env DEVELOPER_DIR="/Applications/Xcode-14.1.0.app" xcrun $tool

That makes it very easy to see what's being used and roots everything to the given Xcode version – just in case you didn't know about DEVELOPER_DIR (I didn't before building the build system :)).

[heckj]

Ahh yes, I remember that! I think I learned about it when I was first mucking about with Swift Package Manager and working on it, but I haven't used it in practice in a couple years now. I should go back to using that!