Decide and implement a solution for internal-facing APIs

[sffc]

We've hit multiple snags recently where we wanted to change a mostly-internal function that was pub and were blocked because of semver:

• cmp_iter functions in icu_locid
• new_owned and Cart types in icu_provider
• Probably more

Some thoughts/comments/problems:

1. Even if these had been public-internal functions, we still have some desire to keep them stable-ish because then we can break hard tilde dependencies between components/utils (Revisit policy around component crate interdependent versions #4343)
2. If the functions are #[doc(hidden)], they are less discoverable including to ICU4X developers. We can't expect ICU4X developers to look into the source code just to see if a function they want happens to be there.
3. These functions should most likely be excluded from FFI by default
4. If we are poking holes in a util crate, the holes we poke should probably be part of the public API of that crate, because it is a utility crate. However, within ICU4X component and provider crates, it is okay and expected for us to poke holes.
5. Previous discussion: Document traits as either sealed or implementable #4338 (comment)

Here's a proposal:

Add a macro_rules that generates the following boilerplate:

#[doc = "<div class=\"stab unstable\">"]
#[doc = "🚧 This code is internal to ICU4X; it may change at any time, in breaking or non-breaking ways,"]
#[doc = "including in SemVer minor releases. It can be enabled with the \"icu4x_internal\" Cargo feature."]
#[doc = "</div>"]
#[cfg(feature = "icu4x_internal")]

The macro_rules can live in icu_locid for now. We can move it later.

To annotate an internal API, you can do this:

/// These are docs for my internal API
icu_locid::_internal::icu4x_internal_api!()
pub struct Foo {}

The icu4x_internal Cargo feature does not bubble up to the metacrate. It can be selectively enabled whenever it is needed. One advantage of making an icu4x_internal feature is that we can disable it when uploading docs to docs.rs (https://docs.rs/about/metadata), although I don't think there's a way to disable just that one feature; we would need to switch from --all-features to an explicit feature list per-crate, which is error-prone. Alternatively, I'm fine if we say we don't introduce a Cargo feature for this and simply let these APIs live in the docs.

In the 1.5 or 2.0 timeframe, I suggest that we scrub our API surface and add this annotation where appropriate. A lot of things that are currently #[doc(hidden)] should probably use the annotation instead, and likewise things that are currently public stable should also probably use the annotation.

If we agree on this, I may suggest we make a similar annotation for the provider module.

Needs approval or input from:

• @Manishearth
• @zbraniecki
• @hsivonen
• @robertbastian

[sffc]

Another solution I'm open to considering is to have the macro generate #[doc(hidden)], in addition to the internal warning, and then when generating internal docs use --document-private-items which includes doc(hidden) APIs.

It would make me very very happy if Rust itself had an annotation for this. Anything like "this API might not be semver safe" which generates a warning when using it cross-crate.

[Manishearth]

I think what we should do for these is that we have the macro generate cfg_attr(not(internal_docs), doc(hidden)) and generate our internal docs with RUSTFLAGS=--cfg internal_docs. We shouldn't use a cargo feature, we should use a cfg (since cargo features get enabled with --all-features)

Generating that doc header seems nice, if we use a macro we can tweak the precise solution over time.

[sffc]

@Manishearth's proposal SGTM.

Either way I would like a macro so that it's easy to find and tweak over time.

[Manishearth]

Concrete proposal:

• We have a macro
• It marks things as doc(hidden) except in cfg(icu4x_internal_docs) mode
• ICU4X docs-gen has that CFG set
• (optional) there's a nice "unstable feature" header

LGTM to me and Shane.

[sffc]

Specifically, the cfg should be set on the job that creates the docs uploaded to https://unicode-org.github.io/icu4x/rustdoc/, and the "unstable feature" box should be included so that people referencing those docs don't think the function is public stable.

[sffc]

Open question: where should the internal docs macro live? I propose it lives in a macros file that we copy around the repo into each crate and have a CI job to ensure it remains updated, similar to LICENSE.

Needs approval:

• @Manishearth
• @robertbastian